A meditation on the causes and concerns of observed rise in scientific journal retractions

Rise in Scientific Journal Retractions Prompts Call for Reform, by Carl Zimmer in the NYTimes is an interesting discussion of the recent observed increase in scientific misconduct and its possible causes such as publishing and grant pressure from Universities. Additionally, Zimmer has a post on his discovery blog linking to the Retraction Watch , and to an article in the Boston Globe.

It is pointed out there that recent advances in methods for detecting scientific fraud could also partially explain the part of the increase in retractions. However there is much indicating that misconduct may be a growing problem in research and academia. So, assuming the observations are accurate and there is an actually increase in scientific misconduct over time.

Why is that?

Grant and publication pressure have been said to be the culprits, sure. But why were they introduced in the first place, and what changes to the system does their such a metric create?

Talking to friends and colleagues more in the academic fray than myself I get the feeling that many institutions, often successful ones, have adopted a corporate-like strategy to research. The result is in many ways a production rule for professors rather than for science.

Scientific work is still needed in order to climb the career ladder but having the motivation "I want to become a professor" rather than "I am curious about science" is probably going to get you further. This is not to say that someone interested in a career can not be interested in science, and vice versa. A skilled administrator and teacher, with a good feeling for science, is perhaps the archetype of an excellent professor with a good scientific output.

What I am saying however is that it seems as if interest in an academic career is required - and measured - while interest in science is assumed , taken for granted. Not so strange really. Interest in a specific subject was, and hopefully still is, the main motivation for going into academia. While the science-priority assumption is holding, measuring publication and grant counts ought to work splendid, serving as a framework for assessing productivity and promoting hard-working researchers.

However the prerequisite for a working system may no longer be true. Gradually science has shifted into the spotlight of society and an increased number of students are pursuing a higher academic degree. Both very positive trends likely increasing scientific awareness and rational thinking in our society. However this also puts strain on the existing academic system due to harder competition.

Competition is often desired as it has the effect of making a process more efficient within some environment. But, all such optimization is with respect to whatever the environment awards; in some way selects for. In modern academia this primarily means grants and publication counts. Scientific detail and quality is desired but hard to measure and therefore simply not rewarded.

Taking a scientific shortcut in the interest of a continued academic career is beneficial in this system. The corners cut may not even be clear cases of misconduct initially, perhaps making sure ones name is on a paper with the least amount of work, or selectively picking test methods, or brushing time-consuming, while just, reviewer suggestions under the carpet.

However, in order to compete with the careerists others will need to cut some more corners and the behaviour of scientists will gradually shift. Do what others are perceived to do, but be a bit better at it.

Collectively this creates a sliding slope. I am not suggesting that this is a common conscious strategy of the individual scientist - even though it may be in some cases - but rather a trend and behaviour of the academic system as a whole.

Universities for their part have even fewer reasons to enforce strict scientific principles than the scientists. A portion of their budget usually comes from overhead costs from research grants. The amount probably outweighs potential bad publicity of random cases of misconduct that may or may not affect the opinions of prospective students; especially for well-known institutions with a good reputation. In addition, modern universities usually have a large management structure primarily organized for business with little or no training in scientific principles. Finally many modern universities have an outspoken strategy of creating intellectual property, something that may hinder full dissemination of results and affect the ability of identifying misconduct.

Again, this is a generalization for the organization, and not on the individual level. But where did the career focus, the publication pressure, and the grant count originate? Why are they being measured and quantified in the first place? The closest explanation at hand is the need for grant givers to measure performance and follow up their involvement. However I do not know if it has always been so. I am afraid my own research on the matter is lacking, and brief at best.

Science the endless frontier , Vannevar Bush's 1945 report to the U.S. president regarding strategies for transition of the science effort during the second world war to a civilian era, sometimes cited as the inspiration for modern grant agencies of North American model is interesting reading, but not specific in this case. While the need for basic research and scientific freedom is accentuated, and a structure for a national research foundation is indeed proposed, I found no mention of how the knowledge output should be measured.

I find it improbable that the grant system has always exhibited today's very outspoken reliance on measured performance, and will assume it has evolved over time. It would be interesting to do some more proper research into the history of scientific funding and the history of different grant agencies. For now, I will however rely on the sparse and somewhat incomplete Wikipedia pages for the History of Science Policy and Science Funding and hypothesize that current quantification of results has evolved as science has grown in importance and clashed with political and economic factors.

The History of Science Policy page has a very brief paragraph on how the Oil crisis affected government supported research in the U.K. and U.S. It is not a far stretch to imagine that spending cuts lead to heavier resilience on the evaluation system used to divide the new, slimmer funding portion among institutions. I actually do believe that untill then the mechanism - encouraging competition through a simplified environment focused on output - could have had a beneficial role in the birth of modern science. Especially in the period after the World Wars.

Initially it made it easier to motivate funding and evaluate projects, thus accelerating many academic fields and increasing the role of science in society. In the near future on the other hand the current system may, if not reformed, slow the process down instead.

When the basic assumption - that all academics put science first - is no longer true for some critical number, the system becomes unstable. Those less adherent to their scientific principles will initially be at an advantage as quantified output, not ideals or true knowledge production, is what determines access to resources. A gaming behaviour may spread as a winning strategy, deteriorating scientific curiosity and principles over time, causing the system to grind to a halt. An individual optimization strategy does not necessarily lead to a global optimum for the system.

In order to stabilize such a system resources must be spent on quality control, slowing the process down instead. Interestingly, while as scientists we believe that we have started to see a deterioration in quality, and rise of misconduct in our own field, the scenario is perhaps not new in society. It could be the latest example of a development many processes go through as they become increasingly important to (Western) society.

By building a framework of artificial rules it is possible to quantify the process more rapidly. That what is measured is not the original product, but a number in some metric. A principal component, easily optimized. Initially there is a boost in production due to faster turn-around times and competition while assumed - but not selected for - principles are still a legacy saturating the system.

It does however lead to a niche for gaming the process. In fact, not even gaming, as the rules have changed. Merely focusing on what counts.

With the passage of time the original motivation and principles are forgotten, and no longer piggybacking on the new efficient environment. More success and resources can be found by not spending energy adhering to them.

For a couple of examples outside science think of industrial mass production and illegal doping in sports. The former is often attributed to the advent of machines and steam power, but without managers and a system for measuring production rate the results would only have been craftsmen with better tools. Instead most of the modernized world have access to mass-produced trinkets and technology.

At the same time we have an unspoken understanding that these are of lower quality and lack craftsmanship. In fact, industrial production and artisanship were once aligned. If my memory serves me, Matthew B. Crawford writes in his book Shop class as Soulcraft that craftsmen were the first hired by the factories, and as a result the crafting knowledge disappeared. Something that created a small management crisis when this initial generation disappeared. It turned out that spending time getting craftsman skills was not that essential for many conveyor-belt-jobs. Artisanship was the assumed quality but not the measured one.

Professional sport is also experiencing the effects of result focus, having been forced to develop clear strategies for dealing with doping and other forms of cheating the last few years. Athletic competitions have a long history, but their roles in society and their economic value have gotten explicit over the last century. Quantification of results has been essential to this process. Winning is to have the best result. Except the equivalence only holds in sports if the result is achieved fairly. Fairness, the athletic ideals, were in many ways an assumed quality of sports,but the quantified result is, and have been, in focus. As a result professional sports organizers have had to deal with an increased amount of doping, spending much resources trying to battle it. The explanations heard from athletes caught cheating in this way reads in many cases very similar to those that have started cropping up from scientists in the same category. The pressure to produce. All focus on the results.

Will we also hear that "everyone else is doing it" from academics soon?

But what about the other side of the coin? The same focus on world records tempting the cheater is the spur responsible for today's specialized athletes striving to be even stronger and faster. Likewise, there is no denying that industrialization and mass production have shaped the world we live in today. The industrial technologies used to mass manufacture low quality junk share process ancestry with those used to precision machine delicate components for space travel, the internet, and modern medicine. They go hand in hand, the former once accelerated the later, and even if I do not think we need the junk industry today, it was probably once prerequisite for high-tech engineering. It evolved, spurred by competition. Could our civilization have created the medicines, the moon rockets, and the LHC from a pre-industrialized state without mass production of trinket and junk?

Perhaps, but I believe it is possible to argue it would have taken much longer. Right or wrong, I can not say. It happened. Perhaps we should not regret the past but try to reform for the future? Now, taking this back to science, I believe that the key observation in the above examples is higher production rate at the cost of increased failure rate (due to the noted change in production rules).

Think of the observed increase in scientific misconduct and questionable research findings; compare it with error-prone and mass-produced trinkets for instance.

We have a potential information output most probably unmatched history. The question is: how much of that information is knowledge? Has quality. Does the increased output rate make up for the associated increase of false knowledge due to academics focusing on rushing out results, or even deliberately cheating, rather than applying the scientific methods? Perhaps the change to a quicker, quantifiable, yet more corruptible academic system was an unavoidable development for science. For society to reach the next level of whatever we are becoming.

Sometimes I have a vision of humanity as running towards a chasm. There might be no turning back but to go faster and faster and hope to fly before the cliff edge. Still, I remain wary.

Meditating on the issue is fine, but specific action becomes an issue of politics and economics; outside the scope for this essay. No doubt if academia continues the same path as other functions in society such as manufacturing, finance, or indeed professional sports, we need a reform.

As in the other areas there will be debates on the different way of going about doing so. Some will argue for laws and control, others for self-regulation. But no matter what combination of remedies proposed the purpose will most likely be to patch the current system. To enforce quality by creating a new artificial environment attempting to select partially for the assumed and lost quality property while still allowing the accelerated growth. An ersatz for scientific curiosity. I don't know if that is the right way to go. Perhaps there truly is a need for a more throughout reform, one that let us keep the pace we have built up during the last century, but keeping with quality.

One alternative approach that has been tried is bounty-science where a (cash) prize is promised for the solution to a specific scientific or technical problem. Examples include the Millenium Prize Problems from the Clay Mathematics Institute , and the various challenges from the X Prize Foundation . In the bounty approach measurement of success is easy: either a solution to the given problem is found, or it is not. There is no or little separation between purpose and measurement, and competition becomes a positive driving force.

There are two main drawbacks however: first the prize is paid after a satisfactory solution is found, and second the problem must be defined. Thus it is no general solution for grant agencies who pays before the work is done, and neither for basic research where a big part may be to actually discover new problems, not solving existing ones. Of course while we are debating the scientific system continues to change. Novel reviewing and publishing mechanisms, web archives, and online social networks will perhaps be the answer to some of the problems. A reform taking academia in a completely new direction. This would be an evolved fix, and not a planned one. A self stabilizing system. Evolved or regulated however, nothing says the conflict of quantity and quality will stop.

There is another side to this whole discussion: the principles of science. Principles that mean you do your best to scrutinize your own results first, that you try to have a skeptic's eye, and be a critical reader. Not that these things will prohibit errors - I've produced a few myself all while doing my best - nor will they stop bad science. But they are a safeguard, and part of the assumed qualities of scientists. The danger of continuously patching an artificial system is that the original principles, once assumed universal, may disappear altogether. The scientific principle form the ethics and a moral code for science.

So, now we are back at an old problem. Morals. Personal freedom. Do you want to honour an ethical code, or have a third part policing you? You can not have both without the system collapsing. Not in a world where we perform that separation of quality and measured quantity. Perhaps the scientific process can be reformed thoroughly; though I am pessimistic enough will happen in time.

Any reform of the academic system in the near future will likely consist of patching through quality control or the illusion of self-regulation, most likely oscillating between the two. Science has become too important to society. Or rather knowledge production; that might be a more appropriate name. Yet it will still be called science of course, and word and meaning will in worst case separate. The scientific ideals and principles will exist, even without a name, but may not be as prevalent in our society. Perhaps they have never been more than idealistic dreams of a Utopia that never existed.

But ideals is something to strive for, and I believe that there is something very real lost for society if scientific principles in academia deteriorates. In a production system quantified to the extreme disregarding ethics will always be a winning strategy.

But as with antivaxxers having an intuitive delusion that not getting the shot is the winning strategy - which it may be only in case (almost) everyone else get the vaccine - we run the risk of losing our herd immunity. Against ignorance.

.L