Milton Packer thinks that readers must now be the decisive judges of quality Simon Dack, MD, was the editor-in-chief of the official journal of the American College of Cardiology for 34 years. His office at the Mount Sinai Hospital in New York City was located adjacent to the fellows' room, and in the 1970s, we often dropped by to see how he made editorial decisions. It was one of the most amazing learning experiences of our careers. Dr. Dack was the quintessential editor. The journal was his vision; it reflected his values. He solicited papers from the best and brightest. If he asked you to submit a paper, you took the invitation seriously. If you submitted original research, you made sure that it was worth his time. Starting in the 1970s, the concept of external "peer review" blossomed in cardiology. Officially, the refereeing process moved out of the inner circles of learned societies and involved the critique of papers by outsiders of equal competence. Dr. Dack sent papers out for peer review, but he considered the feedback as advice, rather than an authoritative word. One afternoon in 1978, I watched him reject three papers that had received two positive reviews, while accepting a paper that had received two negative reviews. I asked him how he could do that. His response: I read every word of every paper and every review. I know the reviewers' strengths, weaknesses, and biases. I ask for their opinion. I am not asking them to vote. I am the editor; this is my journal; I make the decisions; and I take responsibility. The process was not intended to be flawless. Dr. Dack readily admitted that he made mistakes, but he went out of his way to fix them. The process was certainly not democratic or unbiased. But it worked. He gave a voice to hundreds of young investigators. Many of the most important (and paradigm-shifting) papers in cardiovascular medicine were published because of decisions he made. If there was controversy, he fed the flames. He was exceptionally receptive to new ideas that had scientific merit, and he gave them a platform. If he asked you to review a paper, you accepted the invitation and worked hard at it. If you submitted a lazy review, he knew it, and he told you so. You never made that mistake again. Because most journals in medicine at the time were led by unimpeachable intellects like Dr. Dack, it was easy to keep up with scientific advances. For many, one only needed to read the New England Journal of Medicine each week and a few other journals each month and remember what they published. Readers trusted journals to be a reliable source of information. But 40 years later, the principles, philosophy, and practices of Dr. Dack have disappeared. The trust that physicians formerly placed in journals has evaporated. The reason: the peer-review process doesn't work anymore. Now there are hundreds of cardiology journals, and each publishes hundreds of papers each year. It is really easy to submit a paper online, but what happens then? Only a few journals have a single visionary editor who knows every reviewer personally. Instead, the typical journal has dozens of editors who may or may not have the time to read each paper carefully. Instead, they spend a lot of time finding colleagues to perform external peer review. How easy is it to get good reviewers? It is impossible. Currently, most leading researchers in any given discipline routinely decline to be reviewers. Doing a good review takes hours, and they just don't have (or won't make) the time. Many simply say no. Others hand the work over to junior associates -- without carefully reviewing their submitted opinions. Editors routinely struggle to find external reviewers. Often, they must invite 10-15 people to find two or three who agree to review. Even then, the reviews are often superficial and unhelpful. Some reviewers spend only a few minutes looking at the data, and make recommendations based on their fondness (or lack thereof) for the authors or for the conclusions -- rather than based on solid standards of scientific examination. If two or three reviewers carry out their responsibilities with equal lack of rigor, egregious errors can be missed, even in top-tier journals. When the reviews come in (often quite late), editors often feel compelled to accept the opinions of the reviewers even if they are inadequate or biased. Editors are reluctant to overrule the reviewers, fearing that they will refuse to review again in the future. The desire to keep reviewers happy means that even minor revisions are returned to them for a final blessing, thus adding months to the peer-review process. What happens when a paper is rejected? Typically, it makes little difference. The authors will instantaneously resubmit to another journal -- without necessarily fixing any of the errors or limitations that led to the previous rejection. The process continues until some journal is willing to publish the work. There are more journals than there is worthwhile content, and many lower-tier journals struggle to fill their pages. Some will accept nearly every paper, especially if an author is willing to pay outrageous publication fees. In these cases, the peer-review process is a mere formality. If authors are sufficiently persistent, their papers eventually get published somewhere, and sadly, they reside as apparent equals along with their more worthy counterparts on PubMed. Peer review is not dead, but it no longer achieves what it is intended to do. Moreover, authors can now bypass the process completely by posting unreviewed work on publicly accessible preprint servers. Currently, these are intended to constitute a transitional state, but soon, postings on a preprint server may replace traditional peer review entirely. Think of this the next time you read a paper and ask: How did this awful manuscript ever get published in a peer-reviewed journal? Here is what you should be thinking instead. Of course, this paper got published. Now you need to read it carefully to see if it says something credible and worthwhile. The responsibility of distinguishing quality has shifted from the editors to the readers. Reading a published paper critically is an awesome responsibility. It takes time, effort, relevant background, and methodological experience. But these days, it is more important than ever. Source
