Guest Post by David M.Schultz On The Review Process

On April 15 2010 I posted

Comments On The Peer-Review Journal Publication Process And Recommedations For Improvement

Following is a guest post in response by the author of the excellent BAMS paper. David is a Professor in Division of Atmospheric Sciences and Geophysics Department of Physics University of Helsinki and Finnish Meteorological Institute
and Centre for Atmospheric Science School of Earth, Atmospheric and Environmental Sciences University of Manchester.

GUEST POST BY DAVID SCHULTZ ON THE REVIEW PROCESS

I agree with your post almost entirely.  I agree that more openness in the peer-review process should be encouraged.  I am Asst. Editor of a relatively new journal called the Electronic Journal of Severe Storms Meteorology www.ejssm.org that posts all substantive reviewer comments and maintains a discussion board for discussion of each new article that gets published.  I appreciate the openness and supportiveness for the publication process that the reviewers (for the most part, I have to admit) have embraced.

I would like to address concern I have about your recommendation that just one reviewer recommendation for rejection should not be sufficient for rejection. I actually find it quite rare that two or all three reviewers recommend rejection, except for the most egregiously bad manuscripts.  Were I to follow your recommendation, Monthly Weather Review (where I am the Chief Editor) would have a rejection rate easily less than 10%, allowing a lot more substandard papers through to get published.  When I mean substandard, I do not mean disagreement about methods or interpretation (which we see from time to time in meteorology, but the arguments are not as vicious as in climate science or in data assimilation, for sure).  I mean poorly written, mistakes, unsubstantiated claims masquerading as evidence, low-quality figures, to just name a few issues.

One of my favorite quotes is by G. K. Batchelor, and it is a philosophy I believe in:

“Papers of poor quality do more than waste printing and publishing resources; they mislead and confuse inexperienced readers, they waste and distract the attention of experienced scientists, and by their existence they lead future authors to be content with second-rate work.”

If all reviewers were created equal and were award-winning reviewers, then I would agree with you: only reject a manuscript if 2 or 3 reviewers recommend it.  However, my experience is that current Monthly Weather Review reviewers (who are not that dissimilar from other reviewers that I have seen at my “other” editor jobs at Atmospheric Science Letters and EJSSM) don’t often recommend rejection.  As Schultz (2009) in Scientometrics shows, only 18% of reviews recommended rejection.  I believe this is because (i) most reviewers do not catch many of the errors in the paper, (ii) other reviewers might be willing to give the benefit of the doubt to the author and give them a chance to fix the manuscript up, (iii) others may just be gun shy and not be willing to pull the trigger and reject, or perhaps some other reason.  

In my experience as editor (approaching 500 manuscripts), for many papers that I have received one recommendation for rejection and I have proceeded with allowing revisions, I have been disappointed in the ability of the authors to fix the problems or the peer-review process drags out to three or more rounds, which I think is unacceptable.  I should have rejected those manuscripts outright.

Therefore, the only statement in your post that I would disagree with is where you say that my paper shows that “there is a serious problem with the peer review process.”  Peer review isn’t perfect, I agree.  But, I don’t think my paper makes any such claim.

For what it’s worth, I think the emphasis at universities on publishing in Science and Nature is overrated and misdirected.  One of the worst papers I’ve ever read was published in Nature, so they are not a bastion of “prestige” in my opinion.

Furthermore, Science and Nature are their own breed.  If we all boycotted them and their self-inflated sense of prestige, and we said that’s not how we want our science to be, things would change or they would go away.

Thanks for allowing me the opportunity to respond to your blog post.

David Schultz
Chief Editor, Monthly Weather Review
Assistant Editor, Electronic Journal of Severe Storms Meteorology
Associate Editor, Atmospheric Science Letters
Editorial Board, Geophysica
University of Helsinki, Finnish Meteorological Institute, and the University of Manchester