July 23, 2014

Peer review: New approaches to an old system

by Bonnie Denmark

A scientist’s reputation and funding for research can hinge on recommendations by other scientists in the centuries-old peer review process. To ensure a credible body of scientific knowledge, research papers and proposals are evaluated by other scientists in their field—or peers—before they are accepted for publication or funding.

A reviewer at the National Institutes of evaluates a grant proposal. (Center for Scientific Review, National Institutes of Health)

A reviewer at the National Institutes of Health evaluates a grant proposal. (Center for Scientific Review, National Institutes of Health)

However, the system is far from perfect. The process is slow, often taking 18 months from submission through peer review to publication. Concerns arise over quality: Since reviewers are asked to give time-consuming, in-depth evaluations for no compensation, reviewers may suffer from burnout after reviewing numerous articles or proposals. A 2010 study revealed that 92% of reviews by individual peer reviewers deteriorated in quality and usefulness over a 14-year period.

To encourage honest evaluation, reviewers are usually anonymous although the identity of the researcher submitting the article or proposal is known. This one-sided practice of single-blind reviews invites its own set of problems. Reviewers may let personal bias interfere with their assessment, even subconsciously. In the worst case, anonymity may facilitate idea-stealing. Outright unethical behavior is rare but not unheard of in the scientific community.

Many researchers have called for changes in the peer review process and favor a speedier, more transparent, less taxing review procedure. Here are some interesting alternatives explored recently:  

  • Applicant as reviewer (reported in Science, 18 July 2014). In a “radical alternative” to traditional peer review, the National Science Foundation (NSF) ran a pilot in which applicants doubled as reviewers. Each applicant evaluated and ranked seven other proposals from researchers competing for the same funds. To prevent unfair judging of competitors, reviewers earned extra points if their assessments were in line with the consensus. One potential disadvantage of the bonus point system is that it could discourage creative thinking by discouraging evaluators from thinking outside the box. But overall, results were encouraging. The method saved time and money and may have produced higher quality reviews since each proposal was reviewed by seven peers rather the usual three to four, and the reviews were found to be longer. 
  • Double-blind peer review (reported in Nature, 15 July 2014). Keeping both reviewers and authors anonymous in the review process leads to greater equality in scientific publishing, particularly for women and minorities, who are greatly underrepresented in science. The Nature article reports on a study in which identical CVs were ranked higher when assigned male names than when assigned female names. And an article this month in Conservation Biology reports that double-blind review led to a 7.9% increase in articles with female primary authors. The journal is considering instituting a mandatory double-blind review process within the next year. Further, a 2011 article in Nature reported that only 16% of NIH applications from black researchers were funded, compared with 29% from white applicants in a review of 83,000 NIH grant applications. Alastair Brown, associate editor at Nature Climate Change, says of double-blind reviewing: “Removing the opportunity for subconscious bias is a good thing.”
  • Portable, authored-paid peer review (reported in Nature, 12 February 2013). Traditionally, whenever an article is submitted to a different publication it goes through another round of peer review. Work is duplicated, publication is delayed, and time and money are wasted. As an alternative, authors may now take advantage of a service offered by Rubriq, a company that aims to reduce publishing costs and speed up the publishing cycle by charging authors approximately $500-$700 for a standard single-blind review that they can carry with them. This approach outsources what was once the job of each publisher, permitting speedy turnaround and allowing independent reviews to accompany manuscripts as they travel from journal to journal. The concept has been piloted with several publishing groups and more than 500 reviewers.
  • Review via social media blitz (reported in Nature, 19 January 2011). Social media allows the scientific community to respond to work openly and publicly as soon as it is available. Rapid-fire critiques as papers are picked apart in blogs and tweets can rattle scientists who are accustomed to conventional review channels, but many researchers see an advantage in the flurry of comments that can quickly call attention to questionable research. Poorly conducted research has a better chance of influencing the work of others when not subjected to this type of extreme open review. David Goldstein, director of Duke University’s Center for Human Genome Variation, believes that “this chatter is not going to hurt solid work” but cautions against a “herd mentality” that can accompany a social media reviewing blitz. Open, online peer review is not a new concept, but the implementation needs refining and standardizing to be most useful.

 

LEARN MORE

See our module Peer Review in Scientific Publishing, and explore the Resources section.

 

REFERENCES

Callaham, M., & McCulloch, C. Longitudinal trends in the performance of scientific peer reviewers. Annals of Emergency Medicine, 57(2), 141-148. February 2011.

Corbyn, Z. Black applicants less likely to win NIH grants. Nature, 18 August 2011.

Cressey, D. Journals weigh up double-blind peer review. Nature, 15 July 2014.

Darling, E. S. Use of double-blind peer review to increase author diversity. Conservation Biology, 4 July 2014.

Mandavilli, A.  (2011). Peer review: Trial by Twitter. Nature, 469, 286-287, 19 January 2011.

Mervis, J. (2014). A radical change in peer review. Science, 345(6194), 248-249, 18 July 2014.

Van Noorden, R. Company offers portable peer review. Nature, 494(7436), 12 February 2013.

 

 

 

 

Written by

Bonnie Denmark holds an MA in linguistics and teacher certification in English, ESL, and Spanish. She has devoted her professional life to educational and accessibility issues as a computational linguist, multimedia curriculum developer, educator, and writer. She has also worked nationally and internationally as a language instructor, educational technology consultant, and teacher trainer. Bonnie joined the Visionlearning team as a literacy specialist in 2011, assisting the project by developing comprehension aids for science modules and creating other STEM learning materials.

The views expressed above do not necessarily represent those of Visionlearning or our funding agencies.