Tuesday, October 9, 2007


For the peer review process on Wikipedia, see Wikipedia:Peer review.
Peer review (known as refereeing in some academic fields) is a process of subjecting an author's scholarly work, research or ideas to the scrutiny of others who are experts in the same field. It is used primarily by editors to select and to screen submitted manuscripts, and by funding agencies, to decide the awarding of grants. The peer review process aims to make authors meet the standards of their discipline, and of science in general. Publications and awards that have not undergone peer review are likely to be regarded with suspicion by scholars and professionals in many fields. Even refereed journals, however, can contain errors.
In the case of manuscripts, the editor will pass manuscripts that are accepted for publication to a publisher who will be responsible for organizing redactory services, printing and distribution of the publication. In specialist academic (scholarly) journals, the editor (or increasingly group of editors) is normally a well-respected academic in the field, and edits the journal on behalf of a learned society or a commercial publisher. Some journals have professional editors employed by the owner of the journal. An editor is ultimately responsible for the quality and selection of manuscripts chosen to be published, usually basing their decision on peer review, although the authors are always responsible for the content of each manuscript. The editor does not revise and correct spelling, grammar and formatting - that process is carried out by a copy editor, although the editor controls the quality of the process.

Reasons for peer review
In the case of proposed publications, an editor sends advance copies of an author's work or ideas to researchers or scholars who are experts in the field (known as "referees" or "reviewers"), normally by e-mail or through a web-based manuscript processing system. Usually, there are two or three referees for a given article.
These referees each return an evaluation of the work to the editor, including noting weaknesses or problems along with suggestions for improvement. Typically, most of the referees' comments are eventually seen by the author; scientific journals observe this convention universally. The editor, usually familiar to the field of the manuscript (although typically not in as much depth as the referees, who are specialists), then evaluates the referees' comments, their own opinion of the manuscript, and the context of the scope of the journal or level of the book and readership, before passing a decision back to the author(s), usually with the referees' comments.
Referees' evaluations usually include an explicit recommendation of what to do with the manuscript or proposal, often chosen from a menu provided by the journal or funding agency. Most recommendations are along the lines of the following:
During this process, the role of the referees is advisory, and the editor is typically under no formal obligation to accept the opinions of the referees. Furthermore, in scientific publication, the referees do not act as a group, do not communicate with each other, and typically are not aware of each other's identities or evaluations. There is usually no requirement that the referees achieve consensus. Thus the group dynamics are substantially different from that of a jury. In situations where the referees disagree substantially about the quality of a work, there are a number of strategies for reaching a decision.
When an editor receives very positive and very negative reviews for the same manuscript, the editor often will solicit one or more additional reviews as a tie-breaker. As another strategy in the case of ties, editors may invite authors to reply to a referee's criticisms and permit a compelling rebuttal to break the tie. If an editor does not feel confident to weigh the persuasiveness of a rebuttal, the editor may solicit a response from the referee who made the original criticism. In rare instances, an editor will convey communications back and forth between authors and a referee, in effect allowing them to debate a point. Even in these cases, however, editors do not allow referees to confer with each other, and the goal of the process is explicitly not to reach consensus or to convince anyone to change their opinions. Some medical journals, however (usually following the open access model), have begun posting on the Internet the pre-publication history of each individual article, from the original submission to reviewers' reports, authors' comments, and revised manuscripts.
Traditionally, reviewers would remain anonymous to the authors, but this standard is slowly changing. In some academic fields, most journals now offer the reviewer the option of remaining anonymous or not, or a referee may opt to sign a review, thereby relinquishing anonymity. Published papers sometimes contain, in the acknowledgements section, thanks to anonymous or named referees who helped improve the paper.

to unconditionally accept the manuscript or proposal,
to accept it in the event that its authors improve it in certain ways,
to reject it, but encourage revision and invite resubmission,
to reject it outright. How it works
At a journal or book publisher, the task of picking reviewers typically falls to an editor. When a manuscript arrives, an editor solicits reviews from scholars or other experts who may or may not have already expressed a willingness to referee for that journal or book division. Granting agencies typically recruit a panel or committee of reviewers in advance of the arrival of applications.
In some disciplines there exist refereed venues (such as conferences and workshops). To be admitted to speak, scholars and scientists must submit papers (generally short, often 15 pages or less) in advance. These papers are reviewed by a "program committee" (the equivalent of an editorial board), who generally requests inputs from referees. The hard deadlines set by the conferences tend to limit the options to either accept or reject the paper.
Typically referees are not selected from among the authors' close colleagues, students, or friends. Referees are supposed to inform the editor of any conflict of interests that might arise. Journals or individual editors often invite a manuscript's authors to name people whom they consider qualified to referee their work. Authors are sometimes also invited to name natural candidates who should be disqualified, in which case they may be asked to provide justification (typically expressed in terms of conflict of interest). In some disciplines, scholars listed in an "acknowledgements" section are not allowed to serve as referees (hence the occasional practice of using this section to disqualify potentially negative reviewers).
Editors solicit author input in selecting referees because academic writing typically is very specialized. Editors often oversee many specialities, and may not be experts in any of them, since editors may be full time professionals with no time for scholarship. But after an editor selects referees from the pool of candidates, the editor typically is obliged not to disclose the referees' identities to the authors, and in scientific journals, to each other. Policies on such matters differ between academic disciplines.
Recruiting referees is a political art, because referees, and often editors, are usually not paid, and reviewing takes time away from the referee's main activities, such as his or her own research. To the would-be recruiter's advantage, most potential referees are authors themselves, or at least readers, who know that the publication system requires that experts donate their time. Referees also have the opportunity to prevent work that does not meet the standards of the field from being published, which is a position of some responsibility. Editors are at a special advantage in recruiting a scholar when they have overseen the publication of his or her work, or if the scholar is one who hopes to submit manuscripts to that editor's publication in the future. Granting agencies, similarly, tend to seek referees among their present or former grantees. Serving as a referee can even be a condition of a grant, or professional association membership.
Another difficulty that peer-review organizers face is that, with respect to some manuscripts or proposals, there may be few scholars who truly qualify as experts. Such a circumstance often frustrates the goals of reviewer anonymity and the avoidance of conflicts of interest. It also increases the chances that an organizer will not be able to recruit true experts – people who have themselves done work like that under review, and who can read between the lines. Low-prestige or local journals and granting agencies that award little money are especially handicapped with regard to recruiting experts.
Finally, anonymity adds to the difficulty in finding reviewers in another way. In scientific circles, credentials and reputation are important, and while being a referee for a prestigious journal is considered an honor, the anonymity restrictions make it impossible to publicly state that one was a referee for a particular article. However, credentials and reputation are principally established by publications, not by refereeing; and in some fields refereeing may not be anonymous.
The process of peer review does not end after a paper completes the peer review process. After being put to press, and after 'the ink is dry', the process of peer review continues in journal clubs. Here groups of colleagues review literature and discuss the value and implications it presents. Journal clubs will often send letters to the editor of a journal, or correspond with the editor via an on-line journal club. In this way, all 'peers' may offer review and critique of published literature.

Recruiting referees
Peer review can be rigorous, in terms of the skill brought to bear, without being highly stringent. An agency may be flush with money to give away, for example, or a journal may have few impressive manuscripts to choose from, so there may be little incentive for selection. Conversely, when either funds or publication space is limited, peer review may be used to select an extremely small number of proposals or manuscripts.
Often the decision of what counts as "good enough" falls entirely to the editor or organizer of the review. In other cases, referees will each be asked to make the call, with only general guidance from the coordinator on what stringency to apply.
Very general journals such as Science and Nature have extremely stringent standards for publication, and will reject papers that report good quality scientific work if editors feel the work is not a breakthrough in the field. Such journals generally have a two-tier reviewing system. In the first stage, members of the editorial board verify that the paper's findings -- if correct -- would be ground-breaking enough to warrant publication in Science or Nature. Most papers are rejected at this stage. Papers that do pass this 'pre-reviewing' are sent out for in-depth review to outside referees. Even after all reviewers recommend publication and all reviewer criticisms/suggestions for changes have been met, papers may still be returned to the authors for shortening to meet the journal's length limits. With the advent of electronic journal editions, overflow material may be stored in the journal's online Electronic Supporting Information archive.
A similar emphasis on novelty exists in general area journals such as the Journal of the American Chemical Society (JACS). However, these journals generally send out all papers (except blatantly inappropriate ones) for peer reviewing to multiple reviewers. The reviewers are specifically queried not just on the scientific quality and correctness, but also on whether the findings are of interest to the general area readership (chemists of all disciplines, in the case of JACS) or only to a specialist subgroup. In the latter case, the recommendation is usually for publication in a more specialized journal. The editor may offer to authors the option of having the manuscript and reviews forwarded to such a journal with the same publishers (e.g., in the example given, Journal of Organic Chemistry, Journal of Physical Chemistry, Inorganic Chemistry,...). if the reviewer reports warrant such a decision (i.e., they boil down to "Great work, but too specialized for JACS: publish in ..."), the editor of such a journal may accept the forwarded manuscript without further reviewing.
Some general area journals, such as Physical Review Letters, have strict length limitations. Others, such as JACS, have Letters and Full Papers sections: the Letters sections have strict length limits (two journal pages in the case of JACS) and special novelty requirements. In contrast, online-only journals may have no space limitations.
Proponents of the double-masked process argue that if the reviewers of a paper are unknown to each other, the associate editor responsible for the paper can easily verify the objectivity of the reviews. Single-masked review is thus strongly dependent upon the goodwill of the participants.
A more rigorous standard of accountability is known as an audit. Because reviewers are not paid, they cannot be expected to put as much time and effort into a review as an audit requires. Most journals (and grant agencies like NSF) have a policy that authors must archive their data and methods in the event another researcher wishes to replicate or audit the research after publication. Unfortunately, the archiving policies are sometimes ignored by researchers.

Peer reviewed Different styles of review
One of the most common complaints about the peer review process is that it is slow, and that it typically takes several months or even several years in some fields for a submitted paper to appear in print. In practice, much of the communication about new results in some fields such as astronomy no longer takes place through peer reviewed papers, but rather through preprints submitted onto electronic servers such as arXiv.org.
While passing the peer-review process is often considered in the scientific community to be a certification of validity, it is not without its problems. Drummond Rennie, deputy editor of Journal of the American Medical Association is an organizer of the International Congress on Peer Review and Biomedical Publication, which has been held every four years since 1986.

Criticisms of peer review
In addition, some sociologists of science argue that peer review makes the ability to publish susceptible to control by elites and to personal jealousy. Results published in low-impact journals are usually ignored by most scientists in any field. This has led to calls for the removal of reviewer anonymity (especially at high-impact journals) and for the introduction of author anonymity (so that reviewers cannot tell whether the author is a member of any elite).

Allegations of bias and suppression

Main article: Peer review failurePeer reviewed Peer review and plagiarism
A related form of professional misconduct that is sometimes reported is a reviewer using the not-yet-published information from a manuscript or grant application for personal or professional gain. The frequency with which this happens is of course unknown, but the United States Office of Research Integrity has sanctioned reviewers who have been caught exploiting knowledge they gained as reviewers.

Abuse of inside information by reviewers
It has been suggested that traditional anonymous peer review lacks accountability, can lead to abuse by reviewers, and may be biased and inconsistent,

Dynamic and open peer review
The technique of peer review is also used to improve government policy. In particular, the European Union uses it as a tool in the 'Open Method of Co-ordination' of policies in the fields of employment and social inclusion.
A programme of peer reviews in active labour market policy Each programme sponsors about eight peer review meetings in each year, in which a 'host country' lays a given policy or initiative open to examination by half a dozen other countries and relevant European-level NGOs. These usually meet over two days and include visits to local sites where the policy can be seen in operation. The meeting is preceded by the compilation of an expert report on which participating 'peer countries' submit comments. The results are published on the web.

Peer review of policy

Main article: U.S. Government peer review policies U.S. government peer review policies
Peer review has been a touchstone of modern scientific method only since the middle of the twentieth century.

History of peer review

Main article: Software peer review See also

No comments: