100%(1)1 out of 1 people found this document helpful
This preview shows page 172 - 174 out of 323 pages.
¶Why anyone – least of all the dean of a prestigious lawschool – would dignify such a crude and corrupt assault on justice is a mystery, and verytroubling to me.
AT: Peer Review KeyPeer review is riddled with flaws and should not be the basis for a credible scientific studySmith 06 [Richard Smith, Journal of the Royal Society of Medicine, 2006, "Peer review: a flawed process at the heart of science and journals", ] JMovBut does peer review `work' at all?A systematic review of all the available evidence on peer review concluded that `the practice of peer review is based on faith in its effects, rather than on facts'.2 But the answer to the question on whether peer review works depends on the question `What is peer review for?'.¶One answer is that it is a method to select the best grant applications for funding and the best papers to publish in a journal. It is hard to test this aim because there is no agreed definition of what constitutes a good paper or a good research proposal. Plus what is peer review to be tested against? Chance? Or a much simpler process?Stephen Lockwhen editor of the BMJ conducted a study in which he alone decided which of a consecutive series of papers submitted to the journal he would publish. He then let the papers go through the usual process. There was little difference between the papers he chose and those selected after the full process of peer review.1 This small study suggests that perhaps you do not need an elaborate process. Maybe a lone editor, thoroughly familiar with what the journal wants and knowledgeable about research methods, would be enough. But it would be a bold journal that stepped aside from the sacred path of peer review.¶Another answer to the question of what is peer review for is that it is to improve the quality of papers published or research proposals that are funded. The systematic review found little evidence to support this, but again such studies are hampered by the lack of an agreed definition of a good study or a good research proposal.¶Peer review might also be useful for detecting errors or fraud. At the BMJ we did several studies where we inserted major errors into papers that we then sent to many reviewers.3,4 Nobody ever spotted all of the errors. Some reviewers did not spot any, and most reviewers spotted only about a quarter. Peer review sometimes picks up fraud by chance, but generally it is not a reliable methodfor detecting fraud because it works on trust.A major question, which I will return to, is whether peer review and journals should cease to work on trust.¶Go to:¶THE DEFECTS OF PEER REVIEW¶So we have little evidence on theeffectiveness of peer review, but we have considerable evidence on its defects. In addition to being poor at detecting gross defects and almost useless for detecting fraud it is slow, expensive, profligate of academic time, highly subjective, something of a lottery, prone to bias, and easily abused.