![[Critical Mass]](http://www.erinoconnor.org/archives/cmlogo.gif)
What peer review should be
In academia, anonymity and secrecy are held up as the necessary conditions of responsible peer review at every level — from hiring and promotion to publishing. Reviewers, the thinking goes, will be more honest if there is strict confidentiality. They shouldn’t have to put their names on their assessments of whether your article or book should be published. Nor should you know what they say or write about you when you are up for promotion and tenure. If this looks like an invitation to behave irresponsibly (to be lazy or vindictive) at key moments in other people’s careers, that’s because it is. But the time-honored rationale is that peer reviewers need the freedom from reprisal that lack of transparency gives them to do their work.
But it doesn’t have to work that way. Imagine peer reviewed publishing without anonymous readers’ reports and the distortions that come with them–the shoddiness and brevity of response (why write a detailed, thoughtful report when dashing off a short paragraph will do?), the off-baseness and pot-shotting (see climategate), the gamesmanship and ego-mongering (I’ll scratch your manuscript if you’ll scratch mine; your article is unacceptable because it fails to be all about my work; etc). Shakespeare Quarterly is rethinking peer review — and getting great results.
For this year’s fall issue, a special publication devoted to Shakespeare and new media, the journal offered contributors the chance to take part in a partially open peer-review process. Authors could opt to post drafts of their articles online, open them up for anyone to comment on, and then revise accordingly. The editors would make the final call about what to publish (hence the “partially open” label). As far as the editors know, it’s the first time a traditional humanities journal has tried out a version of crowd-sourcing in lieu of double-blind review.
The verdict from several scholars who took part: mostly a thumbs up, with a few cautionary notes and a dollop of “It’s about time” mixed in.
[...]
[The journal] invited about 90 scholars … to comment. Anybody willing to publish thoughts under his or her own name could join in, but the guest editor wanted recognized authorities as part of the field.
[...]
Some of the authors acknowledged doubts going into the experiment. Would open review be rigorous enough? Was it risky to post work in progress? But “the results were terrific,” said Mr. Witmore, who wrote his paper on Shakespearean linguistic analysis with Jonathan Hope, a reader in the English department of the University of Strathclyde, in Glasgow. “It’s very different from getting a two-paragraph reader’s report from a journal,” Mr. Witmore said. “In this case, what you get is individual readers from a wide range of subspecialties zooming in on a particular paragraph, saying ‘Tell me more about this’ or ‘Why did you do this?’ It seemed more like a dialogue.”
Another scholar, Alan Galey, submitted an article about Shakespeare and the history of information. An assistant professor on the Faculty of Information at the University of Toronto, he worried that an article vetted this way might carry less professional weight—a matter of particular concern to a junior professor going for tenure. “It was very much going on faith in a way,” he said.
Mr. Galey’s dean told him to make sure the process would be rigorous and fair. The stature of the journal also helped reassure him on that point. So did Ms. Rowe’s willingness to answer questions and her decision to invite established scholars to join in. Many crowd-sourcing experiments depend on scale, Mr. Galey pointed out, but this relied “on relationships among scholars where you know you can trust somebody. It wasn’t a Wild West by any means. It was as controlled a process as traditional peer review. It was just controlled in a different way.”
Mr. Galey wound up feeling that the experiment paid off. “I got better feedback from this process than I’ve had from any other peer-review process,” he said.
Participants acknowledged that the process was more time-consuming than traditional peer review. But Shakespeare Quarterly felt that the positives far outweighed the drawbacks, and is planning to try the open review process again.
I think there is a lot to be said for people having to put their names to what they do. Academia’s elevation of anonymity as a procedural virtue has always struck me as a way of rationalizing cowardice and ensuring abuse by removing any semblance of accountability. Conversely, as Shakespeare Quarterly is showing, with names attached, people become at once more responsible and more effective peer reviewers. And so the process recovers its original aim–which was, after all, dialogue and debate among peers aimed at refining and advancing the ideas about which they share expertise and interest.
My experience with peer review has been very odd. Of course, I don’t expect publication on my first few submissions, and I hope for good comments that will help me revise. Of the few manuscripts I’ve gotten back, here are the observations I’ve had:
1. I can tell who has done the review. There are only a handful of scholars in my particular sub-field, so it’s pretty easy to figure out who did your review. Also, because of this (there’s really only one conference that covers the field, with a couple of others that cross over with it sometimes), I think it’s pretty clear the scholars know who is writing the articles if the writer is an established academic. And frankly, some of the articles by “Established” academics I’ve seen are rather terrible, so clearly their articles were approved by the same people rejecting mine.
2. The comments are usually too brief and indicate they likely only read a few pages at random. One person criticized me for not justifying my theoretical approach, ignoring the several pages I dedicated to justifying my approach. Another said that their research disagreed with my conclusions, so I clearly hadn’t thought the matter through enough (but if I had agreed, I bet they would have said it was redundant and didn’t need to be published).
3. However, I did have, in one case, a very, very good and thorough response. Interestingly, that scholar identified himself in the comments, mainly because I was relying a lot on his work. But his comments were excellent and indicated real engagement on his part with my article. So it can work, when academics are not concerned with protecting their turf.
“peer reviewers need the freedom from reprisal”…but the very anonymity that protects the *reviewers* from reprisal allows costless reprisals against the *author* of the article. Should this have been obvious in the first place?
Seems to me that a more open review process, conducted on-line, should result in the publishing of the resultant article in on-line form, with access available to all either free or at a nominal charge. I do not understand what academic, social, or economic purpose is served by the continued existence of very expensive academic journals with very limited circulations. (Other than, of course, the financial well-being of the owners and employees of such journals.) It is extremely irritating to see a reference to an interesting-looking piece of academic research and find that it is available only by paying $35 or so for a single access to the journal in which it appears. Particularly when the research in question is partially or wholly tax-supported.
David — I agree with you whole-heartedly.
I think we need to make a distinction between “feedback” and “candid assessment of quality.” It’s clear to me that the *feedback* to the *authors* in the process described here is more valuable than most of the feedback that comes out of peer review. But editors of journals also want to know whether an article is innovative, original, smart, well-written. etc. Quite simply, they want to know whether the article is something that they should publish. The same is true for search committees, who want to know who to hire, or tenure committees, who want to know whether to give someone a job for life.
I’m not saying that the current peer-review system is perfect, just that there is a point to anonymity. It may lend itself to abuses, but it encourages a certain kind of candor as well.