A New Kind of Peer Review ...perhaps

This from Inside Higher Ed:

Killing Peer Review

       

When a cadre of international scientific research powerhouses announced last month that they were teaming up to create a top-shelf, peer-reviewed free journal in the medical and life sciences fields, some called it a "triumph of open access" — proof that the tide was turning in favor of a once-radical movement aimed at cutting through the traditional oligarchies and turning scholarly publishing on its head.

But to Joe Pickrell, a doctoral student in biology at the University of Chicago, the idea was not groundbreaking enough. It will not do merely to lower the barriers to viewing scholarly articles, he thought; academe must lower the barriers to reviewing them.

As one might expect from an advocate of modern publishing, Pickrell took to the blogosphere. "Left unanswered … is a more fundamental question: why do we publish scientific articles in peer-reviewed journals to begin with?" the Chicago grad student wrote on Genomes Unzipped, a genetics blog he shares with other young academics. "… Cutting journals out of scientific publishing to a large extent would be unconditionally a good thing," he continued, adding that "the only thing keeping this from happening is the absence of a 'killer app.' "

Pickrell went on to describe, in general detail, the features this journal-killing app would require. It would bypass the formal peer review process, taking pre-publication papers and allowing a community of users (scholars and experts, most likely) to vote papers up or down — much like social bookmarking sites such as Reddit do for articles in the popular press. The idea would be to let readers decide which articles deserve top billing, rather than ceding that task to a tiny cloister of journal editors and their hand-picked reviewers. Papers with good feedback would shoot to the top of the list. And if scholars do want proxies to help them decide if an article is worthy of their trust and attention, they could turn to the recommendations of their friends and colleagues.

Reader comments started flowing in. Some cheered Pickrell’s post as a sort of manifesto for a rising generation of scholars. "I think this system of academic publication will continue to gain support as more people from our generation (the ones that grew up using community-oriented sites like Wikipedia, Reddit, etc.) further infiltrate academia," wrote Michael Alcorn, another biology Ph.D. candidate at Chicago. Others had not only had the same thought as Pickrell — they had actually built mock-up sites based on the principles he had described.

Still, skeptics wanted to know: In such a wild west of scholarly publishing, who would check facts? Pickrell’s answer is the same as Wikipedia’s: everybody. "I think the system could be totally self-regulating with a big enough community," he said in an interview with Inside Higher Ed. The most popular articles would receive the most attention, but they would also receive the most scrutiny. Errors are unlikely to escape a critical mass of studied readers. Mechanisms could be put in place to report errors and redact articles. (Think Wikipedia, but with original research and a specialized corps of volunteer editors.)

The challenge, Pickrell said, would be getting scholars to actively take part in the site — which, busy as they tend to be with activities that are more likely to impress a tenure committee, might be difficult. After all, one of the top journals in the life sciences, Nature, was unsuccessful in converting reader interest into actual feedback when it experimented with open peer review in 2006. One can imagine how difficult it would be to build a successful, publicly reviewed journal from scratch.

Ken Van Haren does not have to imagine. As a doctoral student at Duke University last year, Van Haren briefly tried to start a revolution on the cheap. “Despite being the ones who brought the tech revolution about, [scientific researchers] haven’t really used it for their own good,” he said. So Van Haren created Science.I/O, a website where scholars can post and vote up articles that have not been peer-reviewed. But he had no marketing budget and precious little time to spend outside his studies, and the site soon stagnated.

Still, Pickrell’s blog post tapped into a latent backlash against the traditional model that has given rise to pushes for open peer review in other fields...