Crowdsourcing Peer Review and a Response

By Jackie Ammerman
October 25th, 2010

Mike O’Malley’s provocative essay, “Googling Peer Review,” raises interesting questions about the shift from an era of scarcity of information to an era of abundance of information that affect both scholars and  information professionals.

Ear­lier I argued that the era of scarcity in evi­dence was com­ing to a close, because so much pre­vi­ously hard to get mate­r­ial now exists online. Maybe it’s time for the era of scarcity in peer review to end as well. We ought to be able to rethink peer review in ways that make it more effec­tive and less “clubby.” (O’Malley)

He is not the first to suggest that peer review might be re-conceived in a digital context. Kathleen Fitzpatrick, who has been a leader in exploring “open peer review,” has posted a thoughtful response to O’Malley’s essay:

It’s gratifying to see other scholars getting interested in these wacky ideas about reinventing scholarly publishing that I’ve been pushing for over the last several years. In particular, the entry of scholars who are relatively new to the digital into these discussions confirms my sense that we’re at a tipping point of sorts, in which these new modes, while still experimental, are beginning to produce enough curiosity in mainstream academic circles that they’re no longer automatically dismissed out of hand.

All that said, I do feel the need to introduce a few words of caution into these discussions, because the business of open peer review isn’t quite as straightforward as simply throwing open the gates and letting Google do its thing…. (Fitzpatrick)

read the full response

UPDATE (10/26):

Mike O’Malley continues the conversation:

Peer review has not only served us badly: it’s cost aca­d­e­mics more and more cul­tural author­ity. The gen­eral pub­lic, hav­ing more sources avail­able online, is less will­ing to trust experts, and  sees peer review as akin to the mon­keys in Kipling’s Jun­gle Book: “We all say so, and so it must be true.”

Kath­leen Fitz­patrick made an excel­lent post on peer review. I highly rec­om­mend it as a deeper and more nuanced take than my ear­lier polem­i­cal version.

Fitz­patrick explains more about what Google does,  and how unclear Google is about how it ranks pages: since peer review is cen­tral to pro­mo­tion and tenure and career eval­u­a­tion, it’s dou­bly prob­lem­atic that Google hides its meth­ods. She adds that talk­ing about open sourc­ing peer review  it won’t pro­duce peo­ple will­ing to do the hard work. She’s right on all points.

But I still think a case can be made for ignor­ing a specif­i­cally aca­d­e­mic audi­ence for peer review, just ignor­ing it, and enter­ing aca­d­e­mic work in the gen­eral inter­net fray. Here’s why.

read more