Shakespeare Quarterly is one of the most recent scholarly journals to begin experimenting with new models for peer review…
For Shakespeare critics and scholars, among the most significant consequences of media change will be transformations in how we communicate with each other about our work and publish new research. In keeping with the topic of its special issue, 61:4, “Shakespeare and New Media,” Shakespeare Quarterly conducted an experiment in open peer review, for this issue, which ran from 1o March to 5 May 2010…..
The Process: After the initial editorial evaluation, authors were invited to opt into the open review process. The essays of those who opted in were posted here for public commentary and feedback by the journal’s readers. Authors have been invited to respond to this feedback in revision, before submitting their revised essays for final selection. The publication decision was based on the revised essays. (Declining the open review and opting for a traditional review would not negatively affect the selection process — it simply established a different review path.) For further details see “About” and FAQs.
Rothwell, Peter M., and Christopher N. Martyn. “Reproducibility of peer review in clinical neuroscience: Is agreement between reviewers any greater than would be expected by chance alone?.” Brain 123, no. 9 (September 1, 2000): 1964-1969. http://brain.oxfordjournals.org/cgi/content/full/123/9/1964
Abstract: We aimed to determine the reproducibility of assessments made by independent reviewers of papers submitted for publication to clinical neuroscience journals and abstracts submitted for presentation at clinical neuroscience conferences. We studied two journals in which manuscripts were routinely assessed by two reviewers, and two conferences in which abstracts were routinely scored by multiple reviewers. Agreement between the reviewers as to whether manuscripts should be accepted, revised or rejected was not significantly greater than that expected by chance [ = 0.08, 95% confidence interval (CI) –0.04 to –0.20] for 179 consecutive papers submitted to Journal A, and was poor ( = 0.28, 0.12 to 0.40) for 116 papers submitted to Journal B. However, editors were very much more likely to publish papers when both reviewers recommended acceptance than when they disagreed or recommended rejection (Journal A, odds ratio = 73, 95% CI = 27 to 200; Journal B, 51, 17 to 155). There was little or no agreement between the reviewers as to the priority (low, medium, or high) for publication (Journal A, = –0.12, 95% CI –0.30 to –0.11; Journal B, = 0.27, 0.01 to 0.53). Abstracts submitted for presentation at the conferences were given a score of 1 (poor) to 6 (excellent) by multiple independent reviewers. For each conference, analysis of variance of the scores given to abstracts revealed that differences between individual abstracts accounted for only 10–20% of the total variance of the scores. Thus, although recommendations made by reviewers have considerable influence on the fate of both papers submitted to journals and abstracts submitted to conferences, agreement between reviewers in clinical neuroscience was little greater than would be expected by chance alone.
Salo, Dorothea. “Who owns our work?” Serials 23:3, 2010.
abstract: Much turmoil in the scholarly-communication ecosystem appears to revolve around simple ownership of intellectual property. Unpacking that notion, however, produces a fascinating tangle of stakeholders, desires, products and struggles. Some products of the research process, especially novel ones, are difficult to fit into legal concepts of ownership. As collaborative research burgeons, traditional ownership and authorship criteria are stretched to their limits and beyond, with many contributors still feeling short of due credit. The desire for access and impact brings institutions and grant funders into the formerly exclusive relationship between authors and publishers. Librarians, stripped of first-sale rights by electronic licensing, wonder about both access and long-term preservation. Emerging solutions to many of these difficulties threaten to cut publishers out of the picture altogether, perhaps a welcome change to those stakeholders who find publishers’ behavior to block progress.
Cohen, D. 2010. The Idols of Scholarly Publishing. Connexions, May 14, 2010.
Kenneth M. Price’s candid appraisal of the successes and struggles of two digital projects highlighted for me several core tensions confronting scholarly publishing in a digital age. The first issue is the difficulty of drawing boundaries around editorial projects such as the Walt Whitman Archive or the Civil War Washington Project. Regardless of whether they start out with a comprehensive editorial plan, as with the Whitman Archive, or with more modest initial goals and a path toward eclectic expansion in the “always in beta” mode common on the web, as with Civil War Washington, these productions inevitably encounter hard choices of inclusion and exclusion, with the constant worry about too-narrow parameters. Second comes the question of validation. How can the Civil War Washington project receive the same respect and credit given to more traditional forms such as the print monograph? Third, how can all such digital projects sustain themselves?
I too have encountered these issues repeatedly, and so have a shorthand for each, horribly misappropriated from Francis Bacon: the Idols of the Cave, the Idols of the Tribe, and the Idols of the Marketplace. In online publishing, the Idols of the Cave entail a failure of digital collections to look outside of themselves; the Idols of the Tribe, the failure of scholarly communities to think beyond established modes of publication and associated reputational analysis; the Idols of the Marketplace, the difficulty of envisioning sustainable financial models that also promote core academic values.
These are all issues, in the spirit of Bacon, stemming from inflexible mental states—the difficulty of changing perception clouded by tradition, culture, and human nature. To smash these idols, it is helpful to look at the structures of the vernacular web and how they might inform the composition of academic digital work, rather than the reverse: constantly trying to impose academic structures on the medium. Without understanding what makes the vernacular web effective and powerful, academics working online will fail to use it well and our projects will look less like innovations and more like awkward hangovers from a disappearing past. Price, like the rest of us, is struggling to emerge from that past.
The Idols of the Cave
In his black comedy The Thought Gang, the novelist Tibor Fischer writes of a lackadaisical, failed academic philosopher named Coffin who livens up his dismal existence by going on a bank-robbing spree.1 Fischer perfectly encapsulates Coffin’s inadequacy when he notes the philosopher’s choice of specialization: the Ionian philosophers. Why the Ionians? Because the miniscule fragments of their extant thought can be contained in a rather small edited series that one can read in short order. Nothing better for the lazy academic.
Beyond the Presocratics, however, most primary sources are not so easily contained, and all efforts to encapsulate them in edited volumes necessarily fail to capture their entirety. Price lays out this issue well. Is “Whitman” just what he wrote, or what others wrote to him or about him? What about where he lived and what he lived through? What does a thorough description of a city in a place and time include or exclude? Once liberated from the physical constraints of print editions, this concern intensifies enormously. The academic propensity to look for specific value in documents, and thus their inclusion or exclusion from a collection, is confounded by the possibility of including it all online.
If the low cost of digital storage encourages inclusiveness in documentary evidence, however, the nature of the network exerts a countervailing pressure—or at least it should. Living in the world of the Norton Anthologies and multi-decade Papers of… editions we have been conditioned to fret about exclusion and inclusion, but the Internet has taken an entirely different view of the matter. Tim Berners-Lee, building on the network, assumed that to encompass a topic completely you would have to rely on the electronic synthesis of hyperlinked resources—decentralization over centralization. Everything is intertwined on the web, and all boundaries are permeable. The sheer ease of linking and the machine-readable nature of digital collections allows websites to be satisfied with incompleteness, since a viewer can jump elsewhere for complementary content, additional information, or services.
We should therefore take inspiration not from the editorial projects of print but from this core characteristic of the web: network-scale systems. The idea of a strongly bounded editorial series in the digital edge is folly. Instead, we should be pushing toward network-scale scholarship and curation. The Internet is very good at combining resources and services spread across the network and, if done correctly, this can enhance the natural research process while allaying the fear of digital collection providers that they have failed to be comprehensive. When the University of Michigan Library, the Cornell University Library, and the State and University Library Göttingen implemented a common search layer on top of their scans of rare historical works on mathematics to create the Distributed Digital Library of Mathematical Monographs, they were thinking at network-scale rather than within their own cave. When Price’s Whitman Archive offers its DTD for download so that other collections can encode the works of contemporaries in complementary ways, they are thinking at network-scale. When digital collections use APIs, OAI-PMH, and Creative Commons licenses, they are thinking at network-scale.
The vernacular web has become increasingly savvy about how networks work. For instance, the simplicity and availability (from its conception) of Twitter’s APIs and the liberation of its content from the Twitter.com domain has successfully encouraged the development of a tremendous ecosystem around the service. The rather shrewd assumption made by the founders of Twitter is that others will have a better sense of what to do with their content than they will. As much as it may pain us, academics could learn a thing or two from the oft-maligned Twitter.
The interrelations enabled by APIs and open data undoubtedly offend those with traditional views of authorship and edited collections. In his crowd-pleasing keynote to the gathered booksellers at the 2006 Book Expo convention in Washington, John Updike famously ridiculed Wired editor Kevin Kelly’s notion of a universal digital library (in Kelly’s New York Times Magazine article “Scan This Book!”, itself a transparent attempt to provoke) as full of “teeming, promiscuous word snippets stripped of credited authorship,” and stirringly implored his audience to “defend your lonely forts [and] keep your edges dry.”2
Upon reflection, most academic researchers have always disrespected those edges. Perhaps on occasion the literary scholar might receive as much insight as needed from a single poem or novel, or a historian of the ancient world a complete understanding from a single account—or a philosopher a new idea from an Ionian fragment—but for most scholars new knowledge is created through an expansive exercise of gathering texts, images, and other evidence from a variety of sources and then subjecting that ad hoc collection to an interpretive and synthetic process. For us the edges are never dry—we create our own soup of books, articles, maps, data—and in turn our works inspire, and are blended with, other sources.
This is not the loss of authorship or of canonical, bounded collections. Harold Bloom’s sense of genius need not feel threatened. But the act of immersive, singular-work obsession is only one mode of academic attention, often accompanied by other, more promiscuous behaviors. Updike seems to think that the two are polar opposites, but surely both are practiced by most scholars. Indeed, in most research processes shallow scanning leads to immersive reading, and immersive reading to new methodologies and new combinations of ideas and documents. Stewards of digital academic resources should strive to enable these processes maximally, without the assumption that viewers will spend their entire time on one site.
The Idols of the Tribe
When Roy Rosenzweig and I finished writing a full draft of Digital History, we sat down at a table and looked at the stack of printouts. “So, what now?” I said to Roy naively. “Couldn’t we just publish what we have on the web with the click of a button? What value does the gap between this stack and the finished product have? Isn’t it ninety-five percent done? What’s the last five percent for?” We stared at the stack some more. Roy finally broke the silence, explaining the magic of the last stage of scholarly production between the final draft and the published book: “What happens now is the creation of the social contract between the authors and the readers. We agree to spend considerable time ridding the manuscript of minor errors, and the press spends additional time on other corrections and layout, and readers respond to these signals—a lack of typos, nicely formatted footnotes, a bibliography, specialized fonts, and a high-quality physical presentation—by agreeing to give the book a serious read.”
I have frequently replayed that conversation in my mind, wondering about the constitution of this social contract in scholarly communication, which is deeply related to questions of academic value. Price’s article sparked thinking about that contract again, when he wondered what was necessary to get Civil War Washington the same respect and credit afforded non-digital work.
For the ease of conversation, let’s call the two sides of the social contract of scholarly publishing the supply side and the demand side. The supply side is the creation of scholarly works, including writing, peer review, editing, and the form of publication. The demand side is much more elusive—the mental state of the audience that leads them to “buy” what the supply side has produced. In order for the social contract to work, for engaged reading to happen and for credit to be given to the author (or editor of a scholarly collection), both sides need to be aligned properly.
The social contract of the book is deeply entrenched and powerful, especially in the humanities. It is one of the most totemic idols of our tribe. As John Updike put it in his diatribe against the digital (and most humanities scholars and tenure committees would still agree), “The printed, bound and paid-for book was—still is, for the moment—more exacting, more demanding, of its producer and consumer both. It is the site of an encounter, in silence, of two minds, one following in the other’s steps but invited to imagine, to argue, to concur on a level of reflection beyond that of personal encounter, with all its merely social conventions, its merciful padding of blather and mutual forgiveness.”3
As academic projects have moved to the web over the past two decades we have seen great experimentation on the supply side, as the projects highlighted in this conference show. Robust academic work has been re-envisioned in many ways: as topical portals, interactive maps, deep textual databases, new kinds of presses, and primary source collections. Most of these projects strive, like Civil War Washington, to reproduce the magic of the traditional social contract of the book, even as they experiment with form.
The demand side, however, has languished. Far fewer efforts have been made to change the demand side, to influence the mental state of the scholarly audience. The unspoken assumption is that the reader is more or less unchangeable in this respect, only able to respond to and validate works that have the traditional marks of the social contract: a strong filtering process, near-perfect copyediting, authorial control, the imprimatur of a press.
We need to work more on the demand side if we want to move the social contract forward into the digital age. Despite Updike’s ode to the book, there are social conventions surrounding print that are worth challenging. Much of the reputational analysis that occurs in the professional humanities relies on cues beyond the scholarly content itself. The act of scanning a CV is an act fraught with these Idols of the Tribe.
Can we change the views of humanities scholars so that they may accept, as some legal scholars already do, the great blog post as being as influential as the great law review article? Can we get humanities faculty, as many tenured economists already do, to publish more in open access journals? Can we accomplish the humanities equivalent of FiveThirtyEight.com, which provides as good, if not better, in-depth political analysis than most newspapers, earning the grudging respect of journalists and political theorists? Can we get our colleagues to recognize outstanding academic work wherever and however it is published?
I believe that to do so, we may have to think less like humanities scholars and more like social scientists. Behavioral economists know that although the perception of value can come from the intrinsic worth of the good itself (e.g., the quality of a wine, already rather subjective), it is often influenced by many other factors, such as price and packaging (the wine bottle, how the wine is presented for tasting). These elements trigger a reaction based on stereotypes—if it’s expensive and looks well-wrapped, it must be valuable. The book and article have an abundance of these value triggers from generations of use, but we are just beginning to understand equivalent value triggers online—thus the critical importance of web design, and why the logo of a trusted institution or a university press can still matter greatly, even if it appears on a website rather than a book.
Social psychologists have also thought deeply about the potent grip of these Idols of the Tribe. They are aware of how cultural norms establish and propagate themselves, and tell us how the imposition of limits creates hierarchies of recognition. Thinking in their way, along with the way the web works, one potential solution on the demand side might come not from the scarcity of production, as it did in a print world, but from the scarcity of attention. That is, value will be perceived in any community-accepted process that narrows the seemingly limitless texts to read or websites to view. Again, curation becomes more important than publication once publication ceases to be limited.
The Idols of the Marketplace
Price’s frank discussion of sustaining digital projects like the Whitman Archive and Civil War Washington shows two paths to success. The former has had a strong series of funding and is building an endowment; the latter, sweat equity and a good home in an established digital humanities center. Similarly,I have become increasingly convinced that sustainability is most likely at the high and low ends of digital project cost structures, a conclusion also reached by Ithaka in two recent reports on sustaining digital resources.4
On the low end, advances in creating, hosting, and maintaining a website have come down sharply over the last decade. Even if Price did not have access to servers and bandwidth at the University of Nebraska-Lincoln, he and his colleagues working on the site could run it for a few dollars a month and their labor. The advent of high-quality, open-source content management systems such as Drupal, WordPress, and (if I may) Omeka, all of which include plugins for once-complex elements such as maps as well as themes that make them look professionally designed, make it far easier and less costly than it once was to produce a scholarly website.
Of course, as Price notes, there is no free lunch. Many “low cost” digital projects are hosted at digital humanities centers that spread costs (such as running a server) over many projects. The time given by faculty and technical staff—often at the margins of their “regular” work—add up to significant hidden costs that can become troubling if staff move on or the center loses overall funding. On the other hand, the act of freely giving labor is a time-honored way of supporting scholarship. We already give our free labor for the peer review of books and articles. Ithaka also points to “in-kind support from host institutions,” “outsourcing and partnerships,” and “harnessing volunteer efforts” (evidently under consideration by Price and the Civil War Washington Project) as ways to minimize direct costs and do it on the cheap.5
Some projects, however, cannot be supported entirely by these low-cost measures. Perhaps they have a scale of hundreds of thousands or millions of documents, or technical infrastructures that move beyond the garden-variety website. What about ambitious digital projects that do not have access to the purses available to large editorial projects like the Papers of George Washington? What if Civil War Washington grows, through sheer grit and gift labor, into a critical resource for historians, literary scholars, and other researchers? What if the project needs a small, dedicated staff or a more substantial infrastructure (with its associated costs)?
The strategic planners of Ithaka point to value creation as the key: “In our opinion, delivering impact is the key factor in the potential for achieving long-term sustainability; only high impact and highly useful materials will draw the financial support from beneficiaries needed for long-term success.”6 I agree, but I once posited this idea to a venture capitalist. “Sustainable business models don’t come from the value of goods,” he replied matter-of-factly. “They come from the scarcity of goods that have value.” Although I am a strong supporter of open access and open source, we have clearly set ourselves a nearly Sisyphean task by pushing the free and open while prospecting around for revenue. Most of the literature about sustainability fails to note that most commercial enterprises fail, even with the tremendous advantage of charging for their products. Moreover, while the idea of achieving impact and then a business model sounds good, we need to recognize that much of humanities scholarship is about the esoteric. The universe of stakeholders for some fields may be miniscule, making the ability to garner donations or have libraries subscribe to every academic resource, especially in a recession, even more implausible.
The role of funders is often compared to venture capitalists. But the comparison is imperfect at best. Venture capitalists may indeed, like funders of open academic resources, provide “seed funding,” but VCs also often put in additional capital, in increasingly large amounts (i.e., Series A, B, and C funding), to support sustained operations and growth. They also provide critical connections for business deals, marketing, and IPOs, sometimes orchestrating an acquisition themselves. There are no equivalents to these injections of capital, services, or “exits” in the digital humanities. Ithaka does encourage mergers and acquisitions among digital academic resources, but the incentive structure of the university is perversely opposite that of commercial entrepreneurs: if your project is “acquired,” not only do you lose oversight of the resource, you also lose credit for it in the long term. Worse: you’re not a sudden millionaire.
At an effective rate of at least $1 million per staffer, endowments are unrealistic for most projects, and tilt the digital humanities playing field unhealthily toward institutions with established fundraising prowess. (I was delighted to see, however, that the Whitman Archive had achieved at least an initial fundraising goal to support editorial functions.) Advertising and corporate sponsorships, also proposed by Ithaka, can cloud the all-important perception of the resource I have already explored. By far the most likely path to fiscal success is the way that venture capitalist suggested: make your resource scarce and charge for it. That is, sacrifice the ideal of open access on the altar of sustainability. We need more discussion about this tension, and creative ways out of it.
A related question is the leadership of these projects. As one Ithaka report notes, “Running a start-up”—what these bigger digital projects effectively become—“is a full-time job and requires full-time leadership. The mode of principal investigators, in which they divide their time between overseeing a variety of research grants, teaching courses, and other responsibilities, is not conducive to entrepreneurial success.”7 The report is indeed correct that PIs need to operate like CEOs, not professors, being unabashedly competitive with other resources and sublimating weak academic thoughtfulness in the hardheaded effort to find revenue. The report recommends that if professors can’t do this, they need to hire others. But where do these magical entrepreneurs-in-residence come from, and who maintains them between projects? I happen to know several at the Center for History and New Media, critical research faculty with the technical know-how, the intellect, and the energy to make complex digital projects succeed. Sustaining them is, unsurprisingly, my top priority.
- Tibor Fischer, The Thought Gang (New York: The New Press, 1995).
- An adaptation of John Updike’s talk, “The End of Authorship,” was published in the New York Times on June 25, 2006, in the Times Book Review, and is available (ironically) at http://www.nytimes.com/2006/06/25/books/review/25updike.html. The podcast (again, ironically) of his Book Expo talk (June 23, 2006) is available at http://bookexpocast.com/2006/05/26/bea-2-john-updike-speech/ . Kevin Kelly’s “Scan This Book!”, New York Times Magazine, May 14, 2006, is available unironically at http://www.nytimes.com/2006/05/14/magazine/14publishing.html .
- Updike, “The End of Authorship.”
- Nancy L. Maron, K. Kirby Smith, and Matthew Loy, “Sustaining Digital Resources: An On-the-Ground View of Projects Today” (Ithaka Case Studies in Sustainability, July 2009), available at http://www.ithaka.org/ithaka-s-r/strategy/ithaka-case-studies-in-sustainability ; Kevin Guthrie, Rebecca Griffiths, and Nancy Maron, “Sustainability and Revenue Models for Online Academic Resources” (Strategic Content Alliance/Ithaka, May 2008), available at http://www.ithaka.org/strategic-services/sca_ithaka_sustainability_report-final.pdf .
- Maron et al., “Sustaining Digital Resources,” 17-20.
- Guthrie et al., “Sustainability and Revenue Models,” 5.
- Guthrie et al., “Sustainability and Revenue Models,” 7.The author would like to thank Tom Scheinfeldt for his helpful comments on a draft of this paper.
Peer Review in Academic Promotion and Publishing: Its Meaning, Locus, and Future. Diane Harley, Ph.D., Principal Investigator, Sophia Krzys Acord, Ph.D., Sarah Earl-Novell, Ph.D.
PDF document (439 kB)
AbstractAs part of its Andrew W. Mellon Foundation-funded Future of Scholarly Communication Project, the Center for Studies in Higher Education (CSHE) has hosted two meetings to explore how peer review relates to scholarly communication and academic values. In preparation for an April 2010 workshop, four working papers were developed and circulated. They are presented as drafts here. (The proceedings from the April 2010 meeting will be published at a future date.) The topics covered include assessing the myriad forms peer review takes in the academy, which forms of peer review are used for which specific academic purposes (e.g., tenure and promotion, publishing, extramural funding, national and international stature), the considerable costs to universities in subsidizing the entire peer review process through faculty salaries, and the perception that, although peer review represents the best available system, there are nonetheless a multitude of problems with it, including its inherent conservatism.