Disambiguating post-publication peer review

NOTE: OpenAIRE would like to know what you think about open peer review! Have your say here until 7th October! 

Tl;dr – “Post-publication peer review” (PPPR) has gained a lot of traction in recent years. As with much of peer review’s confusing lexicon, however, this term is ambiguous. This ambiguity stems from confusion over what constitutes “publication” in the digital age. PPPR conflates two distinct phenomena, which we would do better to treat separately, namely “open pre-review manuscripts” and “open final-version commenting”.

What is “post-publication peer review”?

CC BY AC McCann, with modifications

Peer review can have two senses, one specific and the other more general. “Peer Review” (henceforth PR) is a well-defined publishing practice for the quality assurance of research articles and other academic outputs. It is intimately tied to the publication process. It traditionally begins when an editor sends a manuscript to reviewers and ends when the editor accepts a manuscript for publication. But “peer review” (lower-case, henceforth “pr”) is just the critique and appraisal of ideas, theories, and findings by those with particular insight into a topic. Such feedback happens all the time. It happens before manuscripts are submitted: in colleagues’ initial reactions (positive or negative) to a new idea, feedback gained from conferences, lectures, seminars and late-night bull sessions, or private comments on late-stage first-draft manuscripts from trusted peers. And it continues after the article’s appearance in a journal, via a multitude of channels through which readers can give feedback, including comment sections on journal websites, dedicated channels for post-publication commentary, blogs and social media, and of course in future research that cites and comments back on the findings.

A major difference between pr and PR is therefore temporal: the former happens all the time (or at least from the initial formation of ideas, theories and hypotheses), while the latter is a well-defined process that occurs precisely in that time between manuscript submission and article publication.

Post-publication peer review tries to bridge this disconnect, to incorporate elements of pr back into PR in order to open research up to wider scrutiny and serendipity, by messing with this standard temporal order of classical peer review (submission, review, publication). But the term is used to signify two distinct ways in which this can be done, namely what I’ll here call “open pre-review manuscripts” (opening manuscripts to the public before or in synchrony with peer review) and “open final-version commenting” (ongoing evaluation of “final” version-of-record publications). The conflation is not helpful, since the two phenomena actually have quite distinct purposes.

PPPR Type 1: Open pre-review manuscripts

Subject specific “preprint servers” like arXiv.org and bioRxiv.org, institutional repositories, catch-all repositories like Zenodo.org or Figshare.com and some publisher-hosted repositories (like PeerJ Preprints) allow authors to short-cut the traditional publication process and make their manuscripts immediately available to everyone. This can be used as a complement to a more traditional publication process, with comments invited on preprints and then incorporated into redrafting as the manuscript goes through traditional peer review with a journal. Alternatively, services which overlay peer-review functionalities on repositories can produce functional publication platforms at reduced cost (Boldt, 2011; Perakakis et al., 2010). The mathematics journal Discrete Analysis, for example, is an overlay journal whose primary content is hosted on the arXiv (Gowers, 2015). The recently released Open Peer Review Module for repositories, developed by Open Scholar in association with OpenAIRE, is an open source software plug-in which adds overlay peer review functionalities to repositories using the DSpace software (Open Scholar, 2016). Another innovative model along these lines is that of Science Open, which ingests articles metadata from preprint servers, contextualizes them by adding altmetrics and other relational information, before offering authors peer review.

In other cases manuscripts are submitted to publishers in the usual way, but made immediately available online (usually following some rapid preliminary review or “sanity check”) before the start of the peer review process. This approach was pioneered with the 1997 launch of the online journal Electronic Transactions in Artificial Intelligence (ETAI), where a two-stage review process was used. First, manuscripts were made available online for interactive community discussion, before later being subject to standard anonymous peer review. The journal stopped publishing in 2002 (Sandewall, 2012). Atmospheric Chemistry and Physics  uses a similar system of multi-stage peer review, with manuscripts being made immediately available as “discussion papers” for community comments and peer review (Pöschl, 2012). Other prominent examples are F1000Research and the Semantic Web Journal .

The benefits to be gained from open pre-review manuscripts is that researchers can assert their priority in reporting findings – they needn’t wait for the sometimes seemingly endless peer review and publishing process, during which they live in constant fear of being scooped. Moreover, getting research out earlier increases its visibility, enables open participation in peer review (where commentary is open to all), and perhaps even, according to (Pöschl, 2012), increases the quality of initial manuscript submissions.

PPPR Type 2: Open final-version commenting

If the purpose of peer review is to assist in the selection and improvement of manuscripts for publication, then it seems illogical to suggest that peer review can continue once the final version-of-record is made public. Nonetheless, in a literal sense, even the declared fixed version-of-record continues to undergo a process of improvement (occasionally) and selection (perpetually).

As with most areas of communication, the Internet has hugely expanded the range of effective action available for readers to offer their feedback on scholarly works. Where before only formal routes like the letters to the journal or commentary articles offered readers a voice, now a multitude of channels exist. Journals are increasingly offering their own commentary sections. Walker and Rocha da Silva found that of 53 publishing venues reviewed, 24 provided facilities to enable user-comments on published articles – although these were typically not heavily used (Walker and Rocha da Silva, 2015). But users can “publish” their thoughts anywhere on the Web – via academic social networks like Mendeley, ResearchGate and Academia.edu, via Twitter, or on their own blogs. The reputation of a work hence undergoes continuous evolution as long as it remains the subject of discussion.

Improvements based on feedback happen most obviously in the case of so-called ‘living’ publications like the Living Reviews group of three disciplinary journals in the fields of relativity, solar physics and computational astrophysics, publishing invited review articles which allow authors to regularly update their articles to incorporate the latest developments in the field. But even where the published version is anticipated to be the final version, it remains open to future retraction or correction. These days such changes are fuelled by social media, as in the 2010 case of #arseniclife, where social media critique over flaws in the methodology of a paper claiming to show a bacterium capable of growing on arsenic resulted in retraction refutations being published in Science. The Retraction Watch blog is dedicated to publicising such cases.

A major influence here has been the independent platform Pubpeer, which  is proud to offer itself as a “post-publication peer review platform”. When its users swarmed to critique a Nature paper on STAP (Stimulus-Triggered Acqusition of Pluripotency) cells, PubPeer argued that its “post-publication peer review easily outperformed even the most careful reviewing in the best journal. The papers’ comment threads on PubPeer have attracted some 40000 viewers. It’s hardly suprising [sic] they caught issues that three overworked referees and a couple of editors did not. Science is now able to self-correct instantly. Post-publication peer review is here to stay” (PubPeer, 2014).

We need better words …

Open pre-review manuscripts and open final-version commenting are distinct phenomena. Open pre-review manuscripts help establish priority and get findings out early. Open final-version commenting, on the other hand, helps filter published articles via peer to peer recommendation and acts as a final watchdog to ensure that false findings can be corrected.

That the two get bundled together under the term PPPR is probably an artefact of confusion about the meaning of the legacy term “publishing”. When is a digital manuscript “published”?  Is it just when it is first made “public”, including in that still malleable form that we call “pre-prints”? Or is “publication” that moment when a (presumed) “final” version of record appears, after peer-review, copy-editing and the rubber-stamp of the journal brand have been applied?

As Cameron Neylon has said in this context, #weneedbetterwordsforthis. That publishing is a process and not an event is, I think, well understood. But perhaps it is now time for more serious recognition, also in the words we use, that “publications” are also processes. On this latter view, the term “post-publication peer review” is a misnomer not just because it is ambiguous in conflating two distinct phenomena, but rather because it really makes no sense to talk about either post- or pre-publication in such fixed terms. What we call “publications” are not fixed artefacts, but are always in a continual state of becoming, as initial ideas bloom and research takes shape, as arguments are sharpened and honed, and research outputs find their place in the scientific conversation of mankind. The legacy language of the analogue print world seems insufficient in this regard. Which words would be better? That’s another conversation (and this post is already long enough!), but as a start I’d argue in favour of ones which talk of growth, maturity and stability …

Refs

Boldt, A., 2011. Extending ArXiv.org to achieve open peer review and publishing. J. Sch. Publ. 42, 238–242.

Gowers, T., 2015. Discrete Analysis — an arXiv overlay journal. Gowerss Weblog.

Open Scholar, 2016. Open access repositories start to offer overlay peer review services [WWW Document]. Open Sch. CIC. URL http://www.openscholar.org.uk/institutional-repositories-start-to-offer-peer-review-services/ (accessed 8.25.16).

Perakakis, P., Taylor, M., Mazza, M., Trachana, V., 2010.Natural selection of academic papers.Scientometrics 85, 553–559. doi:10.1007/s11192-010-0253-1

Pöschl, U., 2012. Multi-stage open peer review: scientific evaluation integrating the strengths of traditional peer review with the virtues of transparency and self-regulation. Front. Comput.Neurosci.6, 33. doi:10.3389/fncom.2012.00033

PubPeer, 2014. Science self-corrects – instantly. PubPeer Online J. Club.

Sandewall, E., 2012. Maintaining Live Discussion in Two-Stage Open Peer Review. Front. Comput.Neurosci. 6. doi:10.3389/fncom.2012.00009

Walker, R., Rocha da Silva, P., 2015. Emerging trends in peer review – a survey. Front. Neurosci. 9. doi:10.3389/fnins.2015.00169

 

 

 

Tony Ross-Hellauer

OpenAIRE2020 Scientific Manager at Göttingen State and University Library, University of Göttingen. Email: ross-hellauer@sub.uni-goettingen.de

More Posts - Twitter - LinkedIn

Tags:

7 Replies

Trackback  •  Comments RSS

Sites That Link to this Post

  1. Disambiguating post-publication peer review – ScienceOpen Blog | 2016-09-15
  1. Tony Ross-Hellauer says:

    NOTE: I originally said that the arsenic life paper was retracted, but it never actually has been. This has now been updated. Thanks to Ivan Oransky for pointing out the mistake!

  2. Richard Fisher says:

    Tony

    This is an important and helpful distinction. Please don’t assume, however, that all disciplines share this sense that fluid is necessarily good…by which I mean that in (say) classics or medieval studies the presumption remains widespread that a published article or (especially) book is indeed, and rightly, ‘fixed’, because that defined state gives the next scholar something definite to argue against (or for), and to reference as a defined and closed entity. In these worlds, where the expectation is that emergent scholarship will still be used and cited one hundred years hence, publication remains both a process and an event. And don’t forget that the (very large) majority of long-form research in these major disciplinary sectors is still consumed in print form, a trend that shows no sign whatsoever of going away…

    • Tony Ross-Hellauer says:

      Richard, Many thanks for this excellent reminder of the differences in research cultures! You’re right of course – what we call “publication” is both a process and state-changes along the way, such as the production of a stable version-of-record, are events!

  3. Readers may find my ideas useful (and hopefully assist with the disambiguation). In essence, anything that appears publicly in any format, whether journal or blog, pre-print or not, is “published”. Because it owns a physical home. There is no future of science publishing without post-pub peer review (PPPR). Some of my own ideas on this topic for readers who may be interested:
    Teixeira da Silva, J.A. (2016) The importance of the anonymous voice in post-publication peer review. Journal of the Association for Information Science and Technology 67(1): 243.
    http://onlinelibrary.wiley.com/doi/10.1002/asi.23588/abstract
    DOI: 10.1002/asi.23588
    Teixeira da Silva, J.A. (2015) For whom the bell tolls: downstream effects of retractions and the bump-on effects of post-publication peer review. International Journal of Plant Biology & Research 3(4): 1050.
    http://www.jscimedcentral.com/PlantBiology/plantbiology-3-1050.pdf
    Teixeira da Silva, J.A., Dobránszki, J. (2015) The role of the anonymous voice in post-publication peer review versus traditional peer review. KOME 3(2): 90-94.
    http://komejournal.com/files/KOME_Silva-Dobranszki.pdf
    DOI: 10.17646/KOME.2015.27
    Teixeira da Silva, J.A. (2015) Fair use in post-publication peer review. Journal of Educational and Social Research 5(3): 13.
    http://www.mcser.org/journal/index.php/jesr/article/view/7694
    DOI: 10.5901/jesr.2015.v5n3p13
    Teixeira da Silva, J.A. (2015) Debunking post-publication peer review. International Journal of Education and Information Technology (Public Science Framework) 1(2): 34-37.
    http://files.aiscience.org/journal/article/html/70390007.html
    Teixeira da Silva, J.A., Dobránszki, J. (2015) Problems with traditional science publishing and finding a wider niche for post-publication peer review. Accountability in Research: Policies and Quality Assurance 22(1): 22-40.
    http://www.tandfonline.com/doi/full/10.1080/08989621.2014.899909#.VJXPV0oBg
    DOI: 10.1080/08989621.2014.899909
    Teixeira da Silva, J.A. (2014) Postpublication peer review in plant science. Science Editor (Council of Science Editors) 37(2): 57+59.
    http://www.councilscienceeditors.org/wp-content/uploads/v37n2p57_59.pdf
    Teixeira da Silva, J.A. (2014) Recent retraction cases in plant science that show why post-publication peer review is essential. Journal of Advancement in Engineering and Technology 1(3): 4 pp.
    http://scienceq.org/archive.php?jname=aet&jid=aet0614333&tit=#.VJXOW0oBg http://scienceq.org/Uploaded/Editorial/532430892.pdf
    DOI: 10.15297/JAET.V1I3.03
    Teixeira da Silva, J.A. (2013) The need for post-publication peer review in plant science publishing. Frontiers in Plant Science 4: Article 485, 3 pp.
    http://journal.frontiersin.org/Journal/10.3389/fpls.2013.00485/full
    DOI: 10.3389/fpls.2013.00485

  4. Chris says:

    I think of the publication system as supporting the scientific debate, which is a central part of the scientific method. So a publication is a contribution to a discussion, as are the comments on that pub. Call it a virtual conference. Sometimes people will talk about unpublished preprints, other times about a paper recently appeared in a journal. Essential is the forum, and how well it supports the debate.
    We don’t have many good places for debate, I think. So the social and self-correcting part of science needs better support.
    More support for open, transparent, archived on-line discussion linked with papers/preprints. Especially, we need an archive/’copy of record’ for debate contributions, and a central hub/aggregtor for the scattered discussions in several places. Importantly, this infrastructure should be run by the community according to scientific criteria.

    • Tony Ross-Hellauer says:

      Great comment Chris – I couldn’t agree more. I especially like your point about needing aggreagators/hubs to bring together all the individual comments around particular research objects. The kinds of infrastructure that this requires are beginning to come together (e.g.,
      altmetric aggregation, Crossref’s DOI tracker, etc.), but still need more work. One thing I’d love to see is a schema for the federation of review content, so that the review could be hosted anywhere, but pulled across to aggregating platforms dynamically as needed (e.g., the users requests the reviews). The peer review CASRAI working group headed by F1000 and ORCID was a step in this direction, but there’s lots more to do!

Post a Reply

Your email address will not be published. Required fields are marked *

Top