Has journal commenting failed?

It’s a great idea: take all the insights, suggestions, and criticisms on scholarly articles, the comments shared in journal clubs and scribbled in margins the world over, and make them accessible to everyone. Attach them to the article itself; make it a conversation, not an artifact. We have blog commenting, video commenting–why not article commenting?

That’s sounded good to a lot of publishers, and over the last five years, we’ve seen article commenting systems become pretty popular. But there’s a growing sense that article commenting isn’t working.

The bad

Gotzsche et al. (2010) look at author replies to BMJ’srapid response” comments. We’d hope the chance to interact with authors would be a big plus for article commenting; however, they found that even when comments could “invalidate research or reduce… reliability,”  over half the time authors couldn’t be bothered to respond.

In another study, Schriger at al. (in press; thanks Bora) examine the prevalence of commenting systems in top medical journals.  They report that the percentage of journals offering rapid review has dropped from 12% in 2005 to 8% in 2009, and that fully half the journals sampled had commenting systems laying idle, completely unused by anyone. The authors conclude, “postpublication critique of articles in online pages provided by the journal does not seem to be taking hold.”

Finally, I collected data on PLoS comments as part of a larger investigation of alt-metrics. As evident from the graphic, the number articles with comments has held more or less steady as the total articles published has grown: again, not a pretty picture for those of us excited about article commenting.

The good

I’m not ready to give up on comments yet, though, because I think there’s a different way to see these findings. The question shouldn’t be “have comments failed,” but “are they succeeding somewhere, and why?”  After all, we’re still in the very early stages of this thing; change in scholarly communication so far has happened on a scale of centuries.

Active, widespread commenting would be a radical change in how scholars communicate, and as with all fundemental shifts, we can assume most early efforts will be failures. In the 1900s, way more automobile manufacturers went broke building lousy cars than flourished making good ones. So in looking at comment ecosystems, we shouldn’t be stuck ogling the crowd of inevitable false starts–we should be trying to spot the nascent Model T.

And when we do see venues where comments are disproportionately successful, we should be trying to figure out what they’re doing right. While half the sample of the Schriger et al. study are stuck without a single commented article, BMJ, CMAJ, and Ann. Intern. Med. all have comments on 50-76%. How are they different? The BMJ articles sampled by Gotzshe et al. had a mean of 4.9 responses each, which is pretty respectable. Why are these here, but not elsewhere?

In the case of PLoS, we can see that even journals from the same publisher and on the same platform show widely different commenting rates. Is it the editors, the nature of the field, or something else that’s making PLoS Biology’s comment rate climb as PLoS Genetics’ holds steady and PLoS ONE’s drops?  This is a great opportunity for research that will help commenting evolve further.

The future

So I think that while we see cases where journal commenting is beginning to succeed, we should continue to put resources behind spreading that success. This said, I have to admit I’m doubtful that publisher-hosted commenting is the future.

Today we have two scholarly communication ecosystems: the formal, peer-reviewed one, and the shadow system encompassing everything from scribbled marginalia, to chats in the lab, to peer reviews themselves. Sooner or later, I believe the shadow ecosystem will migrate to the web; a detailed argument for why is a different post, but there are too many advantages. It’ll happen. The advance guard is already conversing, learning, and collaborating on Zotero, Mendeley, CiteULike, blogs, Twitter, and so on.

Publisher-hosted article commenting is the formal system’s bid to gain a foothold in the informal system as it moves online. And it’s a smart bid, because as the shadow system sheds its ephemerality, it’s going to become increasingly important to how we measure and do scholarship.

But the problem is that journal-based comments are as siloed as the articles they comment on; there’s limited exposure, and no community. Scholars will want to have their conversations with their people, in their ways, in their places.  Today, that mostly means Twitter and blogs (as we saw in #arsenicLife); in the future, it may also be scholar-specific services like The Third Reviewer, COASPedia, or VIVO.

So while I support article commenting as it now exists, I think challenge of the future won’t be moving the shadow communication system online–it’ll be aggregating it so it can be consumed, measured, and filtered efficiently and meaningfully. I think alt-metrics will play a part in that, but again, that’s another post :)
 
 
 

References:

Here’s the dataset and R code for the PLoS graphics; I hope to be releasing the full data next week.

Gotzsche, P. C., Delamothe, T., Godlee, F., & Lundh, A. (2010). Adequacy of authors’ replies to criticism raised in electronic letters to the editor: cohort study. BMJ, 341(aug10 2), c3926-c3926. doi:10.1136/bmj.c3926

Schriger, D. L., Chehrazi, A. C., Merchant, R. M., & Altman, D. G. (In press). Use of the Internet by Print Medical Journals in 2003 to 2009: A Longitudinal Observational Study. Annals of Emergency Medicine, In Press, Corrected Proof. doi:10.1016/j.annemergmed.2010.10.008

6 Comments

  1. Americanbiotech
    Posted January 7, 2011 at 5:26 am | Permalink

    Most of the critical commenting that I’ve noticed takes place during journal club with some dialogue taking place in the editorial pages of the journal. This latter form of criticism usually involves subject matter experts and not your average “bench guy.”

    I believe that most bench folks are more focused on publishing their own material than on commenting in other’s. Most of us read papers either for ideas or to make sure that we’re not going to be scooped. Once we’re published we celebrate. Forget about responding to criticism.

    While generating comments/criticism from Joe bench guy would really be useful for advancing science, the creation of genuine dialogue will require a true culture shift.

  2. Zsolt Almási
    Posted January 7, 2011 at 10:39 am | Permalink

    Thanks for this post! Let me share my experiences with peer-review and commentary.

    Back in 2003 a colleague of mine and me had an idea of creating a community of people interested in Tudor studies, and we founded a free on-line journal which was designed to foster scholarly communication. The idea was that an issue consisted of a leader (a longish scholarly paper) which after a process of editing and previewing was sent over to readers to reflect on it which materialized in shorter articles (reflections). All these after an editorial process were published in the relevant issue. All these articles were then there out in the cyberspace to foster further commentary and discussion. Back then these ideas sounded pretty good. (http://ecolloquia.btk.ppke.hu/)

    But first of all, it was and is rather difficult to commission a leader. Then it is even more difficult to find scholars to reflect on the leader. And during the journal’s history there have not arrived comments at all. And this is not at all about having no readers, because when meeting Tudor scholars they talk about the articles there. The problem is to be resolved somewhere else.

    So your meditation actually explores the problems at the heart of our journal as well, naturally on an abstract level. Thanks for it again!

  3. Peter Binfield
    Posted January 7, 2011 at 2:37 pm | Permalink

    Re. The PLoS comments graph, don’t forget there are various external factors that mean these data.require careful clarification. For example we moved platforms for some titles in that period (and old comments got new time stamps as a result). Also the large spike in PLoS ONE comments was period when we posted reviews as a Comment. The decline reflects when we stopped doing so. Etc.

    Happy to give more background context / info if you need it just mail me.

    Pete

  4. jason
    Posted January 7, 2011 at 2:59 pm | Permalink

    Thanks Pete, good points. I agree, it’s important not to make any bold or intricate claims about commenting at PLoS based on this first pass over the rough data.

    But I do think the data we have nicely illustrate that commenting, even when controlling for the platform and publisher, seems to have be employed at different rates for different journals. And I think that represents a great opportunity to compare these situations, and inform ways to make commenting at PLoS and elsewhere more successful.

  5. Posted January 8, 2011 at 11:48 am | Permalink

    Thanks Jason – this is really interesting. I think turning the question around to ask where commenting is working and why is important.
    When you talk about the advance guard of those moving their informal discussions online, do you think we need to distinguish between commentary that is intraspecialist and that which is meant for a more public audience? I wonder if part of the desire to move these online is their more public nature (reaching to others in related specialities and beyond) and perhaps journal comments are two specialist oriented? What if they don’t fulfill the reasons that people go online to discuss and comment in the first place?
    I don’t have an answer, just throwing some ideas out there…

  6. Posted March 28, 2011 at 9:38 pm | Permalink

    Thank you for this blog post. I recommend you visit The Scholarly Kitchen’s recent blog post on the Schriger et al article you mention (http://scholarlykitchen.sspnet.org/2011/03/28/how-the-internet-changed-medical-journals/).

One Trackback

  1. [...] altmetrics home Has Journal Commenting Failed Twitter Survey Report (Sept 2010) on scribd More stats for the PLoS alt-metrics [...]

Post a Comment

Your email is never shared. Required fields are marked *

*
*