It’s a great idea: take all the insights, suggestions, and criticisms on scholarly articles, the comments shared in journal clubs and scribbled in margins the world over, and make them accessible to everyone. Attach them to the article itself; make it a conversation, not an artifact. We have blog commenting, video commenting–why not article commenting?
That’s sounded good to a lot of publishers, and over the last five years, we’ve seen article commenting systems become pretty popular. But there’s a growing sense that article commenting isn’t working.
Gotzsche et al. (2010) look at author replies to BMJ’s “rapid response” comments. We’d hope the chance to interact with authors would be a big plus for article commenting; however, they found that even when comments could “invalidate research or reduce… reliability,” over half the time authors couldn’t be bothered to respond.
In another study, Schriger at al. (in press; thanks Bora) examine the prevalence of commenting systems in top medical journals. They report that the percentage of journals offering rapid review has dropped from 12% in 2005 to 8% in 2009, and that fully half the journals sampled had commenting systems laying idle, completely unused by anyone. The authors conclude, “postpublication critique of articles in online pages provided by the journal does not seem to be taking hold.”
Finally, I collected data on PLoS comments as part of a larger investigation of alt-metrics. As evident from the graphic, the number articles with comments has held more or less steady as the total articles published has grown: again, not a pretty picture for those of us excited about article commenting.
I’m not ready to give up on comments yet, though, because I think there’s a different way to see these findings. The question shouldn’t be “have comments failed,” but “are they succeeding somewhere, and why?” After all, we’re still in the very early stages of this thing; change in scholarly communication so far has happened on a scale of centuries.
Active, widespread commenting would be a radical change in how scholars communicate, and as with all fundemental shifts, we can assume most early efforts will be failures. In the 1900s, way more automobile manufacturers went broke building lousy cars than flourished making good ones. So in looking at comment ecosystems, we shouldn’t be stuck ogling the crowd of inevitable false starts–we should be trying to spot the nascent Model T.
And when we do see venues where comments are disproportionately successful, we should be trying to figure out what they’re doing right. While half the sample of the Schriger et al. study are stuck without a single commented article, BMJ, CMAJ, and Ann. Intern. Med. all have comments on 50-76%. How are they different? The BMJ articles sampled by Gotzshe et al. had a mean of 4.9 responses each, which is pretty respectable. Why are these here, but not elsewhere?
In the case of PLoS, we can see that even journals from the same publisher and on the same platform show widely different commenting rates. Is it the editors, the nature of the field, or something else that’s making PLoS Biology’s comment rate climb as PLoS Genetics’ holds steady and PLoS ONE’s drops? This is a great opportunity for research that will help commenting evolve further.
So I think that while we see cases where journal commenting is beginning to succeed, we should continue to put resources behind spreading that success. This said, I have to admit I’m doubtful that publisher-hosted commenting is the future.
Today we have two scholarly communication ecosystems: the formal, peer-reviewed one, and the shadow system encompassing everything from scribbled marginalia, to chats in the lab, to peer reviews themselves. Sooner or later, I believe the shadow ecosystem will migrate to the web; a detailed argument for why is a different post, but there are too many advantages. It’ll happen. The advance guard is already conversing, learning, and collaborating on Zotero, Mendeley, CiteULike, blogs, Twitter, and so on.
Publisher-hosted article commenting is the formal system’s bid to gain a foothold in the informal system as it moves online. And it’s a smart bid, because as the shadow system sheds its ephemerality, it’s going to become increasingly important to how we measure and do scholarship.
But the problem is that journal-based comments are as siloed as the articles they comment on; there’s limited exposure, and no community. Scholars will want to have their conversations with their people, in their ways, in their places. Today, that mostly means Twitter and blogs (as we saw in #arsenicLife); in the future, it may also be scholar-specific services like The Third Reviewer, COASPedia, or VIVO.
So while I support article commenting as it now exists, I think challenge of the future won’t be moving the shadow communication system online–it’ll be aggregating it so it can be consumed, measured, and filtered efficiently and meaningfully. I think alt-metrics will play a part in that, but again, that’s another post :)
Gotzsche, P. C., Delamothe, T., Godlee, F., & Lundh, A. (2010). Adequacy of authors’ replies to criticism raised in electronic letters to the editor: cohort study. BMJ, 341(aug10 2), c3926-c3926. doi:10.1136/bmj.c3926
Schriger, D. L., Chehrazi, A. C., Merchant, R. M., & Altman, D. G. (In press). Use of the Internet by Print Medical Journals in 2003 to 2009: A Longitudinal Observational Study. Annals of Emergency Medicine, In Press, Corrected Proof. doi:10.1016/j.annemergmed.2010.10.008