That’s sounded good to a lot of publishers, and over the last five years, we’ve seen article commenting systems become pretty popular. But there’s a growing sense that article commenting isn’t working.

The bad

Gotzsche et al. (2010) look at author replies to BMJ’s “rapid response” comments. We’d hope the chance to interact with authors would be a big plus for article commenting; however, they found that even when comments could “invalidate research or reduce… reliability,”  over half the time authors couldn’t be bothered to respond.

In another study, Schriger at al. (in press; thanks Bora) examine the prevalence of commenting systems in top medical journals.  They report that the percentage of journals offering rapid review has dropped from 12% in 2005 to 8% in 2009, and that fully half the journals sampled had commenting systems laying idle, completely unused by anyone. The authors conclude, “postpublication critique of articles in online pages provided by the journal does not seem to be taking hold.”

Finally, I collected data on PLoS comments as part of a larger investigation of alt-metrics. As evident from the graphic, the number articles with comments has held more or less steady as the total articles published has grown: again, not a pretty picture for those of us excited about article commenting.

The good

I’m not ready to give up on comments yet, though, because I think there’s a different way to see these findings. The question shouldn’t be “have comments failed,” but “are they succeeding somewhere, and why?”  After all, we’re still in the very early stages of this thing; change in scholarly communication so far has happened on a scale of centuries.

Active, widespread commenting would be a radical change in how scholars communicate, and as with all fundemental shifts, we can assume most early efforts will be failures. In the 1900s, way more automobile manufacturers went broke building lousy cars than flourished making good ones. So in looking at comment ecosystems, we shouldn’t be stuck ogling the crowd of inevitable false starts–we should be trying to spot the nascent Model T.

And when we do see venues where comments are disproportionately successful, we should be trying to figure out what they’re doing right. While half the sample of the Schriger et al. study are stuck without a single commented article, BMJ, CMAJ, and Ann. Intern. Med. all have comments on 50-76%. How are they different? The BMJ articles sampled by Gotzshe et al. had a mean of 4.9 responses each, which is pretty respectable. Why are these here, but not elsewhere?

In the case of PLoS, we can see that even journals from the same publisher and on the same platform show widely different commenting rates. Is it the editors, the nature of the field, or something else that’s making PLoS Biology’s comment rate climb as PLoS Genetics’ holds steady and PLoS ONE’s drops?  This is a great opportunity for research that will help commenting evolve further.

The future

So I think that while we see cases where journal commenting is beginning to succeed, we should continue to put resources behind spreading that success. This said, I have to admit I’m doubtful that publisher-hosted commenting is the future.

Today we have two scholarly communication ecosystems: the formal, peer-reviewed one, and the shadow system encompassing everything from scribbled marginalia, to chats in the lab, to peer reviews themselves. Sooner or later, I believe the shadow ecosystem will migrate to the web; a detailed argument for why is a different post, but there are too many advantages. It’ll happen. The advance guard is already conversing, learning, and collaborating on Zotero, Mendeley, CiteULike, blogs, Twitter, and so on.

Publisher-hosted article commenting is the formal system’s bid to gain a foothold in the informal system as it moves online. And it’s a smart bid, because as the shadow system sheds its ephemerality, it’s going to become increasingly important to how we measure and do scholarship.

But the problem is that journal-based comments are as siloed as the articles they comment on; there’s limited exposure, and no community. Scholars will want to have their conversations with their people, in their ways, in their places.  Today, that mostly means Twitter and blogs (as we saw in #arsenicLife); in the future, it may also be scholar-specific services like The Third Reviewer, COASPedia, or VIVO.

So while I support article commenting as it now exists, I think challenge of the future won’t be moving the shadow communication system online–it’ll be aggregating it so it can be consumed, measured, and filtered efficiently and meaningfully. I think alt-metrics will play a part in that, but again, that’s another post :)
 
 
 

References:

Here’s the dataset and R code for the PLoS graphics; I hope to be releasing the full data next week.

Gotzsche, P. C., Delamothe, T., Godlee, F., & Lundh, A. (2010). Adequacy of authors’ replies to criticism raised in electronic letters to the editor: cohort study. BMJ, 341(aug10 2), c3926-c3926. doi:10.1136/bmj.c3926

Schriger, D. L., Chehrazi, A. C., Merchant, R. M., & Altman, D. G. (In press). Use of the Internet by Print Medical Journals in 2003 to 2009: A Longitudinal Observational Study. Annals of Emergency Medicine, In Press, Corrected Proof. doi:10.1016/j.annemergmed.2010.10.008

What’s special about these two papers is that they are the start of  a research project that I hope will become my dissertation, an idea I’m somewhat reluctantly calling “scientometrics 2.0.” (do we really need more 2.0s?) Scientometrics is

…the science of measuring and analysing science. In practice, scientometrics is often done using bibliometrics which is a measurement of the impact of (scientific) publications. (Wikipedia)

My idea is that we should be looking beyond this, and starting to mine Web 2.0 sources for signals of scholarly impact. There are a few big advantages to this approach:

1. It’s much faster.  Once a scholarly article is published, it takes a years for citations to that article to accumulate.  But it can take just days for, say, Diggs or tweets to show up: in our Twitter sample we found that nearly half the links to peer-reviewed articles appeared within a week of those articles’ publication.  This speed could be harnessed to make real-time, personal filters that inform scholars what’s groundbreaking across a broad set of fields. As the velocity and volume of science grow, this could be very valuable.
2. If I cite something, it probably had an impact in my work.  But what kind of impact?  What if I read it and talked about it, and it informed my general thinking–but not enough to cite?  Just looking at citations, we’re missing many other kinds of impact.  Ten years ago, this was the best we could do.  But today, scholars are using online tools like CiteULike, Mendeley, and Zotero to manage their libraries; Faculty of 1000 to review articles;  and Twitter, FriendFeed, and ResearchBlogging.org to discuss them.  Tools like these–and importantly, the open APIs many of them offer–allow us to lift the curtain and observe scholars in their native habitat.  Scientometrics 2.0 offers a chance for us to develop a richer, more nuanced picture of scholarly impact.
3. Finally, this approach allows us to break the centuries-old monopoly of the peer-reviewed article or monograph on scientific communication.  We can measure reactions not just to these articles, but also to blog posts, datasets, or videos.  If a certain blog post in your field is generating lots of buzz, there’s a good chance it’s worth your time.  Scientometrics 2.0 can support a sort of informal, “soft peer-review” that works for free, on everything.

At first, this approach will mostly be used for relatively “pure” academic study–learning more about how scholars communicate how impact is transmitted.  Soon, however, young scholars will start making a case to tenure and promotion committees that their heavily tweeted or bookmarked article should count in their favor. Ultimately, I think we’ll see tools that leverage this information to help direct scholars to the most important and relevant work for them, kind of a PostRank for academics.

Of course, there are some obstacles to this.  The most important one for now is getting people to trust that these alternative sources really mean anything.  Who cares if an article is tweeted a lot?  Won’t people game this?  What about scholars who don’t use social media (a majority, for now)?  These questions have answers, but they need to be taken seriously (see the articles for more detailed discussions).

Ultimately, scientometrics 2.0 is going to have to be something we investigate very carefully, and in the proper context.  However, in that context I think it has the potential to be quite valuable, and I”m excited about working toward this in the next several years.

(Note: for a bunch of relevant citations, see the first article.)

You know, we learn we remember 10% of what we read, 20% percent of what we hear, but 80% of what we actually experience.  Or, wait, maybe it’s 20%.  Or 30?

Of course, as many people know, this delightful little statistic has no backing in any sort of serious research—nor, indeed, could it:

…As Dwyer points out, the reported percentages are impossible to interpret or verify without specifying at least the method of measurement, the age of the learners, the type of learning task, and the content being remembered (p. 10).  Despite the lack of credibility, this formulation is widely quoted, usually without attribution, and in recent years has become repeatedly conflated with Dale’s Cone, with the percentage statements superimposed on the cone, replacing or supplementing Dale’s original categories.

from Cone of Experience (PDF), entry in A. Kovalchick & K. Dawson, Ed’s, Educational Technology: An Encyclopedia. Santa Barbara, CA: ABC-Clio, 2003.

Several bloggers have likewise been struck by the curious disconnect between the popularity of this statistic and its relation to reality.  Despite its readily apparent dodginess (We remember 90% of what we experience?  So I perfectly remember everything I did for nine out of the last ten years?), people love quoting this thing.

So quote they do.  And, since there’s no actual citation for this thing, the meme is free to mutate, which is actually kind of fascinating; the plot above shows the pattern in ghits for different versions of this same ‘principle.’

But why?  Obviously, the meme lives because it has value to people; in this case,  it helps folks prove a point about better ways of teaching.  But that’s not really an answer; there’s no reverse version of this for people arguing the opposite side.  No, the real answer is this: the statistic lives because it demonstrates something that the speaker and the listener both already agree on.  Few people are going to call you on this statistic, because everyone knows that the gist is true in many situations; you probably will learn something better if you involve it in some kind of experience than if you just read about it and move on.

The New York Times did a great story some years ago on related idea, called Scientific Myths That Are Too Good to Die.  It documented how well-known experiments could become sort of “academic urban myths.”  Take, for instance, the experiment that lent it’s name to the oft-cited “Hawthorne Effect” (in which the participants’ mere knowledge that they’re part of an experiment skews results):

”The results of this experiment, or rather the human relations interpretation offered by the researchers who summarized the results, soon became gospel for introductory textbooks in both psychology and management science,” said Dr. Lee Ross, a psychology professor at Stanford University.

But only five workers took part in the study, Dr. Ross said, and two were replaced partway through for gross insubordination and low output.

A psychology professor at the University of Michigan, Dr. Richard Nisbett calls the Hawthorne effect ”a glorified anecdote.”

These “glorified anecdotes” (and glorified ballpark guesses, which is really what the percentage-retention statistic is) hang on, though, because, in Dr. Ross’ words again, “‘sometimes a story deserves to be true.”  That is, the story or number itself may be wrong, but it may be a way to access a point that deserves our attention.

So, then, is a bad statistic in a good cause worthwhile?  What if my “90% retention” number gets that grumpy admin to allow my pet wiki project?  Is it worth it?  I say no, for reasons that lie outside the scope of this post (maybe next one?).  Any other opinions, though?

Quick, Google a picture of two seagulls next to a rock, with a woman in a red jacket in the foreground.  Not too easy, is it?  The problem, of course, is that images aren’t indexed by their content; while text is machine-readable (ergo machine-indexable), image indexing still requires the Mark I Eyeball.

One solution: throw automation out the window and crowdsource the tagging task to a bunch of humans.  Amazon has had success in doing this with its Mechanical Turk, which pays a small piece rate to folks in exchange for performance of “human intelligence tasks” like image-labelling.

Google, though, has been pursuing a different strategy, one I’ll call the “fence-painting technique,” after Tom Sawyer’s famous exercise in motivational psychology.  Google lets users play a game in which they try to add more tags to an image than an opponent.  Google keeps the valuable image information, and players get…um, points.  That’s right, users do a  Human Intelligence Task that they’d get paid for over at Amazon, for free.

The power of games to motivate is profound.  It’s this realization (hardly a new one) that’s fueling much of the growing interest (and debates) in educational gaming.  “If we could get Johnny to concentrate on physics the way he does on Guitar Hero…”

The trick, though, is to not stop with motivation.  Sure, there’s some value in a game that makes it fun for Suzie to memorize her multiplication facts.  But I think that educational games have a lot more to offer, particularly when we get into simulations.  While I doubt they’ll replace classrooms entirely, I think open-ended games that move beyond skill practice—”software toys,” to use the great term Will Wright coined to describe his seminal SimCity—do have transformative potential.  When I see projects like the UW‘s epistemic games, I see a lot more going on than just motivation—I see critical thinking that transfers accross the curriculum, combined with real subject-area learning.

I like Clark’s thought that games may be a qualitative leap in teaching of a kind that hasn’t been seen in a long time.  I disagree that simulations are entirely revolutionary (there are plenty of pre-computer sims; think martial arts practice with wooden swords, for instance)  but there’s no doubt that computing gives us a great chance to make this more real.  When games exploit the synergies between motivation and simulation, I think we’re going to see exciting things.

photo by musebrarian

Zotero [zoh-TAIR-oh] is a free, easy-to-use Firefox extension to help you collect, manage, and cite your research sources. It lives right where you do your work — in the web browser itself.

If you spend time doing research, you’re probably familiar with EndNote or RefWorks; this is the same idea, but with a couple advantages:

1. It’s integrated into your browser. You can download a citation on a web page to Zotero with one click.
2. It lets you write and store notes in the same database as your citations.
3. It’s free (as in both speech and beer).

It’s got a lot of other goodies, too: you can drag-and-drop citations into Word, OpenOffice, or an email; sort with tags and filters; full-text search as-you-type; and store and index pdf’s, web pages, and documents. The video below gives a three-minute overview:

Zotero has attracted some very positive attention. It’s funded by the Andrew W. Mellon and Alfred P. Sloan Foundations, and according to the website it’s also being recommended by libraries at institutions like Harvard, Cornell, Georgia Tech, and dozens more. Surprisingly, though, a lot of my colleagues have never heard of it. If my experience is any indication, that’s going to change.

I’ve been using Zotero for over a year now, and I can say it never fails to impress. Some of the things I love:

• If I want to store a copy of a PDF from a site, I just click and drag it to the correct Zotero entry. No stupid save dialogs.
• Organizing with tags, filters, and saved searches is way more flexible and powerful than nested folders.
• I hate wanting to cite something and not remembering what article it’s from. Indexed full-text search to the rescue. Love it.
• I hate typing. I love dragging several dozen references from Zotero to Openoffice and seeing them pop up as APA formatted citations.
• Having every thing I need–PDF, notes, citation–in one place for each article really speeds my workflow.
• And of course, I can’t even guess how much time one-click citation downloading has saved me over the last several hundreds of citations.

I could go on and on, but I’ll try to stop before(?) I become Mr. Obnoxious Open-Source Advocate Man.

And of course, there are some areas that could be improved. First, there are ways to make Zotero portable, so you can access you collections on different computers. There are ways…but there are not easy ways. The plan is for a central online space in which users can store collections; that would be a great solution, but it hasn’t happened yet. On the whole, Zotero is remarkably polished; there are, though, a few little annoyances here and there. If you generate a report for a collection, for instance, you can’t customize the fields (although see my Zotero report customizer here). Zotero is tied firmly to the Firefox browser; for some folks this is a distinct problem, regardless of how much we Fx users may love the ‘fox.

On the whole, though, I love Zotero. If you do research–especially if you’re not yet using a reference manager, you should give Zotero a look. I think it’s an open-source project that’s ready for the big time.