Jason Priem » infovis

Twitter and the new scholarly ecosystem

jason — Wed, 30 Nov 2011 19:00:56 +0000

This is a copy of a guest post I wrote for the LSE Impact of Social Sciences blog:

In 1990, Tim Berners-Lee created the Web as a tool for scholarly communication at CERN. In the two decades since, his creation has gone on to transform practically every enterprise imaginable–except, somehow, scholarly communication. Here, instead, we lurch ponderously through the time-sanctified dance of dissemination, 17th-century style. The article reigns. Scholars continue to wad the vibrant, diverse results of their creativity and expertise–figures, datasets, programs, abstracts, annotations, claims, reviews, comments, collections, workflows, discussions, arguments and programs–into publishers’ slow molds to be cast into articles: static, leaden information ingots.

Growing numbers of scholars, though, are realizing that this approach is no longer the best we can do. We’re defrosting our digital libraries, moving over a million personal reference collections online to services like Zotero and Mendeley (and in the process making the open reference list a new kind of publication). Scholars are flocking to scholarly blogs to post ideas, collaborate with colleagues, and discuss literature, often creating a sort of peer-review after publication. Emboldened by national mandates and notable successes, we’re beginning to publish reusable datasets as first-class citizens in the scholarly conversation. We’re sharing our software as publications and on the Web. The journal was the first revolution in scholarly communication; we’re on the brink of a second, driven by the new diversity, speed, and accessibility of the Web.

The poster child for this Scholcomm Spring is Twitter. There’s been terrific interest in scholars using Twitter to discuss and cite literature, for teaching, to enrich conferences, or less formally as a “global faculty lounge.” We recently finished a large study to get better data on these uses.

Instead of asking for self-identified scholars on Twitter, we started out with a list of around 9,000 scholars from five US and UK universities, then searched for their names on the Twitter API. After manually confirming all the matches, we downloaded all the tweets each scholar had made and coded the content of these. The graphic below has some details of our findings (click for full-size image), but here’s a summary:

Twitter adoption is broad-based: scholars from different fields and career stages are taking to Twitter at about the same rate.
Scholars are using Twitter as a scholarly medium, making announcements, linking to articles, even engaging in discussions about methods and literature. But the majority of most scholars’ tweets are personal, underscoring Twitter as a space of context collapse, where users manage multiple identities.
Only about 1 in 40 scholars has an actively-updated Twitter account. This may seem small, but keep in mind that Twitter’s only 5 years old; email was still a scholarly novelty 15 years after its creation. Taking the long view, the current count of scholars using Twitter is probably less important than its continued growth, which we see clearly.

Results like these are encouraging for those of us who see social media and related environments as the natural next frontier for communicating scholarship. It seems that scholars, without waiting for approval from the mandarins of the publishing industry, are beginning to explore and colonize the Web’s wide-open spaces.

But perhaps the most exciting thing about this nascent scholarly Great Migration is that the new, online tools of scholarship begin to give public substance to the formally ephemeral roots of scholarship: the discussions never transcribed, the annotations never shared, the introductions never acknowledged, the manuscripts saved and reread but never cited. These backstage activities are now increasingly tagged, cataloged, and archived on blogs, Mendeley, Twitter, and elsewhere. As more scholars move more of their workflows to the public Web, we are assembling a vast registry of intellectual transactions–a web of ideas and their uses whose timeliness, speed, and precision make the traditional citation network look primitive.

I’ve been involved in early efforts to understand and use these new data sources to inform alternative metrics of impact, or “altmetrics.” Altmetrics could be used in evaluating scholars or institutions, complementing unidimensional citation counts with a rich array of indicators revealing diverse impacts on multiple populations. They could also inform new, real-time filters for scholars burdened by information overload: imagine a system that gathers and analyzes the bookmarks, pageviews, tweets, and blog posts from your online networks, using your interactions with them to learn and display each day’s most important articles or posts.

Even better, what if every scholar in the world had such a system? We might do away with journals entirely. The Web can disseminate and archive products for nearly free. The slow, back-room machinations of closed peer review could be replaced by an open, accountable, distributed system that simply listens in to expert communities’ natural reactions to new work–the same way Google efficiently ranks the Web by listening in to the crowdsourced “review” of the hyperlink network.

Of course, this particular vision may not pan out. And although the current signs point toward more growth, scholars might get tired of Twitter. But to hang our hopes on a particular vision or tool is to miss what’s truly revolutionary about this moment. The journal monoculture, long the only viable approach to scholarly communication, is beginning to yield at its fringes to a more diverse, vibrant, online ecosystem of scholarly expression. This new ecosystem promises to change not only the way we express scholarship, but the way we measure, assess, and consume it.

my.altmetrics.org: alt-metrics for your CV

jason — Sat, 15 Jan 2011 06:59:50 +0000

The PLoS alt-metrics study is moving well; we’ve transitioned to an open notebook built on GitHub (which is awesome and a good topic for another post) and findings are starting to emerge.

I’m starting to think about next steps, and to me the obvious one is to build a frontend for our crawler–giving working scholars and funders the opportunity to try out alt-metrics for themselves.

I’m posting some rough mockups (click ‘em to embiggen) of what a public alt-metrics machine might look like. This would be a place where people could upload a list of DOIs (or some other ID) and get a page that’ll let them track, visualise, and analyse the impact of their work in a broader and faster way than citations alone allow–sort of a Google Analytics for your CV.

I’m thinking it’d be cool to also have a way to embed results in another webpage, so you could make your actual CV show alt-metrics in real time. But there are a lot of directions to take this. If you’ve got suggestions, I’d love to hear!

Here’s the homepage. Pretty basic–just a place to upload identifiers:

And here’s the results page where the real action happens. This shows a result from a pretty large set of articles, like a lab or funder might input:

Has journal commenting failed?

jason — Fri, 07 Jan 2011 08:00:53 +0000

It’s a great idea: take all the insights, suggestions, and criticisms on scholarly articles, the comments shared in journal clubs and scribbled in margins the world over, and make them accessible to everyone. Attach them to the article itself; make it a conversation, not an artifact. We have blog commenting, video commenting–why not article commenting?

That’s sounded good to a lot of publishers, and over the last five years, we’ve seen article commenting systems become pretty popular. But there’s a growing sense that article commenting isn’t working.

The bad

Gotzsche et al. (2010) look at author replies to BMJ’s “rapid response” comments. We’d hope the chance to interact with authors would be a big plus for article commenting; however, they found that even when comments could “invalidate research or reduce… reliability,” over half the time authors couldn’t be bothered to respond.

In another study, Schriger at al. (in press; thanks Bora) examine the prevalence of commenting systems in top medical journals. They report that the percentage of journals offering rapid review has dropped from 12% in 2005 to 8% in 2009, and that fully half the journals sampled had commenting systems laying idle, completely unused by anyone. The authors conclude, “postpublication critique of articles in online pages provided by the journal does not seem to be taking hold.”

Finally, I collected data on PLoS comments as part of a larger investigation of alt-metrics. As evident from the graphic, the number articles with comments has held more or less steady as the total articles published has grown: again, not a pretty picture for those of us excited about article commenting.

The good

I’m not ready to give up on comments yet, though, because I think there’s a different way to see these findings. The question shouldn’t be “have comments failed,” but “are they succeeding somewhere, and why?” After all, we’re still in the very early stages of this thing; change in scholarly communication so far has happened on a scale of centuries.

Active, widespread commenting would be a radical change in how scholars communicate, and as with all fundemental shifts, we can assume most early efforts will be failures. In the 1900s, way more automobile manufacturers went broke building lousy cars than flourished making good ones. So in looking at comment ecosystems, we shouldn’t be stuck ogling the crowd of inevitable false starts–we should be trying to spot the nascent Model T.

And when we do see venues where comments are disproportionately successful, we should be trying to figure out what they’re doing right. While half the sample of the Schriger et al. study are stuck without a single commented article, BMJ, CMAJ, and Ann. Intern. Med. all have comments on 50-76%. How are they different? The BMJ articles sampled by Gotzshe et al. had a mean of 4.9 responses each, which is pretty respectable. Why are these here, but not elsewhere?

In the case of PLoS, we can see that even journals from the same publisher and on the same platform show widely different commenting rates. Is it the editors, the nature of the field, or something else that’s making PLoS Biology’s comment rate climb as PLoS Genetics’ holds steady and PLoS ONE’s drops? This is a great opportunity for research that will help commenting evolve further.

The future

So I think that while we see cases where journal commenting is beginning to succeed, we should continue to put resources behind spreading that success. This said, I have to admit I’m doubtful that publisher-hosted commenting is the future.

Today we have two scholarly communication ecosystems: the formal, peer-reviewed one, and the shadow system encompassing everything from scribbled marginalia, to chats in the lab, to peer reviews themselves. Sooner or later, I believe the shadow ecosystem will migrate to the web; a detailed argument for why is a different post, but there are too many advantages. It’ll happen. The advance guard is already conversing, learning, and collaborating on Zotero, Mendeley, CiteULike, blogs, Twitter, and so on.

Publisher-hosted article commenting is the formal system’s bid to gain a foothold in the informal system as it moves online. And it’s a smart bid, because as the shadow system sheds its ephemerality, it’s going to become increasingly important to how we measure and do scholarship.

But the problem is that journal-based comments are as siloed as the articles they comment on; there’s limited exposure, and no community. Scholars will want to have their conversations with their people, in their ways, in their places. Today, that mostly means Twitter and blogs (as we saw in #arsenicLife); in the future, it may also be scholar-specific services like The Third Reviewer, COASPedia, or VIVO.

So while I support article commenting as it now exists, I think challenge of the future won’t be moving the shadow communication system online–it’ll be aggregating it so it can be consumed, measured, and filtered efficiently and meaningfully. I think alt-metrics will play a part in that, but again, that’s another post :)

References:

Here’s the dataset and R code for the PLoS graphics; I hope to be releasing the full data next week.

Gotzsche, P. C., Delamothe, T., Godlee, F., & Lundh, A. (2010). Adequacy of authors’ replies to criticism raised in electronic letters to the editor: cohort study. BMJ, 341(aug10 2), c3926-c3926. doi:10.1136/bmj.c3926

Schriger, D. L., Chehrazi, A. C., Merchant, R. M., & Altman, D. G. (In press). Use of the Internet by Print Medical Journals in 2003 to 2009: A Longitudinal Observational Study. Annals of Emergency Medicine, In Press, Corrected Proof. doi:10.1016/j.annemergmed.2010.10.008

MEDLINE literature growth chart

jason — Mon, 18 Oct 2010 21:35:43 +0000

We all know the volume of scientific literature is growing. I went looking for an infographic showing this, but wasn’t satisfied with what I found, so I made one, based on the publication dates of articles in MEDLINE.

I got the data by searching PubMed with the query
("[year]"[Publication Date])where [year] was each year from 1950-2009. Then I charted the results in R, and resized them in Photoshop.

The data, R code, and images are all CC0 (public domain), and can be used wherever and for whatever you fancy.

small version of graphic

num-medline-articles-published-by-year.txt

# setup.
pub <- read.table("path_to_data_file", header=TRUE)
par(cex=2.2) # controls the relative size of the text
mainTitle <- "MEDLINE-indexed articles\npublished per year"
 
# make the plot.
# see http://www.harding.edu/fmccown/r/ for a nice intro on plot options
plot(pub, main=mainTitle, ylab='', xlab='', type="l", axes=F, 
     col='red', lwd=6, ylim=c(0,1000000))
 
# label the axes
axis(1, at=seq(1950, 2010, 10), lab=seq(1950, 2010, 10))
labs <- c('','','200k','','400k','','600k','','800k','','1M')# quick and dirty labels...
axis(2, at=seq(0, 1000000, 100000), lab=labs, las=2)  # "las" makes labeles display horiz.

Markup languages: who’s who?

jason — Fri, 02 Apr 2010 02:21:22 +0000

Is HTML XML? This question came up in a conversation with Sarah and @k8lin, and ended up being harder than I thought it’d be. There seems to be a fair amount of confusion on the topic, especially given the W3C’s recent abandonment of XHTML 2.0 and growing use of HTML5.

So, I decided to lay it all out in a (relatively) simple timeline format; as far as I know, this doesn’t exist anywhere else. You’re welcome, The Internet. Below are my sources and some notes; where possible, links are to the original recommendations or RFCs:

SGML is an ISO standard from the 80′s. Unlike the other standards on this list, it’s not open (the ISO sells copies for >$200). HTML is an “SGML application“, and has been from the beginning. The Wikipedia article has a lot more information on its origins, as does the W3C.

HTML 2.0 and HTML 3.2 , the first two W3C specs, are both pretty straightforward. Also straightforward is XML, which dropped in February 1998. Like HTML, XML is “an application profile…of SGML.”

In December 1999, the HTML 4.01 recommendation came out, followed a month later by XHTML 1.0. The important thing to note is that both of these are still HTML 4; however, XHTML is “a reformulation of HTML 4 as an XML 1.0 application,” while HTML 4.01 is still plain ol’ SGML.

No one knows yet exactly what HTML5 is going to look like, as it’s still several years off. However, the W3C tells us that the HTML5, like HTML4, is going to have two different “serializations.” One will be an XML syntax, and is currently being called XHTML 5 (wait, why not “XHTML 2?” Hang on, we’ll get there). You might expect that the other serialization would be SGML a la HTML 4.01. You’d be wrong.

Although HTML is technically SGML, most browsers and authoring tools couldn’t care less about the broader SGML standard; they just implement HTML. So the W3C’s plan seems to be to ditch the SGML legecy and replace it with “html” (note the lowercase), an entirely new standard…which happens to look pretty much like HTML has always looked.

Whew, we’re almost done. OK, what about XHTML2? Despite the name, project was not a “next step;” it was a huge break with the whole HTML/XHTML tradition, an effort to completely remake web markup. In July, the W3C decided to let it die on the vine and focus on HTML5. So XHTML 5, with its HTML lineage, will be a more incremental change than XHTML 2 would’ve been.

There you have it. If I missed anything or got something turned around, let me know.

Portrait of the artist as a phrenology illustration

jason — Fri, 22 Jan 2010 21:25:31 +0000

The first assignment in my infoVis class was to make a visual introduction to ourselves. I drew a self-portrait in profile, then added my categorized interests in the style of a 19th-century phrenology illustration (compare with actual period illustrations here and here).

Phrenology is interesting stuff. Though phrenologists had nearly everything wrong, modern neuroimaging has demonstrated that one important part of their core idea was right: many psychological functions really are highly localized in the brain. And they made a lot of really cool infographics. Actually, maybe this is pseudoscience in general; palmistry and astrology also make silly data into some neat-looking infovis. This same exercise would be fun with made-up star charts and palm diagrams.

$35 homemade whiteboard coffee table

jason — Wed, 23 Dec 2009 00:10:14 +0000

Whiteboards are great infovis tools, but expensive and need space. Solution: the whiteboard coffee table. It’s the very poor man’s Microsoft Surface (with no BSOD!). Also, if your taste in home decor tends toward the spartan (as does mine), this makes a great dinner table; it’s durable and really easy to clean. Most importantly, it’s cheap and you only need a drill and few hours to make it. Here’s how:

Materials:

Some 1×2 boards (you can pre-sanded ones for about $2 a piece)
A panel of “tile board,” which you can get from Home Depot or whatever for about 10 bucks.
some 3″ drywall screws
some 1 1/2″ drywall screws
wood glue

Tools:

Drill with a screwdriver bit
handsaw (may need it, may not; see below)
tablesaw or circular saw to cut the tileboard (may need it, may not; see below)

Construction:

Decide on the dimensions you want, and figure how many 1×2′s you need (see the diagram above for the general plan). You may need to be flexible here, depending on the sized of tile board panel you’re able to procure.
Get the materials. If you ask nice, a lot of times the store will cut the tile board for you, or they may have a 2′ x 4′ piece available. You can probably get them to cut the 1 x 2′s for you, as well.
Once you get the materials home, cut anything that still needs cuttin’.
Fasten everything together with the appropriate-sized drywall screws (The diagram shows where they go). I added glue, but you don’t really need it. Once the frame is done, glue the top on. Done!

Prezi: presentation junk 2.0

jason — Thu, 30 Apr 2009 18:50:44 +0000

It’s 2009. I think everyone out there knows that Powerpoint is, at best, overused (at worst:Stalin). Particularly gruesome is the animated slide-transition “feature,” which I think most agree has the same communication effectiveness and subtle charm as “” tags, mouse-cursor trails, and hilarious animated gifs of cats.

So how is it that presentation tool Prezi is suddenly the toast of the town? The quick sell looks like this:

“Prezi allows anyone who can sketch an idea on a napkin to create and perform stunning non-linear presentations with relations, zooming into details, and adjusting to the time left without the need to skip slides.”

I love how the first phrase suggests that there’s this great mass of napkin-sketching geniuses out there who can’t get their ideas out (until now!). I mean, I like mind maps, but turning one into an outline is pretty easy. So the presentations are “non-linear.” Does that mean the audience can interact with them, zooming in on sub-points of interest? If it does, let me show you this thing called “hyperlinks.” And is skipping slides really this tremendous problem?

When it comes down to it, the real selling point of Prezi is just the “stunning” presentation. Now, perhaps I’m jaded, but “zoom-in/zoom-out” leaves me unstunned. More importantly, though, this seems a textbook example of chartjunk: a “really great” visual effect that serves only to obscure or distract from real information. I think (hope) it’ll have the lasting appeal of Powerpoint’s racecar-noise-with-flying-in-bullet-point.

Perhaps I’m missing something (feel free to correct me in the comments) or just being curmudgeonly, but I think Prezi is vastly overhyped. Powerpoint is bad enough. Also: I like how the Prezi logo, by mixing case, suggests that the product may in fact be called “Pretzl.” Ok, now that’s definitely being curmudgeonly.

FeedVis 2.0: custom visualization for your feeds

jason — Wed, 03 Dec 2008 20:27:27 +0000

My FeedVis project–the interactive tagcloud for a group of feeds–has been out for a week now, I’ve been thrilled at the positive response I’ve gotten so far. One rather glaring problem with the program, though, was that you could only look at the top 50 edublogs.

Not anymore. After a few late nights, I’ve got a beta system for uploading and analyzing your own sets of feeds. You just upload your opml, wait a few minutes, and you’re set: FeedVis gives you a custom page that you can bookmark and return to anytime you like; it’ll continue to update every time you visit. You can also browse visualizations of other people’s feeds.

It’s pretty untested, and I’m sure use will uncover some bugs. But it’s got potential; I’m excited to see what people think.

FeedVis: a deeper tagcloud for edublogs

jason — Tue, 25 Nov 2008 07:53:05 +0000

Tagclouds have value, but, as I’ve written before, they’ve a number of shortfalls as well. I’ve just finished my attempt to remedy some of these problems: FeedVis. It’s an animated tagcloud that lets you compare word frequencies accross different time periods and authors, then check out the posts that used the words. The demo is using the feeds for Scott McLeod’s Technorati-compiled list of top 50 edublogs, since that’s what got me started about feeds and tagclouds in the first place (although the program will work with any set of feeds). More details about how it works are on the demo page.

I think what I’m really most excited about is the way this uses animation to let you actually see the words changing from one sample to the next. Motion is such an important part of the way we see the world, and it’s been underemployed in information visualization, I think (although this changing; Hans Rosling’s TED talks have gotten a lot of buzz, for instance).

The project has been really fun, and a great learning experience; it’s gotten me really pumped about inofVis for learning about online interaction. I think there is a lot of potential there for ed tech research. I’m also pretty excited about programming; I started learning in February (with php), and then started javascript a couple months ago. It’s been a really mind-expanding experience, and I’m looking foward to my next project, probably once I get done with grad school apps.