Jason Priem

I’m excited that I’ve had two papers accepted this week: “Scientometrics 2.0: Toward new metrics of scholarly impact on the social Web,” with Brad Hemminger, and “How and why scholars cite on Twitter” (online soon) with Kaitlin Costello.

What’s special about these two papers is that they are the start of  a research project that I hope will become my dissertation, an idea I’m somewhat reluctantly calling “scientometrics 2.0.” (do we really need more 2.0s?) Scientometrics is

…the science of measuring and analysing science. In practice, scientometrics is often done using bibliometrics which is a measurement of the impact of (scientific) publications. (Wikipedia)

My idea is that we should be looking beyond this, and starting to mine Web 2.0 sources for signals of scholarly impact. There are a few big advantages to this approach:

1. It’s much faster.  Once a scholarly article is published, it takes a years for citations to that article to accumulate.  But it can take just days for, say, Diggs or tweets to show up: in our Twitter sample we found that nearly half the links to peer-reviewed articles appeared within a week of those articles’ publication.  This speed could be harnessed to make real-time, personal filters that inform scholars what’s groundbreaking across a broad set of fields. As the velocity and volume of science grow, this could be very valuable.
2. If I cite something, it probably had an impact in my work.  But what kind of impact?  What if I read it and talked about it, and it informed my general thinking–but not enough to cite?  Just looking at citations, we’re missing many other kinds of impact.  Ten years ago, this was the best we could do.  But today, scholars are using online tools like CiteULike, Mendeley, and Zotero to manage their libraries; Faculty of 1000 to review articles;  and Twitter, FriendFeed, and ResearchBlogging.org to discuss them.  Tools like these–and importantly, the open APIs many of them offer–allow us to lift the curtain and observe scholars in their native habitat.  Scientometrics 2.0 offers a chance for us to develop a richer, more nuanced picture of scholarly impact.
3. Finally, this approach allows us to break the centuries-old monopoly of the peer-reviewed article or monograph on scientific communication.  We can measure reactions not just to these articles, but also to blog posts, datasets, or videos.  If a certain blog post in your field is generating lots of buzz, there’s a good chance it’s worth your time.  Scientometrics 2.0 can support a sort of informal, “soft peer-review” that works for free, on everything.

At first, this approach will mostly be used for relatively “pure” academic study–learning more about how scholars communicate how impact is transmitted.  Soon, however, young scholars will start making a case to tenure and promotion committees that their heavily tweeted or bookmarked article should count in their favor. Ultimately, I think we’ll see tools that leverage this information to help direct scholars to the most important and relevant work for them, kind of a PostRank for academics.

Of course, there are some obstacles to this.  The most important one for now is getting people to trust that these alternative sources really mean anything.  Who cares if an article is tweeted a lot?  Won’t people game this?  What about scholars who don’t use social media (a majority, for now)?  These questions have answers, but they need to be taken seriously (see the articles for more detailed discussions).

Ultimately, scientometrics 2.0 is going to have to be something we investigate very carefully, and in the proper context.  However, in that context I think it has the potential to be quite valuable, and I”m excited about working toward this in the next several years.

(Note: for a bunch of relevant citations, see the first article.)