What’s special about these two papers is that they are the start of  a research project that I hope will become my dissertation, an idea I’m somewhat reluctantly calling “scientometrics 2.0.” (do we really need more 2.0s?) Scientometrics is

…the science of measuring and analysing science. In practice, scientometrics is often done using bibliometrics which is a measurement of the impact of (scientific) publications. (Wikipedia)

My idea is that we should be looking beyond this, and starting to mine Web 2.0 sources for signals of scholarly impact. There are a few big advantages to this approach:

1. It’s much faster.  Once a scholarly article is published, it takes a years for citations to that article to accumulate.  But it can take just days for, say, Diggs or tweets to show up: in our Twitter sample we found that nearly half the links to peer-reviewed articles appeared within a week of those articles’ publication.  This speed could be harnessed to make real-time, personal filters that inform scholars what’s groundbreaking across a broad set of fields. As the velocity and volume of science grow, this could be very valuable.
2. If I cite something, it probably had an impact in my work.  But what kind of impact?  What if I read it and talked about it, and it informed my general thinking–but not enough to cite?  Just looking at citations, we’re missing many other kinds of impact.  Ten years ago, this was the best we could do.  But today, scholars are using online tools like CiteULike, Mendeley, and Zotero to manage their libraries; Faculty of 1000 to review articles;  and Twitter, FriendFeed, and ResearchBlogging.org to discuss them.  Tools like these–and importantly, the open APIs many of them offer–allow us to lift the curtain and observe scholars in their native habitat.  Scientometrics 2.0 offers a chance for us to develop a richer, more nuanced picture of scholarly impact.
3. Finally, this approach allows us to break the centuries-old monopoly of the peer-reviewed article or monograph on scientific communication.  We can measure reactions not just to these articles, but also to blog posts, datasets, or videos.  If a certain blog post in your field is generating lots of buzz, there’s a good chance it’s worth your time.  Scientometrics 2.0 can support a sort of informal, “soft peer-review” that works for free, on everything.

At first, this approach will mostly be used for relatively “pure” academic study–learning more about how scholars communicate how impact is transmitted.  Soon, however, young scholars will start making a case to tenure and promotion committees that their heavily tweeted or bookmarked article should count in their favor. Ultimately, I think we’ll see tools that leverage this information to help direct scholars to the most important and relevant work for them, kind of a PostRank for academics.

Of course, there are some obstacles to this.  The most important one for now is getting people to trust that these alternative sources really mean anything.  Who cares if an article is tweeted a lot?  Won’t people game this?  What about scholars who don’t use social media (a majority, for now)?  These questions have answers, but they need to be taken seriously (see the articles for more detailed discussions).

Ultimately, scientometrics 2.0 is going to have to be something we investigate very carefully, and in the proper context.  However, in that context I think it has the potential to be quite valuable, and I”m excited about working toward this in the next several years.

(Note: for a bunch of relevant citations, see the first article.)

Tagclouds have value, but, as I’ve written before, they’ve a number of shortfalls as well.  I’ve just finished my attempt to remedy some of these problems: FeedVis.  It’s an animated tagcloud that lets you compare word frequencies accross different time periods and authors, then check out the posts that used the words.  The demo is using the feeds for Scott McLeod’s Technorati-compiled list of top 50 edublogs, since that’s what got me started about feeds and tagclouds in the first place (although the program will work with any set of feeds).  More details about how it works are on the demo page.

I think what I’m really most excited about is the way this uses animation to let you actually see the words changing from one sample to the next.    Motion is such an important part of the way we see the world, and it’s been underemployed in information visualization, I think (although this changing; Hans Rosling’s TED talks have gotten a lot of buzz, for instance).

The project has been really fun, and a great learning experience; it’s gotten me really pumped about inofVis for learning about online interaction.  I think there is a lot of potential there for ed tech research.  I’m also pretty excited about programming; I started learning in February (with php), and then started javascript a couple months ago.  It’s been a really mind-expanding experience, and I’m looking foward to my next project, probably once I get done with grad school apps.

But why stop there?  Why not a service that analyzes this kind of real-time, viewer-supplied data, selects the most interesting bits, and then displays it?  It would function both as a real-time fact-checker and a window into audience’s reactions.

Lots of people already live-blog these things; it would be easy to get several thousand people to submit their questions and search results to a server, using a standardized interface.  The software then just aggregates, organizes, and presents the results.  Volunteers who try to game the system would be shut out with Digg-style, community-driven user ratings.  If Google would make its real-time query data available, that’d be added, too, significantly broadening the sample’s relevance.

The exciting part comes when this user-created input—both information about facts and people’s more general reaction to what they’re hearing—is presented to the the debate audience and the debaters themselves in real time (via a big display in the venue, for instance.)  For one thing,  the audience and debaters would immediately know of factual errors or half-truths, and have easy access to cited sources.  This would work for relatively picky things, like the pronunciation of someone’s name, but also for more substantive problems in responses; imagine how much answers would improve if the debaters knew they might have to stand next to an 8-foot-high rendering of, “40 million Americans think you just dodged the question.”

But also, the tool could act in more positive ways.  The candidates would immediately know the reactions of a national audience had about what they say–what the audience was interested in, confused about, or skeptical about.  If 80% of people want to hear more about the differences in candidates’ economic plan, they probably will.  You would have a truly participatory, interactive town-hall meeting of sixty million people.   Techniques like this are already beginning to surface in education, with tools like classroom clickers and “google jockying.”  Could they raise the level and relevancy of national politics?

Note: oops, accidentally published this sans links.  fixed now.