What’s special about these two papers is that they are the start of  a research project that I hope will become my dissertation, an idea I’m somewhat reluctantly calling “scientometrics 2.0.” (do we really need more 2.0s?) Scientometrics is

…the science of measuring and analysing science. In practice, scientometrics is often done using bibliometrics which is a measurement of the impact of (scientific) publications. (Wikipedia)

My idea is that we should be looking beyond this, and starting to mine Web 2.0 sources for signals of scholarly impact. There are a few big advantages to this approach:

1. It’s much faster.  Once a scholarly article is published, it takes a years for citations to that article to accumulate.  But it can take just days for, say, Diggs or tweets to show up: in our Twitter sample we found that nearly half the links to peer-reviewed articles appeared within a week of those articles’ publication.  This speed could be harnessed to make real-time, personal filters that inform scholars what’s groundbreaking across a broad set of fields. As the velocity and volume of science grow, this could be very valuable.
2. If I cite something, it probably had an impact in my work.  But what kind of impact?  What if I read it and talked about it, and it informed my general thinking–but not enough to cite?  Just looking at citations, we’re missing many other kinds of impact.  Ten years ago, this was the best we could do.  But today, scholars are using online tools like CiteULike, Mendeley, and Zotero to manage their libraries; Faculty of 1000 to review articles;  and Twitter, FriendFeed, and ResearchBlogging.org to discuss them.  Tools like these–and importantly, the open APIs many of them offer–allow us to lift the curtain and observe scholars in their native habitat.  Scientometrics 2.0 offers a chance for us to develop a richer, more nuanced picture of scholarly impact.
3. Finally, this approach allows us to break the centuries-old monopoly of the peer-reviewed article or monograph on scientific communication.  We can measure reactions not just to these articles, but also to blog posts, datasets, or videos.  If a certain blog post in your field is generating lots of buzz, there’s a good chance it’s worth your time.  Scientometrics 2.0 can support a sort of informal, “soft peer-review” that works for free, on everything.

At first, this approach will mostly be used for relatively “pure” academic study–learning more about how scholars communicate how impact is transmitted.  Soon, however, young scholars will start making a case to tenure and promotion committees that their heavily tweeted or bookmarked article should count in their favor. Ultimately, I think we’ll see tools that leverage this information to help direct scholars to the most important and relevant work for them, kind of a PostRank for academics.

Of course, there are some obstacles to this.  The most important one for now is getting people to trust that these alternative sources really mean anything.  Who cares if an article is tweeted a lot?  Won’t people game this?  What about scholars who don’t use social media (a majority, for now)?  These questions have answers, but they need to be taken seriously (see the articles for more detailed discussions).

Ultimately, scientometrics 2.0 is going to have to be something we investigate very carefully, and in the proper context.  However, in that context I think it has the potential to be quite valuable, and I”m excited about working toward this in the next several years.

(Note: for a bunch of relevant citations, see the first article.)

My FeedVis project–the interactive tagcloud for a group of feeds–has been out for a week now, I’ve been thrilled at the positive response I’ve gotten so far.  One rather glaring problem with the program, though, was that you could only look at the top 50 edublogs.

Not anymore.  After a few late nights, I’ve got a beta system for uploading and analyzing your own sets of feeds.  You just upload your opml, wait a few minutes, and you’re set: FeedVis gives you a custom page that you can bookmark and return to anytime you like; it’ll continue to update every time you visit.  You can also browse visualizations of other people’s feeds.

It’s pretty untested, and I’m sure use will uncover some bugs.  But it’s got potential; I’m excited to see what people think.

Tagclouds have value, but, as I’ve written before, they’ve a number of shortfalls as well.  I’ve just finished my attempt to remedy some of these problems: FeedVis.  It’s an animated tagcloud that lets you compare word frequencies accross different time periods and authors, then check out the posts that used the words.  The demo is using the feeds for Scott McLeod’s Technorati-compiled list of top 50 edublogs, since that’s what got me started about feeds and tagclouds in the first place (although the program will work with any set of feeds).  More details about how it works are on the demo page.

I think what I’m really most excited about is the way this uses animation to let you actually see the words changing from one sample to the next.    Motion is such an important part of the way we see the world, and it’s been underemployed in information visualization, I think (although this changing; Hans Rosling’s TED talks have gotten a lot of buzz, for instance).

The project has been really fun, and a great learning experience; it’s gotten me really pumped about inofVis for learning about online interaction.  I think there is a lot of potential there for ed tech research.  I’m also pretty excited about programming; I started learning in February (with php), and then started javascript a couple months ago.  It’s been a really mind-expanding experience, and I’m looking foward to my next project, probably once I get done with grad school apps.

But why stop there?  Why not a service that analyzes this kind of real-time, viewer-supplied data, selects the most interesting bits, and then displays it?  It would function both as a real-time fact-checker and a window into audience’s reactions.

Lots of people already live-blog these things; it would be easy to get several thousand people to submit their questions and search results to a server, using a standardized interface.  The software then just aggregates, organizes, and presents the results.  Volunteers who try to game the system would be shut out with Digg-style, community-driven user ratings.  If Google would make its real-time query data available, that’d be added, too, significantly broadening the sample’s relevance.

The exciting part comes when this user-created input—both information about facts and people’s more general reaction to what they’re hearing—is presented to the the debate audience and the debaters themselves in real time (via a big display in the venue, for instance.)  For one thing,  the audience and debaters would immediately know of factual errors or half-truths, and have easy access to cited sources.  This would work for relatively picky things, like the pronunciation of someone’s name, but also for more substantive problems in responses; imagine how much answers would improve if the debaters knew they might have to stand next to an 8-foot-high rendering of, “40 million Americans think you just dodged the question.”

But also, the tool could act in more positive ways.  The candidates would immediately know the reactions of a national audience had about what they say–what the audience was interested in, confused about, or skeptical about.  If 80% of people want to hear more about the differences in candidates’ economic plan, they probably will.  You would have a truly participatory, interactive town-hall meeting of sixty million people.   Techniques like this are already beginning to surface in education, with tools like classroom clickers and “google jockying.”  Could they raise the level and relevancy of national politics?

Note: oops, accidentally published this sans links.  fixed now.

But the problems of tag clouds–and their close cousin, word clouds–go deeper, to their usefulness as a visualization method.  These aren’t problems with how the method is used or misused, but with the idea itself.

Moritz Stefaner points out (and presents his own solution for) several problems with the format:

• tag clouds give a great picture of the “big head” of tags: the most frequently used tags that change little over time; they overlook, though, the “long tail”–where many of the interesting tags are located.
• tag clouds don’t show change over time.  Chirag Mehta has created a tag cloud with a time slider, which helps with this.  But as Stefaner points out, animating tag clouds doesn’t work very well, as the changing size of the cloud moves the words around so they’re hard to follow.
• Finally, tag clouds don’t show the relationships between tags (pretty much everyone who criticizes tag clouds mentions this one).

The IBM Many Eyes site has one of the best tag cloud (actually this does word clouds, too) tools I’ve seen, allowing users to get lots of data from each tag while keeping the interface clean and simple.  They make a great point about an inherent limitation of the tool: the size and shape of the words themselves isn’t controlled for.  So, long words seem more dominant than short ones, and words with lots of ascenders and descenders (the vertical strokes of letters like ‘b’ or ‘p’) tend to dominate as well.  This can subtly alter the overall gist that tag clouds are supposed to deliver.

The academic community has noted shortcomings of the technique, as well. Hearst and Rosner (2008) observe that the alphabetical layout of the cloud may lead to a sort of “false clustering” effect, as users misinterpret words because of surrounding tags.  Renninger and Shumar (2007) found that tag cloud quadrants have different rates of recall, a fact which most tag cloud designs ignore.  In fact, their findings suggest that a simple list of tags, ordered by frequency, may deliver a more accurate overall impression than a tag cloud.  Several researchers have sought to improve shortcomings in tag cloud presentation with packing and sorting algorithms that manage whitespace and cluster relevant concepts (Kaser and Lemire, 2007; Seifert, Kump, Kienreich, Granitzer, and Granitzer, 2008).

Now, this isn’t to say that tag clouds have no value; in fact, I think they have great potential. It’s just that we need to know when tag clouds and word clouds are appropriate, know their shortcomings, and (this is the fun part) try to find ways to make them better. Most of the sources cited above have set about doing just that. In my next post, I’ll discuss a few of these “next-generation tag cloud” concepts; in particular, I’ll be examining methods of using word clouds to compare different versions of a text.

You know, we learn we remember 10% of what we read, 20% percent of what we hear, but 80% of what we actually experience.  Or, wait, maybe it’s 20%.  Or 30?

Of course, as many people know, this delightful little statistic has no backing in any sort of serious research—nor, indeed, could it:

…As Dwyer points out, the reported percentages are impossible to interpret or verify without specifying at least the method of measurement, the age of the learners, the type of learning task, and the content being remembered (p. 10).  Despite the lack of credibility, this formulation is widely quoted, usually without attribution, and in recent years has become repeatedly conflated with Dale’s Cone, with the percentage statements superimposed on the cone, replacing or supplementing Dale’s original categories.

from Cone of Experience (PDF), entry in A. Kovalchick & K. Dawson, Ed’s, Educational Technology: An Encyclopedia. Santa Barbara, CA: ABC-Clio, 2003.

Several bloggers have likewise been struck by the curious disconnect between the popularity of this statistic and its relation to reality.  Despite its readily apparent dodginess (We remember 90% of what we experience?  So I perfectly remember everything I did for nine out of the last ten years?), people love quoting this thing.

So quote they do.  And, since there’s no actual citation for this thing, the meme is free to mutate, which is actually kind of fascinating; the plot above shows the pattern in ghits for different versions of this same ‘principle.’

But why?  Obviously, the meme lives because it has value to people; in this case,  it helps folks prove a point about better ways of teaching.  But that’s not really an answer; there’s no reverse version of this for people arguing the opposite side.  No, the real answer is this: the statistic lives because it demonstrates something that the speaker and the listener both already agree on.  Few people are going to call you on this statistic, because everyone knows that the gist is true in many situations; you probably will learn something better if you involve it in some kind of experience than if you just read about it and move on.

The New York Times did a great story some years ago on related idea, called Scientific Myths That Are Too Good to Die.  It documented how well-known experiments could become sort of “academic urban myths.”  Take, for instance, the experiment that lent it’s name to the oft-cited “Hawthorne Effect” (in which the participants’ mere knowledge that they’re part of an experiment skews results):

”The results of this experiment, or rather the human relations interpretation offered by the researchers who summarized the results, soon became gospel for introductory textbooks in both psychology and management science,” said Dr. Lee Ross, a psychology professor at Stanford University.

But only five workers took part in the study, Dr. Ross said, and two were replaced partway through for gross insubordination and low output.

A psychology professor at the University of Michigan, Dr. Richard Nisbett calls the Hawthorne effect ”a glorified anecdote.”

These “glorified anecdotes” (and glorified ballpark guesses, which is really what the percentage-retention statistic is) hang on, though, because, in Dr. Ross’ words again, “‘sometimes a story deserves to be true.”  That is, the story or number itself may be wrong, but it may be a way to access a point that deserves our attention.

So, then, is a bad statistic in a good cause worthwhile?  What if my “90% retention” number gets that grumpy admin to allow my pet wiki project?  Is it worth it?  I say no, for reasons that lie outside the scope of this post (maybe next one?).  Any other opinions, though?