sapientia ex machina

A nice bit of infoVis from the web comic Piled Higher and Deeper.  Kind of not the best news for someone who’s applying to doctoral programs this fall…um, can my app go in a special pile for people who’ve been planning this for years, regardless of what the economy would’ve done?

Tag clouds, those darlings of early web 2.0, have been seeing something of a backlash lately. Zeldman was suggesting that tag clouds were the new mullets back in 2005; more lately, ReadWriteWeb wondered if tagclouds were dead altogether. The main complaint in both cases wasn’t that tag clouds were just no good, but that they’d become trendy and thus overused.  Later criticism has argued that the increasingly common practice of using tag clouds for navigation is fundamentally flawed.

But the problems of tag clouds–and their close cousin, word clouds–go deeper, to their usefulness as a visualization method.  These aren’t problems with how the method is used or misused, but with the idea itself.

Moritz Stefaner points out (and presents his own solution for) several problems with the format:

• tag clouds give a great picture of the “big head” of tags: the most frequently used tags that change little over time; they overlook, though, the “long tail”–where many of the interesting tags are located.
• tag clouds don’t show change over time.  Chirag Mehta has created a tag cloud with a time slider, which helps with this.  But as Stefaner points out, animating tag clouds doesn’t work very well, as the changing size of the cloud moves the words around so they’re hard to follow.
• Finally, tag clouds don’t show the relationships between tags (pretty much everyone who criticizes tag clouds mentions this one).

The IBM Many Eyes site has one of the best tag cloud (actually this does word clouds, too) tools I’ve seen, allowing users to get lots of data from each tag while keeping the interface clean and simple.  They make a great point about an inherent limitation of the tool: the size and shape of the words themselves isn’t controlled for.  So, long words seem more dominant than short ones, and words with lots of ascenders and descenders (the vertical strokes of letters like ‘b’ or ‘p’) tend to dominate as well.  This can subtly alter the overall gist that tag clouds are supposed to deliver.

The academic community has noted shortcomings of the technique, as well. Hearst and Rosner (2008) observe that the alphabetical layout of the cloud may lead to a sort of “false clustering” effect, as users misinterpret words because of surrounding tags.  Renninger and Shumar (2007) found that tag cloud quadrants have different rates of recall, a fact which most tag cloud designs ignore.  In fact, their findings suggest that a simple list of tags, ordered by frequency, may deliver a more accurate overall impression than a tag cloud.  Several researchers have sought to improve shortcomings in tag cloud presentation with packing and sorting algorithms that manage whitespace and cluster relevant concepts (Kaser and Lemire, 2007; Seifert, Kump, Kienreich, Granitzer, and Granitzer, 2008).

Now, this isn’t to say that tag clouds have no value; in fact, I think they have great potential. It’s just that we need to know when tag clouds and word clouds are appropriate, know their shortcomings, and (this is the fun part) try to find ways to make them better. Most of the sources cited above have set about doing just that. In my next post, I’ll discuss a few of these “next-generation tag cloud” concepts; in particular, I’ll be examining methods of using word clouds to compare different versions of a text.

You know, we learn we remember 10% of what we read, 20% percent of what we hear, but 80% of what we actually experience.  Or, wait, maybe it’s 20%.  Or 30?

Of course, as many people know, this delightful little statistic has no backing in any sort of serious research—nor, indeed, could it:

…As Dwyer points out, the reported percentages are impossible to interpret or verify without specifying at least the method of measurement, the age of the learners, the type of learning task, and the content being remembered (p. 10).  Despite the lack of credibility, this formulation is widely quoted, usually without attribution, and in recent years has become repeatedly conflated with Dale’s Cone, with the percentage statements superimposed on the cone, replacing or supplementing Dale’s original categories.

from Cone of Experience (PDF), entry in A. Kovalchick & K. Dawson, Ed’s, Educational Technology: An Encyclopedia. Santa Barbara, CA: ABC-Clio, 2003.

Several bloggers have likewise been struck by the curious disconnect between the popularity of this statistic and its relation to reality.  Despite its readily apparent dodginess (We remember 90% of what we experience?  So I perfectly remember everything I did for nine out of the last ten years?), people love quoting this thing.

So quote they do.  And, since there’s no actual citation for this thing, the meme is free to mutate, which is actually kind of fascinating; the plot above shows the pattern in ghits for different versions of this same ‘principle.’

But why?  Obviously, the meme lives because it has value to people; in this case,  it helps folks prove a point about better ways of teaching.  But that’s not really an answer; there’s no reverse version of this for people arguing the opposite side.  No, the real answer is this: the statistic lives because it demonstrates something that the speaker and the listener both already agree on.  Few people are going to call you on this statistic, because everyone knows that the gist is true in many situations; you probably will learn something better if you involve it in some kind of experience than if you just read about it and move on.

The New York Times did a great story some years ago on related idea, called Scientific Myths That Are Too Good to Die.  It documented how well-known experiments could become sort of “academic urban myths.”  Take, for instance, the experiment that lent it’s name to the oft-cited “Hawthorne Effect” (in which the participants’ mere knowledge that they’re part of an experiment skews results):

”The results of this experiment, or rather the human relations interpretation offered by the researchers who summarized the results, soon became gospel for introductory textbooks in both psychology and management science,” said Dr. Lee Ross, a psychology professor at Stanford University.

But only five workers took part in the study, Dr. Ross said, and two were replaced partway through for gross insubordination and low output.

A psychology professor at the University of Michigan, Dr. Richard Nisbett calls the Hawthorne effect ”a glorified anecdote.”

These “glorified anecdotes” (and glorified ballpark guesses, which is really what the percentage-retention statistic is) hang on, though, because, in Dr. Ross’ words again, “’sometimes a story deserves to be true.”  That is, the story or number itself may be wrong, but it may be a way to access a point that deserves our attention.

So, then, is a bad statistic in a good cause worthwhile?  What if my “90% retention” number gets that grumpy admin to allow my pet wiki project?  Is it worth it?  I say no, for reasons that lie outside the scope of this post (maybe next one?).  Any other opinions, though?