OK, you’ve peaked my interest. Downloaded and installed…we’ll see if I can learn to use it. If it automatically puts citations in APA or MLA format….that is very powerful.
I teach chemistry. I have used this theme in the classroom and often referred to it as a model but not the way it is depicted in the video.
While the theme on the video appears to be chemical elements and their reactions there really is little learning on chemistry conveyed, which is a pity. The idea is one that has some merit but without some careful thought to the plot there’s little to pass on to a student who would like to learn what chemistry is really all about.
Thanks for the opportunity to view a video with a chemical theme.
@Ken: [I think Ken meant to comment on this post] Good point; if the video was the only chemistry instruction your students got, they might end up with some unusual ideas about the subject, to say the least. In the classroom, I think the virtue of the piece would be less in teaching “what chemistry is all about” and more in provoking a discussion or maybe introducing a project.
I liked your Zotero report customizer. But, what if I want to exclude other fields than the ones you selected? With books, for instance, sometimes I don’t need or want info like number of pages, ISBN, short title, call number and repository. Thanks for your work!
Margaret, it wouldn’t be too hard to exclude those as well; I find that I’m pretty short on time since I got my new job, though, so it may not happen for a little while. However, if I get any other requests to add those categories I’ll make it a priority. Check back in a few weeks?
Just found zotero while looking for an OOo compatible reference manager (have been using bibus). Looks great, and your report customizer is (almost) a great help. I would be (almost) completely happy if you were to include the exclusion of the “Abstract”
I pull article refs from ISI Web of Science, and while I like having the abstract locally, it takes up a lot of space in the reports.
As is often the case, I appear to have spoken too soon. It seems the “Create Bibliography” produces exactly what I want for my library without the need to delete the abstracts. I just break it down by year and get manageable, reasonably chronological listings of the refs. Though, a fully tailorable Report editor would still be nice. Thanks again.
Hi, this is wonderful, I only came across it today – it is a really clean visualisation. I’ll be returning to it !
All the best with the programming too – this is pretty amazing considering how long you’ve been learning – congratulations,
Jason, really nicely implemented! Something that I (personally) would like to see is the ability to filter “edu-stopwords” (e.g. those like “education” or “learning” to see themes in a bit more fine-grained way.
@Lilia, thanks! You’ve got a good idea there; I spent quite some time trying to decide on which words should be stopwords and which shouldn’t; I ultimately went with the shorter list. Why not just let the user decide?
If you run the program on your own server, it’s easy to edit the stopword list, but I think you’re right that end-users should be able to do it, too. Shouldn’t be too hard to add…look for it in the next release.
@Willem, I’ve got two suggestions:
First, check the path to your cache in frontend/cache_locations.php; you may need to use absolute (like “/home/yourname/public_html” etc) rather than relative paths. You might also just double-check to make sure you didn’t include a typo or something.
Second, as you suggest, it may have to do with the way permissions on your server are set up. AFAIK, your best bet there is to contact your administrator. I’ll look into it and see if I can’t find another solution, though.
If you still have trouble, feel free to send me your opml, and I can run it on my server while you figure it out.
very impressive, a very nice way to explore tags in detail! keep up the good work – I’m really impressed at the use of javascript to create this as well!
This looks cool! But I have problem: I’d like to use it with a list of Norwegian blogs, but all I get is a list of short, common words, the equivalents of this, that, is, where and so on… I havn’t looked at the code yet, but could there be some way of making a list of stop-words, words that do not show up in the visualisation?
Thanks for your interest, Magnus. Yes, there’s no problem in adding a list of Norwegian stopwords to the others (I already have French, Spanish, English and German). In fact, I’ve already addedthis list. However, looking at the list I’m still guessing that we’re letting some common words slip through. If you send me a list of any common words that I haven’t included, I’d be glad to add it.
There are, however, two more significant problems: First, as you can see, all of the words are displayed in simple ASCII encoding, meaning that dicritics don’t appear. Second, the stemming algorithm I’m using works only for English words. So, while FeedVis knows that “school” and “schooling” are the same word, it won’t be able to make a similar judgment in other languages. Both problems result from me not anticipating the amount of interest this project would get; it was originally just a learning project for me.
Both problems are fixable. I’ve got my graduate school applications due in a week, so I won’t be able to look at FeedVis till after then. I’ll try to update it before the holidays, though.
I wish there was a way of integrating this with a Google Reader, because this is SO AMAZING.
I will take a look at it (at the code)… maybe I can figure a way of making it work better with Portuguese… It doesn’t know synonyms in Portuguese, does it?
I’ve just wrote a post about your great tool on my blog Outils Froids, a french blog for knowledge workers.
It would be again greater if it takes accented words. Actually it just cut them.
Feedvis is really a great tool and I don’t understand why there is no more buzz about it. Hope my post help : http://bit.ly/2g3C
@Cindy: Sorry for not getting your comment up sooner; I’ve been getting over a cold. Thanks for the kind words. No, it doesn’t know synonyms in Portuguese, but it doesn’t really know synonyms in any language. Perhaps you’re speaking of stemming, where words with the same stem (like learn, learning, and learned) are reduced to one. Here, it’s just English, I’m afraid.
If you’re comfortable working with PHP, it should be pretty easy to get a Portuguese stemmer working, though; just replace the stemmer.php class with a Portuguese version and you’re all set.
Also, I know my list of Portuguese stopwords is pretty incomplete; feel free to send any more and I’ll include them.
@Christophe: sorry about the delay; been feeling under the weather. Thanks for the review on Outils Froids; my French is not so great, but what I can understand of your blog, I like. I agree with you that one of the key challenges of the next few year is going to be helping user of the web filter and condense the vast amount of information available.
I’m sorry that FeedVis handles accented words so badly; it’s a side project that I don’t have loads of time for right now, but I am planning some improvements over the next month or so. I’ve gotten a very positive response from the blogosphere en Français, so I’d really like to improve the way FeedVis handles French in particular. Feel free to send me any other suggestions.
Must admit, when I first looked at this I didn’t think it worked so well, but when I found a project I needed it for and looked closer – it looks awesome! thanks!
Do you have any more info on installing it Jason? I’m keen to have a play, but not sure how to get it running on a web server. I’m still a little new to all this stuff. Any advice for the newbie? Would it work on an offline setup via WAMP at all?
@derf: Sorry that FeedVis doesn’t handle accented words so well. It’s on the to-do list.
@Dean: There is some information on installation in the text file that comes with the zip package. If you try that and still have trouble, feel free to shoot me an email. If your server has no web access, there really wouldn’t be much point, since it wouldn’t be able to access any information about blogs.
@Dennis: To include your list of Russian stopwords, just add them to stopwords.txt if you’re running FeedVis on your own server.
Hi, Can you offer more information on how to trigger the feedreader with a cron job. I’ve been looking at the files and running them manually but nothing seem to make any difference. I know how to set up a cron job but maybe I’m not getting the command right. Thanks.
I can get the PHP script to run. It shows the graph with the posts per day, but when I select a particular date to see the tag cloud, the animation runs, but does not actually display the cloud. Have you seen this behavior before? Any thoughts?
Here’s an update: The javascript error logger in FF is kicking back this:
Error: cal.win is undefined
Source File: frontend/js/tagcloud.js
Line: 178
I’m not a programmer, so no idea how to debug, but thought I’d pass along in case other people were seeing this problem. And any advice on how to solve would be greatly appreciated!
Sure enough, when I comment out that line, it works fine. Gonna leave it commented for now, but would love to get that fixed so I could actually see the dates when I hover over the graph. Thanks!
Joss, I never really set up v2 to run on a cron job; there’s no script to update everything at once. Instead, users update their own accounts when they visit them.
I’ll see about adding an update_all file in the next couple days.
well, personally I think you *are* missing something – I think the power here is to use the different levels of magnification as different narrative frames, so that the more you zoom in, the more you dig into the story. Add to that the ability to explore it both sequentially, via a path, and non-sequentially, by zomming and panning, and what you have here is a presentation tool *that is also* a decent multimedia authoring environment, so that the artifact you leave is usable in more than just a sequential, slidedeck way. But then that’d just my $0.02
Scott, I’m with you on the value of “different narrative frames” to help you “dig into the story.” I just can’t help thinking that this is a problem that’s already been solved–and solved more elegantly–by hyperlinked HTML. I can get an outline like
1. point one
2. point two
3. point three
and click to see more on the point that interests me. If page refreshes are the problem, 20 lines of Javascript gives me show/hide for links.
I admit that this may not look as pretty to some, but it also has significant advantages:
It’s accessible to users with screen readers
It doesn’t require a proprietary plugin (Flash), so it runs everywhere (Prezi wouldn’t work on the iPhone, for instance).
It’s easy to create and edit
It’s lightweight (my machine was having trouble rendering the bigger Prezi presentations in real-time)
It’s machine-indexable, and can include metadata
Like with Prezi, you can explore it sequentially or randomly
The artifact that remains after the presentation is based on an open, widely adopted standard that’s not going anywhere.
In return for giving up all this, Prezi gives you: cool zooming effects. I just don’t think that’s a fair trade; if your ideas are good, they’ll look good with out any whiz-bang special effects.
Believe me: as a one-time graphic designer, I have a keen love for beautifully-presented ideas, and I think that (despite what Semantic Web devotees might like to think), presentation is often inseparable from content. Prezi, though, seems less to beautify ideas than tart them up.
It’s likely that Prezi will find an enthusiastic audience, just as Powerpoint has, in boardrooms across the world. But I don’t share your hope that a lot of people are going to learn more because of it.
Prezi follows along in a number of systems that have attempted to show the context of the current slide in the larger collection of slides.
This is a standard visualization technique, called focus-context, and has some value.
The points here are all good, but I believe there’s solid data showing some value in exposing contextual placement in the course of a presentation of slides.
AndyEd, of course you’re right that Prezi isn’t the first do to this; in fact, I understand that MS is working on adding this to Powerpoint right now. My point, though, isn’t that Prezi has a bad goal. My contention, rather, is that Prezi’s secret sauce–zoom ‘n’ pan–doesn’t really do anything to meaningfully enhance communication. In fact, by distracting the audience with (barely) cool effects, it works against the clear expression of ideas.
I think we agree that focus-context is a Good Thing. But Prezi (at least in the examples I saw on their site) isn’t really showing me meaningful context–just a really small picture of the slide. I can see the heading, but not the text. How is that more context than a hyperlinked outline? Maybe there are some examples I didn’t see where the presentations was wholly visual; I suppose that might be more useful.
All that said, my Prezi criticism has nothing beyond the dubious authority of “what I think” to back it up. I’d be more than happy to eat my words if I could see some of the “solid data” you mention.
Your points on the failure of address munging are well-taken, but I would like to object to the general thrust of your argument: “Spam is a problem for you – obfuscation makes it a problem for your users.”
The thing is, spam *filtering* is a problem for your users, too. As a responsible webmaster, I would rather provide as many readers as possible with a way to contact me that they can be certain will actually *work*. The “take all of the problems onto yourself – just post it unobfuscated” path leads either to a mailbox full of crap that users’ letters get lost in, or to aggressive filtering that prevents you from seeing their messages in the first place.
Obfuscation (and here I speak of the image/CSS/Javascript variety) seems to me to be the least bad of the available options. I say this as a websurfer as well. If I’m going to take the time to send a private message to someone whose work I have just read, I’m already going out of my way to open up a communication channel, and a few seconds of checking their address is inconsequential compared to the time I’m going to spend typing at them anyway.
Hi Jason — have been looking at your Zotero Report Customizer. What I want is to change only the order of names within the Creator fields so the result shows surname first and given name(s) following. I can tinker with the Report text enough to get all the fields into a spreadsheet (which I want), but how to re-order the subfields within Creator?
The Zotero people indicated they’d arrange for this, but evidently it hasn’t happened.
I use GMail. Despite the fact that I get several spams an hour none appear in my inbox. All appear in the spam folder and I never look at them. To my knowledge I have never missed a legit email (except for credit card statements from one company which GMail marked as phishing attempts until I whitelisted them by clicking “this is not a phishing attempt”).
So I’m quite happy to post my email anywhere. Heck I’d write it on a bathroom wall.
Hey spammers: it’s angusgraham@gmail.com – go nuts.
I’m with Jason; the burden should be on the spammers, not the users. I am worried about false identification – I tend to regularly scan my caught spam as I do find too much real stuff trapped. However, that’s *my* burden, nor the sender’s burden.
Having managed Tech Support and been engaged in customer service and marketing, it is hard enough to get users to contact with queries, problems and purchases. Anything that gets in the way, including a few seconds of checking and rekeying, is too much.
Thanks for offering up the report cleaner script! I can’t remove table elements also associated with a class, e.g. “Author”. In case you’re interested, here’s a fix suggestion.
Oh, that didn’t post correctly at all, of course! The $tr_withclass_regex is the same as your $tr_regex expression, except that within the opening td html tag it allows for the class expression:
Thanks, Amanda; good observation! I fixed it a little differently than you suggested: I just changed the <th> to <th[^>]*>, so that the table cell can either have a class or not.
Baxil, I see where you are coming from; I staked out a pretty extreme position in my original post partly for the sake of argument. That said, though, I’m not I’m very convinced by your case.
You say that “spam *filtering* is a problem for your users;” I don’t see how that’s true. You offer two choices:
either to a mailbox full of crap that users’ letters get lost in, or to aggressive filtering that prevents you from seeing their messages in the first place.
But I see at least two other options:
better spam filtering or
more permissive spam filter settings and more “eyeball filtering”
True, manually scanning through lot of spam is a pain for you–but that was my whole point. It’s a pain for you, not your users.
@Gmail guy: now that’s some impressive putting your money where your mouth is.
@Jeremy: “It is hard enough to get users to contact…anything that gets in the way…is too much.” Exactly. Because contact is so important, I would think this is the last place you want to put any kind of obstacle in front of the user.
I read Dreaming In Code a few years back and it’s still one of my favourite books- one of the most readable books on the process of creating software. I liked how they weave the development of Chandler — actually the 1st time I heard of it — into stories about how other software had been developed.
Yes!! Stop obfuscating and
Yes!! Use Gmail… I left Gmail for a short period for various reasons (I am using IMAP to have best of both worlds, and temporarily unforwarded my personal domain from my gmail for awhile) – after a week or so of dealing with the spam, I was running back to gmail – it is far away the most accurate spam detector around.
Thanks for FeedVis. It got lots of attention on the web and seems also having a lot of potential. I ave tried on my own machine but I can not get any of the results I want, not event see the demo post because I can’t go back in time and the post of the tags are so old that they not get listed properly. Trying to upload my own opml didn’t work either (I don’t know why).
We try even to understand the code (I’m not a programmer but I have a friend who makes php) and your notice about trying to refactor it and seeing the mess and not having the time makes me thing if there is any way to keep the good ideas of FeedVis rolling, may be even with your small time. Perhaps a mailing list or group could work.
Thanks for the good ideas and preliminar work and sharing your learning and keep the feedback even with the small amount of time.
Let’s face it, guys. It’s a moot point. The reason everyone uses PowerPoint is NOT that it’s a better package. Persuasion, now defunct, was decades ahead of PowerPoint in allowing the user to design multiple layouts; Harvard Graphics was way ahead in classy graphics and charts. People use PowerPoint because Bill Gates figured out a way to get it on every business machine sold, long enough to fool the non-designer business person into thinking it was the next best thing since sliced bread. You gotta have more than zoom-in/-out to overcome that!
Offray, thanks for your comments, and my apologies for not getting back to you sooner; I’ve been overseas and had very limited internet access.
Sorry that you’ve had trouble getting FeedVis to run on your server. If you email me with the specifics of your problem, I’m reckon we can get it solved; so far most people have been successful after we tried a few things.
As for a mailing list or group around FeedVis, you’re more than welcome to get something like that started. I’m afraid getting starting here at UNC is taking 110% of my time right now.
Thanks again for the kind words and encouragement. It’s great living in a time and place where even a novice programmer like me can make something that folks find interesting. The Internets: it’s gonna be big.
Just like everyone else I got attacked by unwanted emails on a daily bases, so I went on a quest to find a solution to stop it.
Part 1
The first step I took was jumping on the other side of the river and think like a spammer. I started to search for software that does the harvesting of emails on the internet.
Using keywords such as “emails, harvest and extract” on Google and I ended up looking at hundreds of software listings, offering an easy way to attack unprotected emails in a few steps…
I picked up software, called EmailSpiderGold to test. Within a couple of hours I ended up in harvesting 15000 webmasters emails to use on my discretion.
Along the way I learned that, on the open are several ways to verify that those emails are active as the very developers also offer Email Verifiers which along many characteristics it checks the validity of recipient’s e-mails addresses by connecting to SMTP-servers and simulating the sending of a message and they work pretty smart too as they disconnect as soon as the mail server informs the program whether the address exists or not. On this conclusion we end up thinking that once the email is out there everyone can harvest it and use it without discretion for their own purpose.
Part 2
Solutions…
I came across to several solutions being offered to prevent the emails from harvesting campaigns. Amongst them I found some interesting ones using java scripts to obfuscate the coding on the page.
Strangely, I didn’t come across with anyone using their own encryption to publish their email on the web page.
Their lack of confidence was the answer for me.
Accidentally I got in touch with an old time software developer that shared the same frustration named Peter Johansson; together we joined forces and experiences to develop a shield to the issue. Only recently we had a winner called ATG, an Anti-Spam Tag Generator with advanced features that hides the real address from robotic harvesters. We tested it and it has proved to work.
Elton, I agree with you that “once the email is out there everyone can harvest it.” In fact, my point is that we should be trying to make it easy to get. Most obfuscations challenge users more than spambots.
I also agree that, for the time being, Javascript-based obfuscation holds the most promise. It’s not a silver bullet, though, as I discuss in my post. The ATG product you mentioned (and sell on your site as a downloadable exe) is a good example. Let’s take a look at what ATG cranks out:
<script type="text/javascript">
function SLMEJMBF(A){
var S = String.fromCharCode(109,97,105,108,116,111,58,116,101,115,»
116,64,101,120,97,109,112,108,101,46,99,111,109);
A.href = S;
}
</script>
<a href="#" onmouseover="SLMEJMBF(this);" onfocus="SLMEJMBF(this);">mail example</a>
For starters, if the client has javascript disabled, it breaks completely. That means tough luck, NoScript user: no email for you. This isn’t an insurmountable problem, though; check out Philip Hutchison’s gracefully-degrading script, for example.
Second, the “encryption” you use is pretty trivial. You rely on Javascript’s “fromCharCode” method to read the munged address–so can the harvester. I added a simple function to my de-obfuscator demo to show how easy this is (it’s example 11).
If I can break this munge with a 10-line function in a few minutes, trust me: someone else already has. Granted, this gets a lot harder to beat if you get just a little trickier; for instance, you might try breaking the address down into 10 strings and then concatenate them out of order–now a simple regex isn’t enough.
But the basic problem hasn’t gone away: your server dishes out your unencrypted Javascript to anyone who wants it, no questions asked. That makes it a fundamentally bad place to put secrets.
so i can filter or send any form of my address to spam and know where it was harvested from
to get around this you could find addresses with gmail domain and remove everything after the + and the periods so its the clean version (unless the normal address has a period like carter.cole@dadada.com)
Another thumbs up for gmail. On average a lottery win or african millionaire only gets through about once a month and to my knowledge (i do check my spam folder periodically), I have not missed any valid emails.
I’m partner in the Prezi competitor Ahead http://www.ahead.com, and would like to argue why the online zooming approach has it’s advantages apart from being stunning.
1) It’s online, hence you can share and embed your presentations anywhere on the web. Try that with a PPT
2) PPT is essentially built for text-based communication. Ahead (and Prezi) are built for communicating visually (which has proven to be the most effective way to comunicate)
3) You can present your information in a wider spatial context. Zoom out to get an overview, zoom in on to look at a detail. Ahead is mostly used by architects, designers and photographers and neither Keynote nor Powerpoint give them an adequate way of presenting visual content in the detail and richness that they require.
That said, I totally agree that for the left-brained business world PPT is still the best and safest bet, and probably will be for quite some time. But for right-brainers the spatial approach Ahead & Prezi offers is far superior.
I’m extremely inexperienced in javascript and web programming in general, so forgive me if this sounds dumb, but what do you think about using a text field and emailing yourself whatever text the user has entered? Is there a way to make this method more impregnable (as far as maintaining email address secrecy is concerned)? I know it doesn’t take much to write a bot that spams such a system with messages, but methinks this presents a more controllable environment.
On a different note, is there some way to use raw IP addresses instead of URLs that could throw a bot scanning for “x”@”y”.”z” off balance without overly confusing a desirable emailer?
Another possibility is the use of email aliasing. Maintain a single account for which you keep the direct address secret. Create an alias on an email server and use that alias “au naturale” in your sites, forwarding messages sent to it on to your central account. When you start to get too much spam through that alias, create a new one and repeat. Is this a reasonable solution, or am I misunderstanding some easy-to-foil step in this process?
Finally, a comment: I notice you keep mentioning that even the most complex obfuscation methods are easily discovered and routed if someone just looks at the code. Well… since there are so many possibilities, any spammer (hell, any programmer) would be hard-pressed to write a bot that could break them all by automation. A spammer would have to look at the code personally for each potential address to be certain of even (I’m guessing) 50% success… why bother, when the spammer could just look at the email address directly as the browser renders it on the page? What I’m trying to say is, I don’t see how “they can figure out how your code works by looking at it” is a reasonable argument against javascript-powered address obfuscation. The goal of obfuscation is to necessitate a human individual’s involvement in the identification of your email address with minimal confusion to that individual… methinks a sophisticated javascript obfuscation method accomplishes that goal.
Let’s face it, guys. It’s a moot point. The reason everyone uses PowerPoint is NOT that it’s a better package. Persuasion, now defunct, was decades ahead of PowerPoint in allowing the user to design multiple layouts; Harvard Graphics was way ahead in classy graphics and charts. People use PowerPoint because Bill Gates figured out a way to get it on every business machine sold, long enough to fool the non-designer business person into thinking it was the next best thing since sliced bread. You gotta have more than zoom-in/-out to overcome that!
[...] along to you. Jason Priem left a comment on my site and I followed the link back to his blog and found this amazing Firefox add-on that I thought some of you might enjoy. I’ve been playing with it for about an hour now and I [...]
[...] discussed in a previous post, I’m an enthusiastic user of the free reference manager Zotero; I’m impressed with how such young, open-source product has managed to quickly outshine [...]
FeedVis [jasonpriem.com] is an online tag cloud generator with some additional interactive features. Users can select specific time periods, common blog themes or individual blog feeds. Individual tags can be further explored to read specific blog pos…
63 Comments
OK, you’ve peaked my interest. Downloaded and installed…we’ll see if I can learn to use it. If it automatically puts citations in APA or MLA format….that is very powerful.
Thanks for the tip!
Yep, pretty powerful. MLA, APA–and a few dozen more to boot (although I’m pretty much all APA all the time, meself).
Kia Ora
I teach chemistry. I have used this theme in the classroom and often referred to it as a model but not the way it is depicted in the video.
While the theme on the video appears to be chemical elements and their reactions there really is little learning on chemistry conveyed, which is a pity. The idea is one that has some merit but without some careful thought to the plot there’s little to pass on to a student who would like to learn what chemistry is really all about.
Thanks for the opportunity to view a video with a chemical theme.
Ka kite
from Middle-earth
@Ken: [I think Ken meant to comment on this post] Good point; if the video was the only chemistry instruction your students got, they might end up with some unusual ideas about the subject, to say the least. In the classroom, I think the virtue of the piece would be less in teaching “what chemistry is all about” and more in provoking a discussion or maybe introducing a project.
I liked your Zotero report customizer. But, what if I want to exclude other fields than the ones you selected? With books, for instance, sometimes I don’t need or want info like number of pages, ISBN, short title, call number and repository. Thanks for your work!
Margaret, it wouldn’t be too hard to exclude those as well; I find that I’m pretty short on time since I got my new job, though, so it may not happen for a little while. However, if I get any other requests to add those categories I’ll make it a priority. Check back in a few weeks?
Thanks! I will!
Just found zotero while looking for an OOo compatible reference manager (have been using bibus). Looks great, and your report customizer is (almost) a great help. I would be (almost) completely happy if you were to include the exclusion of the “Abstract”
I pull article refs from ISI Web of Science, and while I like having the abstract locally, it takes up a lot of space in the reports.
Thanks for your work!
As is often the case, I appear to have spoken too soon. It seems the “Create Bibliography” produces exactly what I want for my library without the need to delete the abstracts. I just break it down by year and get manageable, reasonably chronological listings of the refs. Though, a fully tailorable Report editor would still be nice. Thanks again.
Hi, this is wonderful, I only came across it today – it is a really clean visualisation. I’ll be returning to it !
All the best with the programming too – this is pretty amazing considering how long you’ve been learning – congratulations,
Nicola
Thanks, Nicola! Making it clean and usable was definitely a priority for the project, so I’m glad to hear that you got that feel.
Jason, really nicely implemented! Something that I (personally) would like to see is the ability to filter “edu-stopwords” (e.g. those like “education” or “learning” to see themes in a bit more fine-grained way.
Great view!
I try to get this running for Dutch edublogs, but can’t get it to work.
Does the cache directory need writing permissions?
@Lilia, thanks! You’ve got a good idea there; I spent quite some time trying to decide on which words should be stopwords and which shouldn’t; I ultimately went with the shorter list. Why not just let the user decide?
If you run the program on your own server, it’s easy to edit the stopword list, but I think you’re right that end-users should be able to do it, too. Shouldn’t be too hard to add…look for it in the next release.
@Willem, I’ve got two suggestions:
First, check the path to your cache in frontend/cache_locations.php; you may need to use absolute (like “/home/yourname/public_html” etc) rather than relative paths. You might also just double-check to make sure you didn’t include a typo or something.
Second, as you suggest, it may have to do with the way permissions on your server are set up. AFAIK, your best bet there is to contact your administrator. I’ll look into it and see if I can’t find another solution, though.
If you still have trouble, feel free to send me your opml, and I can run it on my server while you figure it out.
very impressive, a very nice way to explore tags in detail! keep up the good work – I’m really impressed at the use of javascript to create this as well!
Philip
http://blog.FigmentEngine.com
This looks cool! But I have problem: I’d like to use it with a list of Norwegian blogs, but all I get is a list of short, common words, the equivalents of this, that, is, where and so on… I havn’t looked at the code yet, but could there be some way of making a list of stop-words, words that do not show up in the visualisation?
Thanks for your interest, Magnus. Yes, there’s no problem in adding a list of Norwegian stopwords to the others (I already have French, Spanish, English and German). In fact, I’ve already addedthis list. However, looking at the list I’m still guessing that we’re letting some common words slip through. If you send me a list of any common words that I haven’t included, I’d be glad to add it.
There are, however, two more significant problems: First, as you can see, all of the words are displayed in simple ASCII encoding, meaning that dicritics don’t appear. Second, the stemming algorithm I’m using works only for English words. So, while FeedVis knows that “school” and “schooling” are the same word, it won’t be able to make a similar judgment in other languages. Both problems result from me not anticipating the amount of interest this project would get; it was originally just a learning project for me.
Both problems are fixable. I’ve got my graduate school applications due in a week, so I won’t be able to look at FeedVis till after then. I’ll try to update it before the holidays, though.
That does look like a very swish way of keeping up to date with trends in news and could be extremely useful. I hope the applications go well.
I wish there was a way of integrating this with a Google Reader, because this is SO AMAZING.
I will take a look at it (at the code)… maybe I can figure a way of making it work better with Portuguese…
It doesn’t know synonyms in Portuguese, does it?
I’ve just wrote a post about your great tool on my blog Outils Froids, a french blog for knowledge workers.
It would be again greater if it takes accented words. Actually it just cut them.
Feedvis is really a great tool and I don’t understand why there is no more buzz about it. Hope my post help : http://bit.ly/2g3C
@Cindy: Sorry for not getting your comment up sooner; I’ve been getting over a cold. Thanks for the kind words. No, it doesn’t know synonyms in Portuguese, but it doesn’t really know synonyms in any language. Perhaps you’re speaking of stemming, where words with the same stem (like learn, learning, and learned) are reduced to one. Here, it’s just English, I’m afraid.
If you’re comfortable working with PHP, it should be pretty easy to get a Portuguese stemmer working, though; just replace the stemmer.php class with a Portuguese version and you’re all set.
Also, I know my list of Portuguese stopwords is pretty incomplete; feel free to send any more and I’ll include them.
@Christophe: sorry about the delay; been feeling under the weather. Thanks for the review on Outils Froids; my French is not so great, but what I can understand of your blog, I like. I agree with you that one of the key challenges of the next few year is going to be helping user of the web filter and condense the vast amount of information available.
I’m sorry that FeedVis handles accented words so badly; it’s a side project that I don’t have loads of time for right now, but I am planning some improvements over the next month or so. I’ve gotten a very positive response from the blogosphere en Français, so I’d really like to improve the way FeedVis handles French in particular. Feel free to send me any other suggestions.
Must admit, when I first looked at this I didn’t think it worked so well, but when I found a project I needed it for and looked closer – it looks awesome! thanks!
Bjr,
Super outil mais non prise en compte de l’accentuation de la langue française…
Merci.
Do you have any more info on installing it Jason? I’m keen to have a play, but not sure how to get it running on a web server. I’m still a little new to all this stuff. Any advice for the newbie? Would it work on an offline setup via WAMP at all?
Hi. It’s wonderful tool and application.Thanks to author.
I would like use it with russian blogs. What about to add russian stopword list?
I can provide subj.
Thanks.
Denis
@derf: Sorry that FeedVis doesn’t handle accented words so well. It’s on the to-do list.
@Dean: There is some information on installation in the text file that comes with the zip package. If you try that and still have trouble, feel free to shoot me an email. If your server has no web access, there really wouldn’t be much point, since it wouldn’t be able to access any information about blogs.
@Dennis: To include your list of Russian stopwords, just add them to stopwords.txt if you’re running FeedVis on your own server.
Just wanted to say thank you for this. It’s a fantastic tool.
Hi, Can you offer more information on how to trigger the feedreader with a cron job. I’ve been looking at the files and running them manually but nothing seem to make any difference. I know how to set up a cron job but maybe I’m not getting the command right. Thanks.
I can get the PHP script to run. It shows the graph with the posts per day, but when I select a particular date to see the tag cloud, the animation runs, but does not actually display the cloud.
Have you seen this behavior before? Any thoughts?
Here’s an update: The javascript error logger in FF is kicking back this:
Error: cal.win is undefined
Source File: frontend/js/tagcloud.js
Line: 178
I’m not a programmer, so no idea how to debug, but thought I’d pass along in case other people were seeing this problem. And any advice on how to solve would be greatly appreciated!
-R
Sure enough, when I comment out that line, it works fine. Gonna leave it commented for now, but would love to get that fixed so I could actually see the dates when I hover over the graph. Thanks!
Joss, I never really set up v2 to run on a cron job; there’s no script to update everything at once. Instead, users update their own accounts when they visit them.
I’ll see about adding an update_all file in the next couple days.
@Rick: I’m not sure what’s causing that, as I can’t seem to replicate the problem. What browser are you using? Which version of FeedVis?
well, personally I think you *are* missing something – I think the power here is to use the different levels of magnification as different narrative frames, so that the more you zoom in, the more you dig into the story. Add to that the ability to explore it both sequentially, via a path, and non-sequentially, by zomming and panning, and what you have here is a presentation tool *that is also* a decent multimedia authoring environment, so that the artifact you leave is usable in more than just a sequential, slidedeck way. But then that’d just my $0.02
Scott, I’m with you on the value of “different narrative frames” to help you “dig into the story.” I just can’t help thinking that this is a problem that’s already been solved–and solved more elegantly–by hyperlinked HTML. I can get an outline like
1. point one
2. point two
3. point three
and click to see more on the point that interests me. If page refreshes are the problem, 20 lines of Javascript gives me show/hide for links.
I admit that this may not look as pretty to some, but it also has significant advantages:
In return for giving up all this, Prezi gives you: cool zooming effects. I just don’t think that’s a fair trade; if your ideas are good, they’ll look good with out any whiz-bang special effects.
Believe me: as a one-time graphic designer, I have a keen love for beautifully-presented ideas, and I think that (despite what Semantic Web devotees might like to think), presentation is often inseparable from content. Prezi, though, seems less to beautify ideas than tart them up.
It’s likely that Prezi will find an enthusiastic audience, just as Powerpoint has, in boardrooms across the world. But I don’t share your hope that a lot of people are going to learn more because of it.
Prezi follows along in a number of systems that have attempted to show the context of the current slide in the larger collection of slides.
This is a standard visualization technique, called focus-context, and has some value.
The points here are all good, but I believe there’s solid data showing some value in exposing contextual placement in the course of a presentation of slides.
AndyEd, of course you’re right that Prezi isn’t the first do to this; in fact, I understand that MS is working on adding this to Powerpoint right now. My point, though, isn’t that Prezi has a bad goal. My contention, rather, is that Prezi’s secret sauce–zoom ‘n’ pan–doesn’t really do anything to meaningfully enhance communication. In fact, by distracting the audience with (barely) cool effects, it works against the clear expression of ideas.
I think we agree that focus-context is a Good Thing. But Prezi (at least in the examples I saw on their site) isn’t really showing me meaningful context–just a really small picture of the slide. I can see the heading, but not the text. How is that more context than a hyperlinked outline? Maybe there are some examples I didn’t see where the presentations was wholly visual; I suppose that might be more useful.
All that said, my Prezi criticism has nothing beyond the dubious authority of “what I think” to back it up. I’d be more than happy to eat my words if I could see some of the “solid data” you mention.
Your points on the failure of address munging are well-taken, but I would like to object to the general thrust of your argument: “Spam is a problem for you – obfuscation makes it a problem for your users.”
The thing is, spam *filtering* is a problem for your users, too. As a responsible webmaster, I would rather provide as many readers as possible with a way to contact me that they can be certain will actually *work*. The “take all of the problems onto yourself – just post it unobfuscated” path leads either to a mailbox full of crap that users’ letters get lost in, or to aggressive filtering that prevents you from seeing their messages in the first place.
Obfuscation (and here I speak of the image/CSS/Javascript variety) seems to me to be the least bad of the available options. I say this as a websurfer as well. If I’m going to take the time to send a private message to someone whose work I have just read, I’m already going out of my way to open up a communication channel, and a few seconds of checking their address is inconsequential compared to the time I’m going to spend typing at them anyway.
Hi Jason — have been looking at your Zotero Report Customizer. What I want is to change only the order of names within the Creator fields so the result shows surname first and given name(s) following. I can tinker with the Report text enough to get all the fields into a spreadsheet (which I want), but how to re-order the subfields within Creator?
The Zotero people indicated they’d arrange for this, but evidently it hasn’t happened.
thanks, Lois
I use GMail. Despite the fact that I get several spams an hour none appear in my inbox. All appear in the spam folder and I never look at them. To my knowledge I have never missed a legit email (except for credit card statements from one company which GMail marked as phishing attempts until I whitelisted them by clicking “this is not a phishing attempt”).
So I’m quite happy to post my email anywhere. Heck I’d write it on a bathroom wall.
Hey spammers: it’s angusgraham@gmail.com – go nuts.
I’m with Jason; the burden should be on the spammers, not the users. I am worried about false identification – I tend to regularly scan my caught spam as I do find too much real stuff trapped. However, that’s *my* burden, nor the sender’s burden.
Having managed Tech Support and been engaged in customer service and marketing, it is hard enough to get users to contact with queries, problems and purchases. Anything that gets in the way, including a few seconds of checking and rekeying, is too much.
Hi Jason,
Thanks for offering up the report cleaner script! I can’t remove table elements also associated with a class, e.g. “Author”. In case you’re interested, here’s a fix suggestion.
Within the text substitution loop, add:
$tr_withclass_regex = ‘/\s*’ . str_ireplace(’_', ‘ ‘, $word) . ‘(.+?)/is’;
$text = preg_replace($tr_withclass_regex, ”, $text);
Cheers,
Amanda
Oh, that didn’t post correctly at all, of course! The $tr_withclass_regex is the same as your $tr_regex expression, except that within the opening td html tag it allows for the class expression:
class=”(.+?)”
–Amanda
Thanks, Amanda; good observation! I fixed it a little differently than you suggested: I just changed the <th> to <th[^>]*>, so that the table cell can either have a class or not.
Baxil, I see where you are coming from; I staked out a pretty extreme position in my original post partly for the sake of argument. That said, though, I’m not I’m very convinced by your case.
You say that “spam *filtering* is a problem for your users;” I don’t see how that’s true. You offer two choices:
But I see at least two other options:
True, manually scanning through lot of spam is a pain for you–but that was my whole point. It’s a pain for you, not your users.
@Gmail guy: now that’s some impressive putting your money where your mouth is.
@Jeremy: “It is hard enough to get users to contact…anything that gets in the way…is too much.” Exactly. Because contact is so important, I would think this is the last place you want to put any kind of obstacle in front of the user.
I read Dreaming In Code a few years back and it’s still one of my favourite books- one of the most readable books on the process of creating software. I liked how they weave the development of Chandler — actually the 1st time I heard of it — into stories about how other software had been developed.
Link: http://www.yyztech.ca/reviews/book/dreaming-in-code
Yes!! Stop obfuscating and
Yes!! Use Gmail… I left Gmail for a short period for various reasons (I am using IMAP to have best of both worlds, and temporarily unforwarded my personal domain from my gmail for awhile) – after a week or so of dealing with the spam, I was running back to gmail – it is far away the most accurate spam detector around.
Thanks for FeedVis. It got lots of attention on the web and seems also having a lot of potential. I ave tried on my own machine but I can not get any of the results I want, not event see the demo post because I can’t go back in time and the post of the tags are so old that they not get listed properly. Trying to upload my own opml didn’t work either (I don’t know why).
We try even to understand the code (I’m not a programmer but I have a friend who makes php) and your notice about trying to refactor it and seeing the mess and not having the time makes me thing if there is any way to keep the good ideas of FeedVis rolling, may be even with your small time. Perhaps a mailing list or group could work.
Thanks for the good ideas and preliminar work and sharing your learning and keep the feedback even with the small amount of time.
Cheers,
Offray
Let’s face it, guys. It’s a moot point. The reason everyone uses PowerPoint is NOT that it’s a better package. Persuasion, now defunct, was decades ahead of PowerPoint in allowing the user to design multiple layouts; Harvard Graphics was way ahead in classy graphics and charts. People use PowerPoint because Bill Gates figured out a way to get it on every business machine sold, long enough to fool the non-designer business person into thinking it was the next best thing since sliced bread. You gotta have more than zoom-in/-out to overcome that!
Offray, thanks for your comments, and my apologies for not getting back to you sooner; I’ve been overseas and had very limited internet access.
Sorry that you’ve had trouble getting FeedVis to run on your server. If you email me with the specifics of your problem, I’m reckon we can get it solved; so far most people have been successful after we tried a few things.
As for a mailing list or group around FeedVis, you’re more than welcome to get something like that started. I’m afraid getting starting here at UNC is taking 110% of my time right now.
Thanks again for the kind words and encouragement. It’s great living in a time and place where even a novice programmer like me can make something that folks find interesting. The Internets: it’s gonna be big.
Hi Jason!
This tool looks great!
I want to download the source and install in my server but the problem is the that download link is down.
Any other link i can grab the source code?
thanks!
Sorry about that download link; it got messed up in some server houskeeping I was doing. It should work fine now.
Just like everyone else I got attacked by unwanted emails on a daily bases, so I went on a quest to find a solution to stop it.
Part 1
The first step I took was jumping on the other side of the river and think like a spammer. I started to search for software that does the harvesting of emails on the internet.
Using keywords such as “emails, harvest and extract” on Google and I ended up looking at hundreds of software listings, offering an easy way to attack unprotected emails in a few steps…
I picked up software, called EmailSpiderGold to test. Within a couple of hours I ended up in harvesting 15000 webmasters emails to use on my discretion.
Along the way I learned that, on the open are several ways to verify that those emails are active as the very developers also offer Email Verifiers which along many characteristics it checks the validity of recipient’s e-mails addresses by connecting to SMTP-servers and simulating the sending of a message and they work pretty smart too as they disconnect as soon as the mail server informs the program whether the address exists or not. On this conclusion we end up thinking that once the email is out there everyone can harvest it and use it without discretion for their own purpose.
Part 2
Solutions…
I came across to several solutions being offered to prevent the emails from harvesting campaigns. Amongst them I found some interesting ones using java scripts to obfuscate the coding on the page.
Strangely, I didn’t come across with anyone using their own encryption to publish their email on the web page.
Their lack of confidence was the answer for me.
Accidentally I got in touch with an old time software developer that shared the same frustration named Peter Johansson; together we joined forces and experiences to develop a shield to the issue. Only recently we had a winner called ATG, an Anti-Spam Tag Generator with advanced features that hides the real address from robotic harvesters. We tested it and it has proved to work.
E.Hoxha
Elton, I agree with you that “once the email is out there everyone can harvest it.” In fact, my point is that we should be trying to make it easy to get. Most obfuscations challenge users more than spambots.
I also agree that, for the time being, Javascript-based obfuscation holds the most promise. It’s not a silver bullet, though, as I discuss in my post. The ATG product you mentioned (and sell on your site as a downloadable exe) is a good example. Let’s take a look at what ATG cranks out:
<script type="text/javascript">
function SLMEJMBF(A){
var S = String.fromCharCode(109,97,105,108,116,111,58,116,101,115,»
116,64,101,120,97,109,112,108,101,46,99,111,109);
A.href = S;
}
</script>
<a href="#" onmouseover="SLMEJMBF(this);" onfocus="SLMEJMBF(this);">mail example</a>
For starters, if the client has javascript disabled, it breaks completely. That means tough luck, NoScript user: no email for you. This isn’t an insurmountable problem, though; check out Philip Hutchison’s gracefully-degrading script, for example.
Second, the “encryption” you use is pretty trivial. You rely on Javascript’s “fromCharCode” method to read the munged address–so can the harvester. I added a simple function to my de-obfuscator demo to show how easy this is (it’s example 11).
If I can break this munge with a 10-line function in a few minutes, trust me: someone else already has. Granted, this gets a lot harder to beat if you get just a little trickier; for instance, you might try breaking the address down into 10 strings and then concatenate them out of order–now a simple regex isn’t enough.
But the basic problem hasn’t gone away: your server dishes out your unencrypted Javascript to anyone who wants it, no questions asked. That makes it a fundamentally bad place to put secrets.
Thanks for your comment, and good luck with ATG!
with gmail any address that has periods in it are ignored as well as anything after the plus (+) so
c.a.r.t.e.r@cartercole.com is the same as
carter.@cartercole.com is the same as
c.a.rter+spam@cartercole.com
so i can filter or send any form of my address to spam and know where it was harvested from
to get around this you could find addresses with gmail domain and remove everything after the + and the periods so its the clean version (unless the normal address has a period like carter.cole@dadada.com)
and the fact that foo@cool.com auto links as a mailto: anchor really helps…
Another thumbs up for gmail. On average a lottery win or african millionaire only gets through about once a month and to my knowledge (i do check my spam folder periodically), I have not missed any valid emails.
I’m partner in the Prezi competitor Ahead http://www.ahead.com, and would like to argue why the online zooming approach has it’s advantages apart from being stunning.
1) It’s online, hence you can share and embed your presentations anywhere on the web. Try that with a PPT
2) PPT is essentially built for text-based communication. Ahead (and Prezi) are built for communicating visually (which has proven to be the most effective way to comunicate)
3) You can present your information in a wider spatial context. Zoom out to get an overview, zoom in on to look at a detail. Ahead is mostly used by architects, designers and photographers and neither Keynote nor Powerpoint give them an adequate way of presenting visual content in the detail and richness that they require.
That said, I totally agree that for the left-brained business world PPT is still the best and safest bet, and probably will be for quite some time. But for right-brainers the spatial approach Ahead & Prezi offers is far superior.
I’m extremely inexperienced in javascript and web programming in general, so forgive me if this sounds dumb, but what do you think about using a text field and emailing yourself whatever text the user has entered? Is there a way to make this method more impregnable (as far as maintaining email address secrecy is concerned)? I know it doesn’t take much to write a bot that spams such a system with messages, but methinks this presents a more controllable environment.
On a different note, is there some way to use raw IP addresses instead of URLs that could throw a bot scanning for “x”@”y”.”z” off balance without overly confusing a desirable emailer?
Another possibility is the use of email aliasing. Maintain a single account for which you keep the direct address secret. Create an alias on an email server and use that alias “au naturale” in your sites, forwarding messages sent to it on to your central account. When you start to get too much spam through that alias, create a new one and repeat. Is this a reasonable solution, or am I misunderstanding some easy-to-foil step in this process?
Finally, a comment: I notice you keep mentioning that even the most complex obfuscation methods are easily discovered and routed if someone just looks at the code. Well… since there are so many possibilities, any spammer (hell, any programmer) would be hard-pressed to write a bot that could break them all by automation. A spammer would have to look at the code personally for each potential address to be certain of even (I’m guessing) 50% success… why bother, when the spammer could just look at the email address directly as the browser renders it on the page? What I’m trying to say is, I don’t see how “they can figure out how your code works by looking at it” is a reasonable argument against javascript-powered address obfuscation. The goal of obfuscation is to necessitate a human individual’s involvement in the identification of your email address with minimal confusion to that individual… methinks a sophisticated javascript obfuscation method accomplishes that goal.
Let’s face it, guys. It’s a moot point. The reason everyone uses PowerPoint is NOT that it’s a better package. Persuasion, now defunct, was decades ahead of PowerPoint in allowing the user to design multiple layouts; Harvard Graphics was way ahead in classy graphics and charts. People use PowerPoint because Bill Gates figured out a way to get it on every business machine sold, long enough to fool the non-designer business person into thinking it was the next best thing since sliced bread. You gotta have more than zoom-in/-out to overcome that!
7 Trackbacks
[...] along to you. Jason Priem left a comment on my site and I followed the link back to his blog and found this amazing Firefox add-on that I thought some of you might enjoy. I’ve been playing with it for about an hour now and I [...]
[...] discussed in a previous post, I’m an enthusiastic user of the free reference manager Zotero; I’m impressed with how such young, open-source product has managed to quickly outshine [...]
[...] – bookmarked by 2 members originally found by TwoDollarKerplunk on 2008-10-21 Zotero Report Customizer 2.0 http://jasonpriem.com/2008/08/zotero-report-customizer-20/ – bookmarked by 2 members originally [...]
[...] have value, but, as I’ve written before, they’ve a number of shortfalls as well. I’ve just finished my attempt to remedy some [...]
FeedVis RSS Feed Tag Cloud Generator…
FeedVis [jasonpriem.com] is an online tag cloud generator with some additional interactive features. Users can select specific time periods, common blog themes or individual blog feeds. Individual tags can be further explored to read specific blog pos…
[...] source code is available for download if you want to run your own memetracker (kind of). Via [...]
[...] I’ve written before, I love the free citation manager Zotero. And the group and sharing features that just dropped [...]