Sorry for the recent outage on Processing Blogs. For some reason all the wordpress plugins deactivated themselves. I hate the internet, and the internet hates me.
Let's try this again. If you're reading this on Processing Blogs or via its feed, then everything should be working.
If you ever need to run a site similar to Processing Blogs and your web host can run python then I definitely recommend Planet or Planet Venus as the solution. WP-Venus complained a little bit when I converted from Feedwordpress, but it looks good so far: hopefully the archives will be worth the effort.
If you're reading this on Processing Blogs, or via its feed, then everything should be fixed. For some reason, Feedwordpress just stopped working and wouldn't re-subscribe to lots of the feeds that were previously fine.
Feeds are now being grabbed robustly by Planet and merely massaged by Feedwordpress. Here's the current list:
Please do let me know if I'm syndicating too much or too little, and especially if I'm missing people or if I'm syndicating your blog and you don't want me to. Otherwise, the site should run itself for a while.
Whilst I wrestle with my reaction to the reception of Twitter Blocks, it's interesting to look at what other people in information visualisation are working on.
Yahoo's new design research outfit, apparently also known as yhaus, have just put up a site outlining their work so far. The first thing I noticed is that they've snagged an amazing subdomain: design.yahoo.com. The second thing I noticed is that compared to other teams I'm aware of in the field (including Stamen) they have a pretty good gender balance (a thorny issue but I'm noting it anyway). The third thing I'll note is the guest appearance of non-Yahoo work in the portraits of the team: I see Torrent Raiders, Fidg't... what else?
Sadly, all but one of the demos there so far is a big Quicktime movie. I know that with millions of users Yahoo has to be a stickler for browser support and compatibility, but I hope they get a chance to take this work live on the web as well as demo it in movie form. There's some solid realtime Flash and Processing work hiding in there, and people (OK, I) want to see it in its interactive entirety.
There's clearly some healthy collaboration and influence going on there (much as in our work, e.g. Ben Fry's zipdecode looms large over the interactive version of our Trulia search animation). Yahoo's Aaron Koblin is best known for his Flight Patterns piece, and this visualisation by Michael Chang of Yahoo trip planner data is very similar:
Likewise, Aaron's work on traffic patterns bears a close resemblance to Flight Patterns:
I don't want to pick on them too much, because it's really beautiful work I admire a great deal (and it might seem like sour grapes), but both the pieces I've highlighted do suffer from something we've tried to avoid at Stamen: animated information graphics on top of black backgrounds and vector maps can easily look like screenshots from a modern-day War Games:
I'm glad other people as well as us are experimenting in public, and I'm glad sites like Infosthetics and Visual Complexity are cataloguing our efforts. We need our own visual language around this kind of visualisation that doesn't resonate with the imagery of war.
Apple's iphone has made a strong impression with slick transitions in its interface design, but the maps application still borrows from Google's pseudo-shadows and static pins. The playful interfaces of the Nintendo's Wii games certainly offer a different path, but the rest of the games (and movie) industry's cinematic-realism aesthetic exerts a strong influence over our generation of designers and it isn't going to meet these goals any time soon.
It's a fairly regular topic of conversation at Stamen: how can you make a visualisation of e.g. 911 calls actually look like emergencies, and not birthday parties or toilet flushes, without freaking people out and without making it mundane? Is it possible to use great circles to connect air travel destinations without it looking like missiles? Can you animate growing and shrinking red and yellow circles on an aerial map without it looking like Gulf War I weapons company propaganda?
It's a bloggy weekend here at Random Etc, if you're reading along I'm definitely interested to hear your thoughts.
We finally kicked Twitter Blocks out of the door yesterday. It's been in development for about a month, mainly by Ryan Alexander and myself (but all Ryan whilst I was working on Oakland Crimespotting, he's a star). It's the first time we've done 3D things using Flash and it's amazing what's being done with Papervision3D at the moment. On the other hand, having 3D shoe-horned into the 2D Flash engine means the learning curve is a lot steeper than the native-3D Processing/OpenGL worlds Ryan and I are used to.
Whenever Stamen launches a new thing, my immediate reaction is a sigh of relief followed by a slightly obsessive-compulsive trawl of what people are saying about it online. (I use a combination of alerts from Technorati, Google Blog Seach and Bloglines to keep track).
The interesting thing about working with companies like Digg and Twitter is that your work inherits all the criticisms and detractors of those sites as well. Digg's users are clamouring for a picture section, so when we launched Digg Arc many of the responses ignored the piece entirely and chastised Digg for paying attention to visualisation and not to the main site. The same argument is already being used against Twitter Blocks, even though the amount of time Twitter's developers put into it was tiny compared to the amount of time they're putting into stability and new features.
Don't get me wrong: some of the early feedback we're getting is very positive, the team at Twitter have been very receptive and we're proud of our work. This much is good. Some of the feedback we're getting points out that the work isn't immediately understandable (I agree, and maybe we could do some more explaining, but I think we're OK for now).
However, there is also a strong and steady flow of negative comments that I've gathered here so I can think about them all in one place.
“Pretty visualization but I doubt its practicality.” PoppuPot
“Twitter Blocks is the kind of thing that demos well at conferences. Not too useful in real life.” Dave Winer
“exploring myneighbourhood : fun 3D view but so what? not sure i will do that everyday.” jean-michel gobet
“Well its interesting that its a new Twitter toy, but I just don't get it. Functional?” programwitch
“Puzzled but Entertained” … “Not really got the slightest idea why this is anything other than an interesting folly.” Tom Coates
“What is the point? Beats me.” Russel Heimlich
“I have to say I was absolutely gobsmacked by how utterly pointless it is.” EirePreneur
(The last one is, short of a personal insult, pretty much the harshest thing anyone has ever said about work I've been involved in.)
To address Tom Coates' point, if you're entertained we've done our job. If you're puzzled then maybe we can help explain things better next time. But I don't mind if some of our work is seen as a folly (“an extravagant, frivolous or fanciful building, designed more for artistic expression than for practicality”). Not everything that everyone does has to be useful or profound. (Nevermind that we've personally found Twitter Blocks a useful way to explore the Twitter network in the last few weeks, with frequent remarks of “I didn't know X was on Twitter”).
Jim Bumgardner (aka KrazyDad, author of O'Reilly's Flickr Hacks) addressed the “so what?” response to frivolous work in a blog post called Utility is Overrated a couple of months back. In the comments there's a comparison with Susan Sontag's Against Interpretation. In it, she states that “interpretation is the revenge of the intellect upon art”. I don't want to point at Twitter Blocks and say “art” (the Motorola sponsorship in particular makes that tricky) but I think that people are thinking too hard about things if they're looking for the “point” of it.
I'm not asking that people stop casting a critical eye over what's presented to them, especially when it's being hyped to death and it's commercially branded. It's fine to ask “what's it for?”, especially of new tools or things that aim to improve the efficiency or effectiveness of this or that. But why not also accept that some things might just be for entertainment and ask “am I having fun” once in a while instead of looking for a problem to be solved or an important statement to be read? Some things just are.
Sadly (due to my neglect as sys-admin) processinghacks.com hasn't been accepting new edits/users since I moved hosts a few months ago. I finally got around to fixing it today, so everything should be working now.
The site hasn't quite reached the ambitious heights I had in mind for it when toxi and I first put the wiki together and drafted the contents. Some of the hack suggestions are out of date now, and many of the existing hacks don't work with the latest release yet. That said, I hope to put some time towards kicking it into shape over the upcoming weeks and I'd be delighted if you'd join me.
I should also mention that there's a good incentive to get your hacks written up, because we're planning to move Processing Hacks to processing.org - you might have seen a note about that in the Learning section. Fame and fortune (or at least some geeky glory) await!
In the spirit of continuing our impromptu database week on Processing Blogs (my post, toxi's post, then Florian Jennet updating the sql library), I thought I'd post another quick example using Lucene.
Last week Ryan and I needed a reliable way to search inside a data set we were working with. I had previously tried and failed to write my own useful search routine for the same data, so I wanted to take a look at Lucene instead. (This week, I might have used SQLite, but I hadn't tried it last week!).
Lucene isn't a relational database like MySQL or SQLite, although it has a few similarities with the way most database engines speed up queries using indexes. That's because Lucene is "just" the indexing part. You tell Lucene about your data, one part at a time, and then construct queries to ask it which parts of your data match the query. The key thing is that you keep hold of your data yourself and structure it any way you like, Lucene keeps its own representation of the data for searching. Because of this it can index much more data than you can store in memory.
Anyway, I put up an simple example Lucene applet here that indexes the text of Time Machine by H.G. Wells from Project Gutenberg and lets you type queries against it. The text itself and the Lucene index it builds are both quite small (tens of KB, compressed), but the applet is around 750KB. This is because Lucene's core jar file is about 500KB so it's more suited to standalone projects and applications.
The code is kind of documented, but Lucene was really too much for me to understand properly in just one day. Nevertheless, again, I hope people find it useful!
SQLite is a great standalone SQL database engine - not ideal for every situation (particularly large websites), but more than good enough to have already made its way into desktop projects from Apple, and more recently Adobe and Google.
My colleague Mike recently used it as a way to distribute data from a 511.org transit website scraping project he's been doing. I wanted to see if I could download his data (gathered and processed using python) and access it using Processing.
Web searches for SQLite and Java (java wrapper sqlite, etc.) turn up lots of matches, including this promising tutorial from Tim Anderson, but sadly the most prominent matches didn't work for me. It's a real pain to get up and running and installed in a reliable way, mainly due to the need to compile SQLite natively for each platform and then talk to the native code using Java.
Enter this amazing project, a JDBC library for SQLite written in pure java that uses NestedVM to compile the C code for SQLite into something any Java VM can use. It's frightening to think about the amount of misdirection, abstraction and interpretation going on here, but it worked first time and plenty fast enough for my purposes.
Take a look after the jump for the code I ended up with. I hope people find it useful:
A few people have asked if I'm still using Processing now that I've joined Stamen (best known for their Flash work). Whilst it's true that I've been quiet on the Processing front, hard at work learning Actionscript 3 for Flash 9 to enable us to deliver Trulia Hindsight, much of that piece was informed by early sketches I wrote in Processing. (The graphs we made of a day of diggs were also made with Processing.)
Here are four movies we made from Trulia's data to get across the ideas we wanted to develop into Hindsight. It's a lucky thing that Flash 9 can shift many more points around the screen than Flash 8, otherwise we'd have been stuck. That said, it's still way behind Processing with OpenGL for this kind of visualisation, so choose your tools wisely when building a proof-of-concept in a different language to the one your project will be delivered in!
In the first few weeks of working with Trulia, we did some initial work exploring non-geographic views of their data such as tree maps and node graphs and so on. In the end though, the most compelling thing we came up with was to explore the different dimensions of the Trulia database in the form of animated maps.
Here are two movies, the first is San Francisco and the second is San Jose, showing the properties animated along an intuitive axis: the year they were built.
And here are two more movies, also made with Processing, that show the properties that were sold in the last 10 years (under $2m), this time animated by sale price:
The animations by sale price aren't available in Trulia Hindsight (yet) but we hope to work more on these less-intuitive dimensions for animation in the future.
No doubt whichever direction we go in next I'll still reach for Processing to try things out. It's the tool I find it easiest to think in, and although Flex Builder (based on Eclipse) is a great IDE, I still find myself wanting to bend Actionscript to be more like Processing when it comes to prototyping my ideas - it seems I'm not the only one!
If you're "someone who is a designer first and foremost, a coder a distant second, and who's interested in where these areas interleave" then please read the post on our site and consider getting in touch. I can't recommend us highly enough.