Last week, I finally had the pleasure to meet Jon Udell in person (and impress him with our work at MIT).

One of the things that surprised him (who has been reading this blog for years) was how little I write directly about the results of my research job here.

What’s funny is that I thought that was really the only thing I wrote about, but he’s right: what I talk about are the questions, not the answers.

In a sense, my research works by wandering around in a huge solution space, stumbling upon questions, then finding answers, test them in software, release it to the public and observe what happens. Creating the software and observing the public reaction uncovers other questions and the cycle repeats.

Jon was astonished (probably because that’s one of the things he does best) to realize that in that cycle, once we have a particular answer to a question implemented in software, we don’t tell it explicitly, we kinda let it out in the wild, announce it, and assume that people will figure it out on their own why we did what we did. And it’s true: sometimes the thoughts that brought us to make a particular decision are obvious to us but not at all to others.

I’ve always been a bottom-up grass-root type of software builder (and a hands off, “build it and they will come” kinda guy) but I do agree that the world is full of examples where that’s not enough and I could make better use of my resources and dedicate a little time explaining some of the software that we came up with in the last few years.

And what’s best to start if not by eating your own dog food?

So, yesterday I spent a few hours and I pimped up this blog’s archives using two of our own javascript widgets, Timeline and Timeplot.

You probably came across Timeline already (it’s used in many places) but Timeplot is less known, mostly because it doesn’t (yet) work on IE (which means, that if you’re reading this blog with IE, you should at least try to load it with firefox to get the full experience of the archives).

Both Timeline and Timeplot are time-based pure-javascript visualization widgets, the first implemented by David and the second by “yours truly”.

While Timeline is more focused on visualizing individual events and periods at different time scales (and does an outstanding and very innovative job at it!), Timeplot is focused specifically on uncovering causality dependency between events and trends. Also, since Timeplot’s code is built on top of Timeline’s, it uses the exact same XML data file for events.

Here is how the timeplot of my blog looks like in Firefox today:

This timeplot is generated by ‘mashing up’ the event data about blog posts generated by this Wordpress Timeline plugin (which, unfortunately doesn’t work on Wordpress 2.3.1 since they changed the Category system, so I had to cut that part out to make it work) and that feeds the archives timeline with the trends data which is generated by a cron-ed Python script that mines my apache logs (and that I wrote previously for the SIMILE historical stats). Being all javascript, you can “view source” on the archives web page and understand exactly what everything is doing (and, even better, I can use Wordpress “theme editor” to tweak it directly from my browser, which is really nice).

The vertical gray lines are the blog posts, while the plots are trends color coded by type and plots on different scales (and without value references) to make you focus on trends rather than absolute values (which I find distracting for causality emergence).

The data can be ‘mashed up’ because both events and trends include time-based information and can therefore be correlated at that level. In the interactive version, you can mouse over a particular blog post and uncover which one it is: that is the first important aspect of Timeplot that is truly original, it allows to easily spot and correlate events and trends.

In the blog archive timeplot case, it’s somewhat obvious to assume that it’s the blog post to influence the trends and with that in mind, it’s intuitive to uncover ‘key posts’ that have a larger impact.

The most obvious to spot are the two recent blog posts that made relatively big traffic spikes are about color theory and Google’s Android Dalvik VM. The first was picked up by the del.icio.us crowd and made it to the ‘popular‘ page, the second was picked up by the blogosphere and made it into techmeme’s first page.

Another interesting fact that the timeplot uncovered is that during the 6+ months in 2006 when I didn’t blog, my readership somewhat shrank (which is only natural, as people eventually prune their feed subscriptions or news readers alter their update frequency based on the blog’s update frequency itself) both in volume of feed requests and in number of unique subscribers (which here means ‘how many different IP addresses asked for the blog feed each day’). Let’s see if you can spot at least another 2 examples of this happening in the plot.

But the nice part about such timeplots is that they give you questions, not answers, they are great inspiration for further inspection and analysis.

For example, the plot shows a local minimum for readership around jul/aug 2007 which seems to imply that the post around that time helped changing the trend. The post in question is about procrastination. The interesting thing about that post is that, while it received pretty good feedback about it privately, there is no indication that it got any traction on the web (for example, del.icio.us has only 7 bookmarks on it), so where did these people come from? I still don’t have an answer for that.

Let me close by quoting Pablo Picasso:

Computers are useless. They can only give you answers.

I fully resonate with that mentality but I translate it about software: there is a class of programs designed to give you answers and another, unfortunately still much smaller, designed to give you both answers and further questions.

I want to focus mainly on enlarging this last one and Timeplot is an example of that I’m proud of.

As some of you probably noticed, after 4 years of good service, I’ve decided to retire the cocoon-based blogging engine that powered this blog (codenamed linotype) and adopt somebody else’s code (WordPress).

When I first wrote Linotype, it had some unique and modern features, such as WYSIWYG in-browser editing and extensible metadata support, but the world caught up and surpassed it.

As soon as I was able to put my ego on the side and admit the above, the decision to migrate was relatively obvious: there is no reason for me to keep spending cycles on maintaining a software that has one user in the entire world (me), that doesn’t have any better feature than the blog engines already available as open source and with big development communities around them and that gives me a pluggable framework for a community (and myself) to add functionality to it.

But that got me thinking.

Unfortunately, not all migration decisions are that easy (even when ego is not involved): when do you know it’s better to stop pushing for a particular idea, technology, software, vision, give up, and change direction?

And what if the lack of traction is something that is ultimately depending on you and not on the environment around you?

There are uncountable examples of individuals achieving incredible success because of their perseverance and individuals achieving astonishing nothings because of their stubbornness. What is the difference?

And there are numerous examples of individuals abandoning a failing vision to succeed with the next or continue to quit them, unable to perceive the signs of growth.

Who’s standing on the shoulders of giants and who has giants on their shoulders?

Sun released their “free java” source code under the GPLv2 to both win the free software crowd and capture peripheral innovation and bug fixing from the community. For the java standard edition (aka “the cat is out of the bag”) there is an exception to the GPLv2 that makes it “reciprocal” only for the Java platform code itself but not for the user code running on it (or most people wouldn’t even dare touching it with a pole).

But such exception to the GPLv2 is not there for the mobile edition (aka “where the money is”).

This brilliant move allows Sun to play “free software paladin” on one hand and still enjoy complete control of the licensing and income creation for the Java ME platform on mobile and embedded devices on the other (because cell phone makers would rather pay than being forced to release all their code that runs on the phone under the GPLv2… or, in many cases, they can’t even if they wanted to as they don’t own the entire software stack).

Which is also why their TCK license they offer for free to Apache to certify Harmony (which is a ’standard edition’ implementation but could be easily turned into a mobile edition one) includes ‘field of use’ restrictions. It’s funny that Sun’s own CEO considers the very same ‘field of use’ restrictions ridiculous (but only when it is Sun to have such field of use restrictions applied on them, apparently), but Sun imposes them nevertheless and no open letter has been able to change their position.

This is the stall in which Google announced the release of their Android platform, which would be able to run Java applications on a mobile phone but it would also be released under the Apache License v2.

This raised more than one eyebrowse, and sure did make me raise mine: how did Google manage to get Sun to license off a platform that could very well kill their own?

Turns out, they didn’t: their move was even smarter than Sun’s.

Today Google released the Android code and I took a serious look at its internals… and found the solution for the licensing problem. It’s called Dalvik and it’s the new name of Sun’s worst nightmares.

Dalvik is a virtual machine, just like Java’s or .NET’s, but it’s Google’s own and they’re making it open source without having to ask permission to anyone (well, for now, in the future expect a shit-load of IP-related lawsuits on this, especially since Sun and Microsoft signed a cross-IP licensing agreement on exactly such virtual machines technologies years ago… but don’t forget IBM who has been writing emulation code for mainframes since the beginning of time).

But Android’s programs are written in Java, using Java-oriented IDEs (it also comes with an Eclipse plugin)… it just doesn’t compile the java code into java bytecode but (ops, Sun didn’t see this one coming) into Dalvik bytecode.

So, Android uses the syntax of the Java platform (the Java “language”, if you wish, which is enough to make java programmers feel at home and IDEs to support the editing smoothly) and the java SE class library but not the Java bytecode or the Java virtual machine to execute it on the phone (and, note, Android’s implementation of the Java SE class library is, indeed, Apache Harmony’s!)

The trick is that Google doesn’t claim that Android is a Java platform, although it can run some programs written with the Java language and against some derived version of the Java class library. Sun could prevent this if they had a patent on the standard class library, but they don’t and, even if they did, I strongly doubt it would be enforceable since Android doesn’t claim to be compatible (and in fact, could very well claim that their subset/superset is an innovation on the existing patent and challenge Sun’s position).

Not only this allows Google to avoid having to battle thru the JCP for any change to the Java ME “standard” or tolerate Sun’s unique ability to veto any JCP change, but gives users a much more ‘fresh‘ and modern class library to play with and built-in hooks to all the hardware features of your phone, including things like OpenGL, bluetooth, USB and all those things, which don’t come with the standard Java ME and therefore cannot be taken for granted by the developers.

So, here we are: Apple makes the iPhone, incredibly sweet, slick and game-changing and yet incredibly locked. Google makes Android and not only unlocks development abilities on the mobile phone but also unlocks millions of potential Java mobile programmers from Sun’s grip on it.

Plus, they’re targeting 3G phones, not the crappy 2.5G that the iPhone supports now.

Screw the iPhone and screw Java ME with all its profiles, midlets and the stupid requirement to crypto-sign your application to run on your own phone: I can hardly wait to get my hands on hardware that can run Android… and if they can’t support multi-touch out of the box because Apple owns patents on it, I’ll download the patch that enables it from a country where such nonsense doesn’t apply.

Update and Errata (a few things worth pointing out):

• The Android SDK does not compile your Java source code into Dalvik’s bytecode directly, but it first uses a regular java compiler to generate regular java bytecode (say, javac or the built-in Eclipse compiler) and then converts that bytecode into Dalvik’s bytecode (the “dx” tool does this: converts .class/.jar into .dex files). Still, the substance remains: there is no need to ship a java virtual machine on your Android-powered phone and you can use your regular Java standard edition to develop your phone application (means, you don’t need to use Java ME anywhere at all).

• Google has released the binary SDK for Android and only the source code for the examples. It did indeed contribute patches to the Linux kernel to support 3G Qualcomm chipsets (under the GPLv2, so that it wouldn’t create a impedance mismatch in the kernel ecosystem) but it has not, to this day, released the source code for Android itself even if it has explicitly claimed in their announcement that they will do that in the future and under an Apache License v.2. I strongly doubt this will include source to every single piece of the Android middleware (the video codecs seem the hardest part to relicense) but I really hope that the Dalvik VM itself will be open sourced or a lot of the appeal of this platform will just vanish.

• A good friend reminded me that there is no such thing as “routing around” IP licensing restrictions and that bigco routinely use shake-up moves like these to re-establish alliances or to call other bigco’s bluff. It is worth nothing that Sun has never made available a public list of the IP they own and that you need to license in order to legally be able to run Java (personally, I think they don’t know themselves: such things are really tricky to figure out, even internally). All their licensing deals (including the one with Microsoft) have been broad and very general and there is no way for us of knowing if Android infringes even just one of the patents that Sun owns around the Java platform (my friend suggests java’s internal security manager, which is clever and original, but that depends on whether Dalvik supports the same model or a sufficiently different one to stand in court).

• The Apache License v2 forces the contributor to license the IP along with the code but only the IP that the contributor owns (you can’t give away stuff others own, obviously). This means that if Android does, in fact, use some of Sun’s IP, it is entirely possible for Sun to sue any hardware vendor that ships Android with their phones and prevent them from shipping. Also, Sun could knock at your own door and ask you to license the Java IP that you need to be able to run Android on your phone (in case, say, you bought an OEM phone and installed the software yourself) even if there is no trace of ‘java’ on your phone memory.

• Sun is under fire for IP issues related to ZFS and they claimed that it’s an “attack on free software”. I’m curious to see how they’re going to spin their own IP attack on Android (because I bet there’s going to be one).