There is a simple yet very effective litmus test for outstanding software: if you can’t imagine working without it. Such wonderful pieces of software are rare and their value subjective, but they are, nevertheless, game changing for some people.

David has done it again: it’s called “Seek” and it’s a Thunderbird extension that adds faceted browsing to your email archives.

If you’re like me, you keep your inbox as your todo list and you move processed email into archives. This a very effective way to deal with both information and ‘getting things done’ but has one big drawback: my email archives tend to be a big, messy pile of stuff.

My archives (I call it ‘the vault’) are organized one per year, separated in received or sent. I use another thunderbird extension that maps the “Y” key to the “move the mail from the inbox to the archive” as in “done with it, get it out of my way”. My message-processing task ends when my inbox is fully read and all the messages that don’t require any further action are all archived in the vault.

This system works great (and I’ve been using it for years) but it has one little drawback: there are times when you need a particular information out of your archives and finding it is a major pain. You don’t remember exactly who sent it, nor when, nor the subject line… but you can certainly ‘orient’ yourself between various choices to get to it.

Problem is, email clients don’t generally provide “orienteering” interfaces to your email archives but a simple (or, at best, boolean) search box. Fact is: most of the time I don’t even know what to search for, or the search is so general that the amount of results is still overwhelming (no pagerank for your email, unfortunately, so we are back in altavista days or worse).

And this is where Seek changes the day: faceted browsing is a great way to allow you to ‘orient’ yourself into a pile of semi-structured information when you can’t define exactly (aka write into a text box) what you’re looking for. “I’ll know it when I see it”, it’s generally the feeling that drives me in digging into my email archives and faceted browsing is a tremendous help in that regard.

For SIMILE, we’ve been using and researching ways to apply faceted browsing for the last 4 years and Seek feels completely natural to me, both in the way it works and it the way it helps you drill down your semi-amorphous pile of email and emerge enough structure from it to be able to guide your navigation in it. So much that I can’t even imagine myself functioning without it.

Goes without saying, Seek is open source software and released under a BSD license. There is also a nice screencast that shows you exactly what you can do with it.

What are you waiting for? Stop searching and start finding.

… and this time is with “yours truly“.

Many thanks to Jon for making this happen.

Enjoy.

As a long time open development advocate, it’s rare for me to get surprised when a well-maintained project manages to quickly get enough traction to change the face of a particular sector but I got blown away the other day when I realized that, thanks to the catalyzing effort of OpenStreetMap, I had all the tools and data at hand to build a top-of-the-line hiking map for my hand-held Garmin GPS receiver, legally and for free!

But let’s start from the beginning.

I love hiking and climbing. I like sailing and surfing too but I would take a mountain over an ocean any day.

I also love exploring. I love getting to cities or forests/mountains I don’t know and try to find my way around without maps: navigation and orienteering is something I’m apparently really good at, at least relative to the people I explore with.

At the same time, having a hand-held GPS receiver is nice: not only it records where you’re been (which you can later use to geo-tag your images, if you remembered to make sure your digital camera’s internal clock is set right) and tells you how much distance you’ve traveled, but it can be useful in expanding your horizon by providing you a bird’s eye view of your surroundings.

While city exploring has a bunch of advantages (mostly strong cell-phone coverage which enables all sorts of things, public transportation maps all over, advertising with built-in maps), wilderness exploring it’s mostly a remote and off-line experience and that’s where an hand-held GPS receiver can be most helpful.

The problem with GPS receivers is that you go on your hike and there is absolutely nothing drawn on the screen but your current location and (for some good receivers) the heading/direction. Trying zooming in or out doesn’t help: the base map that comes with the device covers major cities, highways and state/country borders… in the middle of a national park you’re on your own.

At this point you have three choices:

1. go without a map and just record your tracks for your pictures and for your own track collection
2. buy a map
3. build your own map

I’ve tried #2 but maps are pricey (sometimes more expensive than the device itself, which feels completely absurd to me… razor+blades business model maybe?) and, worst of all, you can’t really tell if they cover the area you’re interested in (which gets worse when you hike in two different continents, 9000 km apart) and, most important, what features such maps have.

In fact, a trail map is not really useful for somebody like me that loves to explore and actually find my way to get from A to B… what I want to know is mostly places of interests (peaks, lakes, refuges) and, most important, the configuration of the terrain around me behind what my eyes can see from my current observation point.

Italian hikers know that the best maps you can find are the military maps of the Alps: they are incredibly detailed and, best of all, they feature very precise contour maps and come in digital form, but they are expensive and they don’t sell them online.

My holy grail hiking/exploring map should have points of interests, morphology (contours/rivers/lakes), access roads and trails… in that order of importance.

example of A Contour Map

The best way, of course, would be to build your own and I did research a bit into ways to do that, but I had to give up pretty early on: there is a lot of software on how to get data in and out of your Garmin but not a lot of freely available mapping data and, worst, there was no community where people would get together and share experience and tools.

Until OpenStreetMap came along, that is.

OpenStreetMap is, basically, a wiki for maps: the idea is that if enough people carry around a GPS receiver when they go places, upload their tracks to a centralized location, convert those tracks into features and agreed to license their work in a way that others can build upon (open source style), the result will be a world-wide, freely available and perpetually updating map of the world.

It may seem like a rather ambitious enterprise, but so was writing an entire operating system (GNU/Linux) or writing an entire encyclopedia (Wikipedia) and not only they were successful, but changed the entire ecosystem around them. This story is about how OpenStreetMap (OSM), only few years old, has already changed the game.

The first big thing to note about OSM is that the United States and the Netherlands are already fully covered… this is because map data in the US is public domain (because it was done by survey paid by the government with tax payer’s money and all that data goes automatically in the public domain by federal law) and the dutch data was donated by a mapping company.

This got me started: I wanted to see if I could export OSM data on my Garmin for Los Angeles (where getting lost without a navigator is pretty easy, especially if you miss your highway exit).

That got me to find a few programs: Josm, Osmosis and Mkgmap. The first is one of the official OSM map editors and it’s capable of getting OSM data directly from the central database for a particular region of the world and help convert your GPX track data into maps that you can then upload back to the server.

Here OSM already impressed me: every single point and relationship on the map graph has provenance metadata, which means that you can automatically know the people that are interested in the areas that you’re interested in and join forces. Social ties like these are incredible amplifiers of network effects, which, at the end, decide whether or not an open developed ecosystem ultimately strives or perpetually stagnates.

One feature of the OSM map web server is that it refuses to answer queries that return too much data and unfortunately, Los Angeles maps are so detailed that you have to make very small queries and tile them up all together.

Another solution is to download the entire “planet.osm” file (which is an astonishingly big 60Gb XML file!) and use Osmosis to slice it into smaller pieces according to various rules.

After creating a smaller OSM file from the planet one, you use Mkgmap to generate a “.img” file that you can upload to your Garmin (or, in my case, simply move it over to the microSD card with any SD card reader which is much faster and doesn’t require you to buy a USB cable for your Garmin, which is sold separately in the low end versions).

The latest version of Mkgmap can also get multiple “.img” files and merge them together: which is very useful because Garmin handhelds only support one map.

So, that is how I made a map of Los Angeles for my Garmin for free and legally, using public domain data and free software.

But I wasn’t satisfied: while reading the OSM wiki, I came across this other software named Srtm2Osm which claimed to be able to generate contour maps for any location in the world covered by the SRTM dataset.

I had no idea what SRTM was, so I looked it up and was shocked: I had no idea such a dataset existed! Basically, it contains a very precise measure of the altitude of any location of Earth between +60 and -60 degrees of latitude. It was done by NASA and, therefore, it’s in the public domain as well.

After more looking, I found a few scripts in the OSM SVN repository that did exactly what I wanted: sliced the SRTM dataset and generated contour lines. The interesting thing about this script is that the heavy lifting of contour generation is done by Gnuplot, so the script is basically a Perl glue between the SRTM data format and the format Gnuplot needs and then the output of Gnuplot and the OSM XML schema.

The contour map above was generated with those scripts and then rendered with Kosmos, a nice OSM map renderer written by the same guy who wrote Srtm2Osm (which, unfortunately, it’s .NET as well, so I had to run it in WinXP running in Parallels).

Then I used Mkgmap to convert the contour maps into “.img” and moved them over to the Garmin.

There are a lot of moving parts and poorly documented features (which is why I decided to write this post also for me to remember how to do it in the future) but I think that we have come a long way since the days ESRI was saying “jump” and all the poor digital geographers of the world could reply was “how high”?

Enjoy!

A few words of warning:

1. the “planet.osm” file is truly huge, I’m not kidding when I’m saying that it expands (after being bunzip2-ed) into a 60Gb file. But note: you need this file only if you care about OSM roads, cities and points of interests. If you care only about the contour maps, you don’t need this at all.
2. the SRTM perl script found in the OSM SVN repository generates OSM data in an old schema. I found this alternative script that generates OSM data in the latest schema and that is compatible with the latest OSM tools.
3. the mkcntr2.pl script requires Gnuplot version 4.2 or older. It won’t work with version 4.0 which is what Fink installs by default (took me a while to figure that one out, so I’m warning you). I had to download Gnuplot and build it myself but then it worked like a charm.
4. don’t try to make contour maps for big areas or gnuplot will suck a lot of your ram, maybe enough to strangle your machine, so try starting small and grow bigger incrementally.
5. don’t try to load the contour OSM data into Josm because it will be too slow to be of any use. And don’t try Osmarender either because it’s super slow (which is why I had to resort to run Kosmos in windows to see the results). If you can’t run Kosmos, it’s probably faster just to try it on your Garmin directly.
6. in the script “mkcntr2.pl” I had to change line 25 from 1000 to 9000 to make sure it worked past the 1000m altitude (no mountain on earth is higher than 9000m so you’re safe). If you want, you can play with those numbers and have only slices of your contours which might be useful in some cases.
7. don’t make the contours too close or you’ll have to zoom in a lot in your garmin to see anything useful (de-cluttering unfortunately doesn’t work with contour lines). I think 50m or 100m should be optimal but I still have to experiment with that.
8. look at the “example.sh” script to see how you should invoke the scripts, in what order and with what parameters.