People that know what Freebase is think of http://www.freebase.com/, but they probably don’t know that we get a lot more traffic thru our web service APIs that expose the querying interfaces (both for reading and writing). Freebase.com (or ‘the client’ as we call it) is only one of the many applications that use Freebase data for their own uses.

Notable other Freebase data consumers are Microsoft’s new search engine Bing and The Wall Street Journal, to cite a few.

But we wanted to make it really easy for people that are interested in particular data domains not only to store, cleanup and share the costs of maintaining their data, but to create highly tailored web applications that can present the data precisely the way users want and offer mashup and writing services that make such data useful.

We offer a wide variety of ways for people to use Freebase from inside their applications:

1. a set of RESTful web services
2. a set of libraries for many programming languages that you can use to simplify the job of calling those web services
3. a web application hosting service called Acre that allows you to write and run your web applications directly on Metaweb’s server infrastructure

My work has been mostly on writing Acre itself and writing Acre apps to show off its functionality or to implement useful tools with it, so I want to focus on that part of the platform.

Those of you who know me from my Cocoon past know that I’ve been very interested on the use of Javascript on the server side since at least 2003. Cocoon pioneered the use of web app control flow and native continuations in server side javascript, but its XML-heavy skeleton made it hard for people to perceive its value in so much complexity.

Which is why I later wrote Butterfly, a slimmed down version of Cocoon’s flowscript engine that is currently used in production in one of MIT’s online projects but never reached wide adoption (also because I never really pushed hard on spreading the word about it).

When I interviewed at Metaweb and Nick (of LiveConnect fame) showed me Acre (which he had started) and it was a pleasant surprise for me to see so much alignment and overlap there was between Butterfly and Acre, even if Acre was considerably more ambitious: while Butterfly was a regular application server running trusted code, Acre was an entire hosting environment, designed to run unsafe user-contributed code on Metaweb’s machines and handling thousands of virtual hosts and web applications at the same time.

Both were written in Java and used Rhino as the execution engine, but Acre was designed from the ground up to offer the ability to host potentially malicious code and isolate its execution securely, both in terms of what actions could be performed and what could not, but also how much server resources each request could consume before being considered abusive and be autonomously terminated.

User code in Acre runs thru three layers of sandboxing: Rhino first presents the user code with a carefully restricted API set that severely limits what the user script can do (just like the browser does with web pages), Rhino then compiles the script into Java bytecode which is then interpreted and JIT-compiled by the Java virtual machine, which is then executed under an unpriviledged user by the OS.

Two things the JVM didn’t offer and we had to introduce ourselves: time and memory quota for the execution of a thread.

Acre is run on a cluster of specially patched JVMs that provide it with special resource-limited threads that can be set to ‘expire’ if the code they are executing either runs for too long or consumes too much memory (either directly or thru a native call). This is done to prevent malicious users from attempting DoS attacks to the Acre cluster by running code that either never ends or that keeps consuming the system memory without releasing it.

I’m very proud of the work that my team and I did on Acre, especially because of all the very cool applications that were written on it, from data games to domain-specific visualizations, from social network add-ons to utility applications, from exploration tools to powerful libraries integrations, and the wonderful experience of writing a software that becomes part of your daily toolkit.

I’m particularly proud of Acre’s native support for OAuth which Alex and I designed and then I (painfully) implemented. Getting OAuth support to one single line of code was not easy, but we did it and it’s so simple to use that OAuth is the default conduit that Acre uses to talk to Freebase, making all Acre apps de-facto citizens of the web of distributed services.

I’m also very happy about not only the engine design and the server infrastructure that can allow us to scale horizontally with ease, but also of the entire surface with the user, from the browser-based development environment (the Acre App Editor) to the extensive documentation hub, from the introductory screencast to the application gallery that showcases what everybody is doing in the ~900 applications that Acre already hosts.

So, without further ramblings, point your browser to http://freebaseapps.com/ if you want to know more and start from there: my suggestion is to watch the screencast that will introduce you acre and show you how to build an app with it.

Feel free to email me at ‘stefano at metaweb.com‘ in case you need help or you have suggestions or criticism: while very exciting, we know this is only the beginning and we’re thrilled to know about ways to make this even better and more useful.

Enjoy!

Or you think Freebase is cool and all, but you have no idea how you could make use of it or why you should care in practice?

Or simply you’re curious about what I’ve been working on for the last 9 months that involves something so tricky that a standard java virtual machine wasn’t enough and we had to patch it and run our own modified version in production? (yeah, I know how crazy that sounds.. believe me, I know)

So, if you’re reading this and find yourself close to downtown San Francisco Saturday July 11 2009, consider showing up at the free Freebase HackDay event that we are organizing to show and promote all the developers’-related activities we’ve been working on to make Freebase a useful and interesting platform to integrate with your web applications.

This day is also very important for me because we’ll be releasing Acre 1.0, our server-side javascript-powered hosted web application platform, that my team and I been working on since I joined Metaweb.

The event is free and Metaweb will be offering food, drinks and plenty of whiteboards and wireless connectivity, but if you plan on coming, please RSVP.

See you there!

I think this is a very important milestone: not only the new site sports a completely redesigned (and fresher, if you ask me) style and a special logo, most importantly, it outlines Metaweb’s renovated focus to go after communities of data enthusiasts, rather than trying to entice anybody in being a data contributor.

There are several good reasons for this change of direction:

1. reducing data agoraphobia – it might start by feeling mighty powerful to be able to access and edit anything in a database that contains 5 million different topics, but it turns into scary pretty fast. How do you know you’re not doing something bad? That you’re not damaging somebody else’s application that depends on this topic having this type set even if it’s totally empty and feels out of place? The best description I heard of this feeling is the ’stadium effect’: editing Freebase sometimes feels like walking in a stadium full of people to do your thing. Unfortunately, not exactly ideal to entice volunteer contributions.
2. one size does not fit all – one of the results of solutions that want to be general enough to apply to a vast variety of problems is that they tend to look so flexible they tend to feel boring and built out of compromises. Freebase is only one way of interacting with the underlying data that Metaweb hosts (which is why it’s called ‘the client’), but it’s clearly the most visible and for that reason the most important. But one size cannot fit all comfortably, and bases are the first step (of more to come and currently in progress) toward giving the communities of data enthusiasts the tools to personalize the way the data they care about is presented, viewed, edited, controlled and ultimately used.
3. increasing relational density – both for data and for the social network of the contributors that care about it, the value is not necessarily in the quantity of nodes in the graph, but how well connected these nodes are to one another. I strongly believe that relational density is a much more meaningful indicator of data quality (and value!) than others, like quantity, completeness or coherence. It used to be an heretic position in the data community, but times are changing and it’s all very intriguing for me since I’ve been researching in this space (how to mine and emerge insights and information from highly dense relational networks) for many years. To me, bases represent an important tool to increase relational density both for the data and for the social network of contributors.

Very exciting times.

But stay tuned because other announcements are in the pipe and coming real soon!

Also, if you happen to be in San Francisco next Saturday (Nov 09, 2008), come to the Freebase HackDay, it’s free and you’ll get to see a lot of shiny new data toys.