People do crazy things before they get married

Note: embedded video below… if you don’t see it, click here.

My mental model of creativity follows the one on research and exploration: given a (probably very multi-dimensional) solution space for a given problem, the act of creativity, research and exploration is to find a particular spot in that solution space (aka a solution) that minimizes/maximizes a particular metric on that solution space.

A geographical example would make this easier to understand.

Imagine to find yourself on a unknown geographical location as an explorer and your goal is to find the highest peak of the whole planet. You are given a GPS compass that tells you exactly where you are and the altitude of your current location but no map.

Now imagine that it’s foggy and your visibility is about 50 meters (~150 feet).

This is how creativity and research feel to me.

Here, it would be already interesting to discuss the various strategies that one could adopt to find such peak (hill climbing, brownian motion, levy flight, random waypoint) [see also Ben's post on the subject].

But there’s another dimension in the exploration strategy that I haven’t taken into consideration until recently: the impact of interruption frequency.

Imagine yourself again in the geographical exploration scenario I described above, but now there’s a twist: Captain Kirk is using you on a multi-planet search and is a little ADD so he’s telling Scotty to play tricks on you and teleport you randomly, in the middle of your search, onto another planet.

Now, your goal is still to find the highest peak but now on each of the planets you are teleported on. And without the ability to take notes, and without knowing when you’ll be teleported or where ahead of time. The only active thing you can do is to report what you consider your best finding to Captain Kirk and have Scotty teleport you back on the planet where you were last teleported from to continue your search there.

Note how hard this search is: the planet is huge and you know you’re not going to find *the* highest peaks of all but others are searching too and you’ll be rewarded based on how higher your findings are compared to others’.

The act of being teleported models in the geographical search scenario the effect of interruption during a creative exploration.

The overall game gets even more challenging if the tasks change depending on the planet you’re teleported on: on one you have to find the highest peak, on another the lowest canyon, on another find water, on another find a particular flower and so on.

The overall effect I’ve noticed on myself as the rate of interruption gets more frequent and the tasks to solve more diverse, is that my overall ability to accomplish things increases but my performance on the single problem solving degrades dramatically. Also my frustration level after each interruption grows very quickly (further impacting my ability to perform an optimal search strategy)

And the overall emotional effect is overwhelmingly disappointing as one tends to measure not the sum of all successes (which is probably higher), but the variation on the single one relative to the optimum that I have experienced in the past on a single task solution.

The best way to model this in the multi-planet geographical exploration problem is to add fog to each planet’s athmosphere with a thickness what is proportional to the frequency of teleport jumps; the more you get teleported, the less you’re able to see around you, the worse your performance in estimating whether or not you have reached an optimal point becomes.

In reality, what I think it’s happening is that the interruption-induced context switching drains a lot of the resources and effects dramatically the ability to perform research strategies that are more complicated that a simple hill climbing. This forced context switch is also dramatically stressful as lots of brain activity needs to be channeled into making sure that the context switch happens at the speed that the person interrupting you wants you to act.

It is critical to understand that the interruptor does *not* need to switch contexts, only the interruptee.

I suspect that it is not all that true that people tend to get less inventive and creative as they get older (as it’s generally thought to be true). What happens is that their previous successes cause their interactivity levels to grow and, as a natural consequence, their interruption rate to increase and their search/creativity strategies to degrade.

What is very interesting to note is how much our society and even human resource management tend to completely ignore the impact of interruption frequency in the environmental effects of people’s well being and productivity levels, especially in creative jobs.

This why I think that working from home has tremendously beneficial performance impact on my work: even when I lived in Cambridge and MIT’s campus as a 10min bike ride away, I would stay home to work, without phone, IM, IRC or email to read papers and draw on whiteboards.

Similar is how many people report getting more work done on airplanes rides in 8 hours than in two months at the office. We’ll see how this effect gets erased as WiFi gets more and more available on planes.

Considering ‘attention’, ‘focus’, ‘creative flow’ as valuable resources that one has to protect, maintain and fight for is a new trend that is slowly emerging and that I’ve always understood being very important.

Unfortunately, sometimes life’s events overthrow that equilibrium and as interruption frequency increases, creativity performance drops, resulting in a sense of failure and emotional debilitation.

It’s only partially comforting to know that this is just a fact of life and that it’s not a personal deficency.

But it’s a constant struggle to find better ways to work and interact with others without increasing the ‘fog levels’ around you too much (and to balance your needs for focus with other’s needs to interrupt you).

In game theory, a Nash equilibrium is a state in a two or more persons game in which no player has anything to gain by changing only his or her own strategy unilaterally.

Basically, when two players can’t or won’t coordinate (for lack of ability to communicate, luck of trust, whatever) and rely only on maximizing their immediate personal benefit, Nash has shown that there is always a state in such a game that is at equilibrium, meaning that none of the players, acting selfishly would want to change it (as it would reduce their payoff).

The very interesting thing about Nash equilibrium is that such state it’s not necessarily the one that maximizes the payoff that the players would obtain if they had coordinated.

One classic example of this is the so-called “prisoner’s dilemma“, a situation in which two prisoners can decide to be silent or betray the other and obtain different payoffs depending on their choice and the other persons’ choice, but without the ability to coordinate.

If both prisoners stay silent, they get 6 months, but if one stays silent while the other betrays, the betraying one goes free while the silent one goes to jail for 10 years. If both betray, they get 5 years each.

The best outcome is for both people to remain silent, but the selfish drive (and therefore what makes the ‘both silent’ state optimal but imbalanced) is to betray since even if the other betrays as well, 5 years is less than 10 (the situation where you trusted the other person to remain silent but he betrayed you).

It is worth nothing a few things here:

1. if the prisoners get 10 years in jail both if they both betray and if one gets betrayed while remaining silent, the ‘both silent’ state becomes both optimal and at Nash equilibrium.
2. we can therefore infer that the optimality and nash equilibria of a particular game are not only a function of its rules, but of its paybacks as well.

This means, at the very least, that Nash stability and optimality of non-coordinated strategies can be influenced by tuning the paybacks without altering the rules of the game.

Now, think of distributed and decentralized data modeling efforts as a non-cooperative game: the players involved that require to model their data for their own use will try to do so in a way that minimizes their effort and maximizes their benefit. If both modeled their data in the same way, they could reduce their data integration costs, if not their integration costs will be substantial. The problem is that it’s hard for the players to predict the cost of each integration and, most importantly, their need for a particular one, especially as the number of players grows large.

With absolutely no coordination, the only Nash equilibrium of such a system is the ‘babel’ state: everybody does their own thing.

On the other side of the spectrum, the state that would globally minimize integration costs for everybody (which is, everybody using precisely the same way to model data) is not at Nash equilibrium, as individuals would perceive that improving their immediate modeling needs would increase their (easy to predict) immediate payoff more than it would increase their (hard to predict) future integration costs.

Note how the above is basically saying that no matter how descriptive, complete, well-thought-out and encompassing your data model is, it’s usage won’t be at Nash equilibrium, which will naturally bring diversity, dialects and changes into its uncoordinated usage.

It is critically important to realize that this is *not* a function of the quality of the data model, but it’s a function of the difference in difficulty in predicting the immediate present benefit against the future integration ones. Thus, spending more time polishing the data model won’t make any difference in the outcome as it diverges to reach Nash equilibrium.

One of the things that irritates me the most about the semantic web and its advocates is the naive presumption that optimal states in non-coordinated data modeling systems are necessarily stable and therefore will happen naturally.

While this was the case for the web (where selfish decentralized activity brought both local improvements and global ones at the same time and it was therefore relatively easy to bootstrap), this is not the case for the semantic one; this fact is often called the “chicken and egg problem”.

Many (including myself) have tried over the last decade to solve this bootstrapping problem by forcing existing data to surface, hoping to catalyze activity and applications that would further push for more data to surface and for more applications to exist.

But one thing that I’ve come to realize recently is how surfacing data might not be enough to bootstrap an autonomous system if we don’t find a way to align the Nash equilibria and the optimal states of the distributed data modeling and integration game.

What this means in practice is that we must find a way to tweak the paybacks of the data integration game (which is clearly a non-zero-sum game) so that its Pareto optimal states are also at Nash equilibrium (a thing that the Prisoner’s Dilemma shows it’s far from granted).

I personally think that Exhibit and Potluck are the best examples out there of solutions that don’t specifically change the nature of the game but shift the paybacks, thus attempting to reduce the gap between Pareto optimal states and Nash balanced ones.

A lot more has to happen on the Potluck front, of course, being practically just paperware and a lot more has to happen about harvesting the collective intelligence of people using these tools, to further improve on their use and emerge data that could be useful to increase coordination and make it easier to predict integration costs.

We are still far from solving the bootsrapping problem, but one thing is clear in my mind: exposing a bunch of data as RDF (no matter how well inter-linked and how many URIs can be dereferenced as URLs) is not going to be enough without a deeper and more serious analysis of the socio-economical dynamics around data modeling and data integration.

“Build it and they will come”, this time, might not be enough.