In his book review of Wikinomics by Don Tapscott and Anthony D. Williams, Venkatesh Rao discovers the power of open innovation and the culture of collaboration. The book takes a deep dive into the world of technology, virtualization, and open innovation, showing readers how these concepts shape the future of business. From being open and working with industry standards to embracing self-organization and thinking beyond win-lose mindsets, the principles of wikinomics offer valuable insights for success in today's rapidly changing world.

Wikinomics by Don Tapscott and Anthony Williams

Despite the name, which suggests both a me-too jumping on the Levitt/Dubner Freakonomics bandwagon and a possible reductive identification of all evolving Web technologies with wikis, this is a surprisingly good book, written at a calibrated level of abstraction, with a tasteful blend of concepts, anecdotes and statistics. It has none of the anecdotes-of-a-gunslinger-economist machismo of Freakonomics, and the wiki in the title is synecdoche, not reductive imagining.



I read the thing in one sitting at Barnes and Noble a few months ago, taking notes and annoying the staff (I conspicuously displayed my coffee cup to show I was legitimately avoiding buying the book — I now have a preprint copy via a colleague who knows Don Tapscott personally). That I could do so tells you that the content was pretty light reading. It was concise and valuable though, and about as deep as you can reasonably expect a first-to-market book on the topic to be. It’ll be a few years before we see the dense, canonical and empirically-validated book we all really want, so until then, this book is a pretty good operating manual. You cannot divorce the themes of the technology of virtualization and the culture of open innovation, and this book proves the point. It is as much about open innovation as it is about things like wikis and blogs.

Key Points

Few authors summarize their key points comprehensively in the introduction, so I always appreciate it when a writer does that. You can actually get away with reading just the introduction if you want 80% of the value and are willing to accept the assertions as best-faith truths. For good measure, the conclusion reiterates the same points with some added depth. The summary is presented in the form of a few “principles of wikinomics” that almost constitute a doctrine:

Being Open: maintaining porous organizational boundaries, working with industry standards, moving to a culture of non-secrecy, above and beyond legal needs, and being open to globalization and global trade. Well argued.
Peering: This uncomfortable neologism is meant to capture the themes of encouraging self-organization and working with non-hierarchical organizational structures where possible. Moderately well argued.
Sharing: To move from a win-lose business-is-war mindset to a mindset where there are win-win/coop-etition options available. The key idea here is to think about business IP as a managed portfolio that includes things you give away (including to competitors) alongside things you protect with patents or hide with secrecy. This is a slightly more elaborate conceptualization of the IP market than Henry Chesbrough’s in Open Innovation which I’ll get to.
Being Global: This point is made a little obscurely, and I didn’t quite see where it fit in with the rest of the picture, but never mind. One must counter Tom Friedman’s annoying Flat World metaphor where possible, so this might just be called community service. There’s some resonance, so let’s leave it there.
Generation-N Norms: The idea being that this whole culture shift is being driven by Generation N (his word for Baby Boomleters) norms, that include speed, freedom, openness, innovation, mobility, authenticity and playfulness. Yes, I too gagged a little at the primary-colored bubblegum utopia-ness of it all, but to be fair, the authors do justify the list throughout the book with substantive examples. I can just see skeptics translating that to sloppiness, lack of accountability, competitive carelessness, lack of execution discipline, ADD, disloyalty, political naivete and silliness. That set of dichotomies is a whole other post.
Highlight-Worthy Themes

Some highlights cherry-picked for your edification. There’s a chapter-wise summary after this section, for those of you who must have completeness.

Goldcorp

Rather smartly, the authors chose to start the book off with a success-story anecdote from a very traditional industry. This is the story of Goldcorp, a gold-mining concern in Canada. Goldcorp was faced with diminishing yields from its mines and an unclear idea about where to dig for more gold. The company’s traditional staff geologists failed to make much progress with their safe, ponderous way of exploring as they’d always explored before. The CEO made an incredibly bold decision to throw open the exploration problem to everyone, making all the geological data available to all (apparently unheard of in the mining industry, where this data is held very closely). An enthusiastic army of outsiders, ranging from computer scientists to mathematicians went to work (along with lots of outside geologists too, of course) and came up with a huge number of suggestions for places to dig, half of which the in-house staff hadn’t thought of. The company went from $100 million to $9 billion in the space of a few years (this was 2000-2004 I believe). Remember the famous story in The Wisdom of Crowds (a review coming up, not to worry) about the US Navy finding a sunken submarine? This is that story on a larger scale.

Stepping back, the whole darn book has this flavor. Yes, there are chapters covering Linux and Apache and Wikipedia, but repeatedly the authors emphasize the principles and non-traditional examples. They clearly know who they are trying to pitch the ideas to, and it is NOT the Linux choir.

Coase’s Law

One of the new things I learned was Coase’s law, which can be informally stated for our purposes as “Organizations exist in the form they do because of the relative costs of information per transaction” (my wording). Specifically, every action that adds value on a product/service’s path to market involves a search cost (finding someone to do/make it), a transaction cost (the cost of making a deal with the producer via negotiations) and the actual cost of doing the work.

It is a neat phenomenological law to help think about the dynamics of creative destruction in the sense of Schumpeter. Traditional organizations are large not just to achieve economies of scale, but also to ammortize the first two kinds of cost over many instances of the third kind of cost. The reason this is important for Wikinomics? When the search and transaction costs plummet (think Google+eBay or eLance), tasks can drift around in the economy, in and out of porous boundaries, to find their least-execution cost homes. The implication is that traditional organizations are structurally irrational in the present day, and mostly need to shrink to achieve their right size to reflect the new cost-of-information dynamics. The leaking function gets out there into a brave new frontier of intermediate markets.

Global Plant Floor

Okay, so we’ve all heard the classic open-source success stories, ranging from Linux to the Wikipedia. Can a similar philosophy build physical products? Can you create an open-source Boeing 747 purely through a mob refining a design on the public Internet and using commoditized manufacturing shops for fulfillment?

Second question. Can the process truly invent new ideas? After all, the biggest open source successes are clones or minor variants of previous commercial products, not totally new ideas in software — yes wikis and blogs are cool ideas that could be said to have open origins, and yes, Richard Stallman did invent emacs, but still, there isn’t a compelling radical innovation story in the open source world.

The book offers little insight into the second question, but for the first question, the book provides a tentative answer: Yes. China’s motorcycle industry (Lifan) is offered up as one example (though it is a swarm economy that mainly rips off Japanese original designs — innovation without invention). The more impressive example is the Boeing 787, contrasted with the Airbus A380, which is a logistics and manufacturing planning disaster (and possibly a design integration and multicultural disaster as well). One way to assess the A380 is to call it a disaster of waterfall planning of the Windows Vista variety. Contrasted with this, the Boeing 787 dreamliner has been developed with a very carefully calibrated open-market approach where the entire supply chain of subcontractors has been brought into the design phase with a “the guys who will manufacture it should design it” philosophy. Boeing has only retained the critical-to-design features like the tail fin.

This would be an impressive example if the 787 were already in the market, beating the A380 hollow. We’ll have to actually see how that battle plays out.

The critical value drivers of the ‘global plant floor’, per the authors are:

Orchestration (what Boeing is doing)
Rapid iterative design processes (obvious for software, but hmm… maybe we don’t need a decade long gestation for complex hardware systems either).
Modular architectures (the The Age of Modularity is now on my reading pile)
Build an innovation ecosystem (let’s all develop Amazon S3 clones)
Share costs and risk with partners
Wikinomics Design Principles

Now this is another interesting list from somewhere in the book. I’ll defer a discussion to after I post about disruptive innovations and open innovation, since deconstructing this list requires analysis beyond the scope of a book review.

Take cues from lead users
Build critical mass
Produce an infrastructure of collaboration (open standards, legal foundations)
Make sure value is shared by all prosumers
Abide by community norms (the N-gen ones?), including IP sharing.
Chapter Summary

After the introduction chapters, there are 8 chapters, each with a cutesy name. The overall structure is one of ‘Let’s collect the high concepts’ — don’t expect a business model recipe. In brief:

Peer Pioneers: Covers Wikipedia, Apache and Linux. Some new points here, even if you’ve already heard these stories
Ideagoras: Play on Pythagoras, I presume, is about open economy broker outfits like Innocentive (biochemical industry), Nine Sigma and a bunch of industry leaders. Some interesting statistics here: Proctor and Gamble discovered it used less than 10% of the patent IP it generated. Now it gets 50% of its IP needs from outside and cheerfully trades stuff it doesn’t use. IBM gets $1 billion from IP sales. More when I talk about Open Innovation.
Prosumers: Includes coverage of Second Life, the “hack this product” story from Lego Mindstorms (they tried to close the API and sue hobbyists who tried hacking it, and then realized that was actually a good thing and opened the product up). Steve Jobs’ iPod is in there too, as a story with mixed lessons.
New Alexandrians: Is an attempt to get at the research culture of Wikinomics, and there is a discussion of Intel’s university-lab strategy, as well as arXiv (the physics preprint clearing house), MIT’s open courseware, Merck releasing 15,000 formulas and the human genome project. This is a weak chapter — there is a rather forced air of trying to fit too many not-quite-appropriate examples into the conceptual framework.
Platforms for Participation: This covers the open-API+innovation-ecosystem-around-us represented by Amazon web services and the Google API. Also includes a discussion of the crowd-coordinated work in the aftermath of Katrina and the opening up of that famous piece of closed software, SAP.
Global Plant Floor: I covered this as a highlight, so let’s ignore this one.
Wiki Workplace: The money chapter for most of cubicle dwellers, but with surprisingly little value for understanding how the workplace is changing (either for workers or management). You want to understand wikis and blogs in your daily world, use them there and find out for yourself. Books full of ambitious abstractions is the wrong way to learn that game.
Perfect Storm: Ties the whole book together and weaves all the themes into a narrative of looming seismic shifts in culture. Hard to summarize this chapter. Suffice it to say, this will be the watershed chapter. You’ll either walk away saying the chapter is too timid in its predictions, or believe it is all hype.
The book has its own website, with some sample chapters, in case you want to get the story straight from the source.

Book Review: Wikinomics

Venkatesh Rao delves into the fascinating world of conceptual thinking and prototypical instances. He poses the question of why we tend to choose certain instances as prototypes and explores how these prototypes shape our arguments and definitions. Using the example of the concept of a triangle, Rao takes us on a mental journey to understand the role of "prototypicity" in our thinking processes. This article provides a unique perspective on how our choice of prototypes influences our understanding of abstract concepts and the information we derive from them.

We think about abstract concepts in terms of prototypical instances. These prototypical instances inform how we construct arguments using these concepts. At a more basic level, they determine how we go about constructing definitions themselves. Prototypes pop up in all sorts of conceptual domains, ranging from “war” to “airplane” to “bird.” So how do prototypes work in our thinking? Let’s start with an apparently simple example — the concept of triangle — that can get tricky really quickly.


If I asked you to draw a triangle, you would probably draw one that looked something like the one below, a scalene triangle, almost certainly drawn with the longest side as base and obtuse angle, if there is one, on top. Call this a prototypical triangle, understood as the sort of instance most people would draw. Why we draw such instances is the question of interest here. Let’s exercise this instance in a simple argument to see what role “prototypicity” plays in thinking. We will convince ourselves of the validity of the formula for the area of a triangle — half base times height — through mental visual manipulations.

Prototypical triangle

 

Imagine a line dropping from the top vertex vertically down to the base. This line enables you to visualize two right triangles. Now imagine a copy of the triangle on the left being rotated clockwise 180 degrees. Position this imaginary triangle so that you now have a complete rectangle on the left. Repeat the process for the right. The two imaginary rectangles now form a larger rectangle. The area of this rectangle is the product of the base and height of the original triangle. Since you constructed this rectangle by copying, rotating and pasting two triangles that exactly covered the original triangle, the original triangle must have an area given by half the product of the base and height.

You used imaginary visual manipulations to convince yourself of the formula for the area of a triangle. Now ask yourself, why did you not start with any of the following set of perfectly legal triangles:

Non-prototypical triangles

We have here an isoceles right triangle, an isoceles triangle, an equilateral triangle, a long, skinny triangle, a straight line segment and a point. The first 3 cases possess symmetries and the last two contain degeneracies.

Now work through the visual proof of the area of the triangle for each of these cases. The last two are the easiest: the answer is zero by examination, so the formula is trivially correct. More importantly, the answer does not tell us much — any constant instead of 1/2 would work, so we have validated a non-unique candidate. The first three, I assert, are also degenerate. Not in terms of their explicit geometric structure, but because the visual proof of their satisfying a particular asserted property — the area formula in our example — collapses to a simpler case than in the scalene triangle case. Recall your mental manipulations for the scalene triangle. Can you see how there are fewer steps in convincing yourself of the truth of the area formula? In each case, a single cut-and-paste visualization suffices. The scalene triangle proof visualization works (if inefficiently) for the symmetric cases, but the reverse is not true.

So one answer to “why do we choose as prototypes the instances we do?” is an information-theoretic one. A prototypical instance of a concept is one that contains the maximum information that an entity satisfying the definition possibly could. We are naturally inclined to work with the richest-information-structure case. In the case of the explicitly degenerate cases, the information poverty showed up in the non-uniqueness of the formula that was satisfiable. In the more subtle cases with symmetries, it showed up in terms of the degeneracy in the proof construction which would not work in the general case.

That doesn’t explain it all. What about our “long skinny triangle” which is close to degenerate, but not strictly so? Why didn’t we draw something like that? I suspect this has to do with the precision of comparisons we need to make when we mentally manipulate geometric figures: we want enough asymmetry and non-degeneracy to clearly illustrate the information capacity of our concept, but not so much that the precision required of the representation is too high. I am not completely happy with this hand-wavy account, so I’ll revisit this when I come up with something better. If you have a better account right now, post a comment.

A final point: why did we choose to draw our original scalene triangle with the longest side as a visual base? One proximal reason is that the necessary manipulations require more effort if we were to draw it with, say, the obtuse angle at the base (try it). A less obvious reason is that we operate with orientational metaphors that determine notions like “up” and “down” when dealing with abstractions. These metaphors inform both our language (“base” and “height”) and probably explain why non-standard orientations are mentally harder to work with, even though the explicit visual-proof steps are orientation agnostic. These conceptual framing metaphors will come up later when I talk about George Lakoff and his work on metaphor, so I’ll defer discussion of this aspect of prototypicity to later.

When we move from instantiations of abstractions to sets of entities (real or imagined) that we want to define, we run into problems with other methods of picking out elements, such as archetypes and stereotypes, that get in the way. We also run into issues of intension, with an ‘s’. That’s for later.

I first encountered this notion of prototypicity in a biology class when I was about 13. The teacher asked the class clown to come up to the blackboard and draw an amoeba. He drew a neat block ‘L’ shape, and the class burst out laughing. The teacher got mad and told him to stop clowning around and draw a proper amoeba. He countered that since we’d been taught that amoeba proteas could take on any shape, a regular ‘L’ was as much an ‘any’ shape as an irregular blob.

Concepts and Prototypes

Venkatesh Rao challenges the conventional wisdom surrounding organizational silos. While they are often viewed as barriers to productivity and innovation, Rao argues that silos can actually be beneficial if properly understood and managed. He provides a detailed definition of what constitutes a silo and explores the potential advantages they bring to corporate ecosystems. By challenging the negative perceptions and offering a fresh perspective, this article invites readers to reconsider their views on silos and their role within organizations.

The Silo Reconsidered

Venkatesh Rao explores the complexities of defining concepts. Using the example of concavity, he takes readers on a journey through the challenges and philosophical questions that arise when trying to pin down the true meaning of a concept. With a mix of mathematical insights and philosophical musings, this article will leave you questioning the nature of definitions and the inherent fuzziness of language.

Let us say you are the sort of thoughtful (or idle) person who occasionally wonders about the meaning of everyday concepts. So there you are, at the fair, laughing at yourself in a concave mirror, when suddenly it hits you. You don’t really know what “concave” means. You just recall vague ideas of concave and convex lenses and mirrors from high school and using the term in general conversation to describe certain shapes. So you decide to figure out a definition.

What do you? How do you make up a definition? Let’s get you into some trouble.


So the first thought you have is: concavity has something to do with “indentations” or inward curvature of shapes. You quickly abandon open-ended curves, of the sort you see in graphs of population growth and the like. Being smart, you realize that the concavity there is not fundamental — you could turn a concave graph upside down, and make it convex with respect to your preferred visual orientation of “up is against gravity.”

So you decide that the notion of concavity probably only makes sense for closed curves: things with an inside and an outside (congrats, you just found a use for the Jordan curve theorem!). You draw yourself a prototypical closed curve, like the one on the left below, and stare at it:

 

Concavity and convexity

You think, “hmm… really what is going on with concavity is that I can sort of take a short cut across some parts by going outside it.” That leads to your first stab at a definition: a figure is concave if there exists a pair of points in it such that the straight line between them is not contained entirely within it. You draw a couple of lines and convince yourself, like on the right. At this point, if you like to rush to math, you might even write down an equation like this one:

[tex] \bar{x}(\theta)= \theta\bar{x}_1+(1-\theta)\bar{x}_2 [/tex]

and go, “Aha! a closed curve is concave if and only if you can find a pair of points like so, and for some theta, the point on the line given by my clever equation isn’t inside the figure!”

You’ve just found an attribute of convexity that you think is necessary and sufficient to define it. But then, suddenly a thought occurs to you. You sketch:

Disconnected set

and go, “Uh Oh!”

What just happened here? Why does this bother you? You’ve found a way to draw a line across two closed figures that don’t “look” concave to you, and satisfied a formal notion of concavity. You really want to say that concavity is a notion that only applies to single figures. So what do you mean by “single”? Is a Figure 8 single? Is it concave?

You are in trouble. At this point, if you really cared enough, you’d go on to reinvent a good deal of topology, invent the notion of “simply connected,” figure out that you need the notion of closed and open sets and interiors and boundaries (to handle the Figure 8 case) and so forth. But let’s not go down that road. Let’s ask the more interesting question, why didn’t you just define concavity to be anything that satisfies your original “straight line” test? (For many purposes in math, that is in fact exactly what you do, use the definition without worrying about connectedness — that’s the impatient, technical, “let’s get on with it” aspect of mathematics, but you and I like to fuss over what we mean instead of getting somewhere).

The interesting thing about the way our minds work is that math and formalism is subservient to a fuzzier notion of “what I want to get at.” As we refine technical definitions (or natural language definitions of entities like “culture”) we tend to move the definition to get at an understood but inexpressible concept. We practically never reduce the concept itself to the definition we are working with. This sort of thing is an example of the operation of what philosophers like to call intension (with an ‘s’). Intension, roughly speaking is the “true” meaning of a concept we are after. The difference between definition and meaning is what philosophers like to characterize as primary (or a priori) and secondary (or a posteriori) intension. The primary intension of “water” is “watery stuff.” That is why a sentence like “Ammonia is the water of Titan” makes sense to us — we imagine ammonia oceans. By contrast, “Water is H20” is a secondary intension. David Chalmers has a beautiful discussion of intension in The Conscious Mind: In Search of a Fundamental Theory.

Does this apply to this example? Concavity, unlike “water” is an abstraction of real-world things like inkblots, bays, dents, holes and so forth. It references too many things in the real world for us to useful say something like “concavity is sort of like bays or dents.” Despite this, however, our brains seem to work with a primary intension of concavity that draws efforts at definition spiraling towards itself. We grope towards what we mean through attribute-based tests expressed in terms of simpler concepts (like “straight line” in our case).

What makes math special is that starting with a few prototypes that suggest a useful notion, we can often converge in a finite number of steps to a watertight characterization of an abstract concept within a useful closed domain. Brouwer, was perhaps the only major mathematician who tried to articulate this fundamental aspect of the structure of mathematical thought — that technicalities follow from trying to capture intuition.

Leaky abstractions like “culture” and “war” though, are another matter. I don’t yet have a good handle on how to think about the process of achieving clarity with such concepts. Until then, all I can offer is my own rule of thumb, “Seek to capture the intension!”

How to Define Concepts

Venture into the fascinating world of Indian innovation and business as Venkatesh Rao delves into the unique DNA of the iconic Tata group. From the humble origins to the game-changing acquisitions and groundbreaking $2000 car idea, Rao explores what sets Tata apart from the rest and its impact on the global economy. Unravel layers of feel-good storytelling and discover the signature aesthetic and high concept of Indian businesses through the lens of Tata's history and evolution. Prepare to be captivated by the Parsi factor, the role of history, and the true essence of innovation in this thought-provoking essay.

And just what might those be? I can’t quite answer the question, but I can provide you with some raw material so you can come up with your hypotheses.As a kid in pre-economic-liberalization India, I grew up with the phrase “import substitution,” and surrounded by a low-credibility innovation culture best captured by the following joke (which we retold in various forms): The US, USSR, Germany, Japan and India decided to collaboratively build a new rocket. The US supplied the design, the USSR the engines, Germany the manufacturing and Japan the electronics. Punchline: what did India contribute? We added a “Made in India” label. Today, fortunately, that joke wouldn’t work. There is a small but growing culture of true innovation taking root. A question that I have been pondering lately is “what is the DNA of the emerging Indian innovation culture?” Whatever the answer, it definitely includes the genes contributed by the


I have a great deal of affection for the group, its flagship company Tata Steel (where my father worked for nearly 40 years), and of course the group’s hometown Jamshedpur, where I grew up. So the question about the unique features of the Tata genome interest me at a personal level. Recent events though, have made Tata interesting in more objective ways.

Starting with the acquisition of Tetley by Tata Tea, then onto the acquisition of Daewoo’s ailing truck business by Tata Motors, and most recently, the minnow-swallows-a-whale shock of the Tata-Steel-Corus deal$2000 car idea, which if successful, should be a business model for the books for years to come. This is not the somnolent Tata I grew up with. This is an innovative (if not inventive) Tata. A Tata that has some business model tricks up its sleeve., the group is suddenly on the global economic radar. For the first time, we can ask about the group’s (and by implication, India’s) unique business identity in the global economy. The Tata group is now on the order of $21 billion in size, last I checked, which is probably a serious underestimate of its actual economic significance. Tata Corus is now the 5th largest steel-maker in the world, and set a record as thelowest-cost producer of steel a couple of years ago (Tata Steel was the epitome of unproductive bloat a decade ago). Tata Motors is developing the seriously innovative

Certainly, if there is a signature aesthetic and high concept to Indian businesses, we should start looking for it in Tata.

But to get to the reality of Tata, you have to penetrate layers of opaque feel-good story-telling. Like any other large conglomerate, the story of its growth is shrouded in layers of hagiography and tales of royal screw-ups retold as glorious tales of redemption. There is your standard origin myth centered around Jamshedji Tata, this guy —

J. N. Tata

and the usual host of stories told by the usual admiring courtier-biographers, such as R. M. Lala (author of such works as The Creation of Wealth and several books on J. R. D. Tata, the third-generation Tata chairman of the group and focus of the largest personality cult among all the Tatas). The basic story is told well enough in the Tata Group wikipedia page, so I can get straight to the DNA question. Here are some starter thoughts:

The Parsi Factor: At parties growing up, I’d hear jokes about “Parsiality.” That was a reference to the notable nepotism among the Parsis who, until recently dominated upper management in the Tata group. Other business-dominating ethnic groups of India, such as the Marwaris and Gujaratis, have usually been subject to worse nepotism charges though. Might there be something unique to the Parsi approach to business that explains Tata’s successes (and failures)? Is there a Parsi ethic at work rather like the Protestant Ethic? Certainly in India, the Tata brand evokes a great deal more trust than, say Reliance.
History as a Nanny Company: I remember most vividly the enormous, benign and maternalistic presence of the Company in every aspect of daily life in Jamshedpur. The company would provide buses for the school picnics. We hung out in the company-sponsored British-style clubs (us plebs went to the United Club, which I just found on WikimapiaIspat bhi hum banate hain” (“We also make steel”). A good deal of all this though, has changed. I am told Tata has transitioned from a paternalistic, indulgent, family-managed culture into a lean, professionally-managed, McKinsey-advised global corporation., wow!) . We learned Tata history in school, went to the the Tata Main Hospital when we got sick, ran schoolyard rivalries along company lines and vacationed in company guest houses. In fact the whole city of Jamshedpur was (and I suppose still is) organized around the various Tata companies that dotted the landscape. Even the town councils were run by the companies. A TV ad campaign by Tata Steel in the late 80sended with the tagline “
History of Innovation-Infrastructure Philanthropy: The Tatas founded the premier research institutes of India (including the Indian Institute of Science and the Tata Institute of Fundamental ResearchHouse of Birla, but I have no insight into that group.) and gave slackers like me college scholarships. So, simply on the basis of raw infrastructure alone, research and technology in India wouldn’t even exist without the influence of the Tatas (J. R. D was close to Jawaharlal Nehru, so I am sure he had a hand in influencing the latter’s nationwide policies as well). Quite impressive, isn’t it? The Carnegies come to mind for a family with a comparable influence in shaping the innovationand education culture of a large nation. The next-best industrial house in India in terms of contribution to India’s innovation culture is probably the
So does all that add up to a unique innovation DNA? Is there a single word or phrase that characterizes it, like kaizen?

The House of Tata and Indian Innovation

Venkatesh Rao delves into the deep scientific and sociological troubles surrounding superstring theory in his thought-provoking book review. Rao reviews Peter Woit's "Not Even Wrong" and Lee Smolin's "The Trouble with Physics," uncovering the flaws and limitations of the widely accepted theory. Rao's review challenges popular beliefs about superstring theory, shedding light on its lack of experimental evidence and questioning its status as a true scientific theory. Dive into this intriguing review that offers a fresh perspective on the complexities of physics and its societal implications.

Two recent popular science books provide a startling peek into the deep scientific and sociological troubles in the world of superstring theory. Not Even Wrong by Peter Woit and The Trouble with Physics by Lee Smolin together triangulate the core of the trouble. If you, like me, have been distracted from the foundational problems of physics by the ongoing two-decade fascination with chaos and complexity in the popular literature, now is the time to get back to observing the “deep” stuff. It is starting to get seriously interesting again.


 

I suspect if you’ve been following physics through casual browsing and media reports like I have, you have probably formed an opinion along these lines: superstring theory is the only game in town, and is basically in good shape except that it cannot be experimentally verified because the energies involved are “too high.” That is, you probably thought it is a sound scientific theory in principle, just not verifiable in practice. What I learned from these two books is that first, it is not sound even in principle, and second, it is not even a theory at all — just a collection of mathematical clues to the existence of a potential theory.

Taken together, the books make for perhaps 3-4 weekends worth of serious reading effort. Since the authors’ metaphoric/natural language explanations of various technicalities are actually more confusing than the math itself would be, you might want to read the books with wikipedia handy, to look up technical bits and pieces that you think you might understand better if you saw the actual equations. But the reading is absolutely worth it. I have rarely been this seriously challenged when reading popular science, and I came out of the experience feeling exhausted, but with a sense that for once, I’d earned that warm glow of wisdom, instead of having been deluded into a false sense of comprehension by a clever writer.

The Two Viewpoints

Peter Woit, now an adjunct at Columbia and author of the Not Even Wrong blog is a physicist by training who shifted to mathematics early in his career. This is the less polished of the two books, but paradoxically, more useful and interesting as a result. Had Woit carefully judged his potential audience and picked the appropriate mix of exposition, metaphor and mathematics, the book would have been easier to read, but not been as revealing as it is. Unlike most popular science authors, Woit doesn’t attempt to make the reader feel like a genius. If you are not a physicist, but have training in some other mathematically-oriented discipline, you’ll probably get about a third of the way through the book before you start to drown. But the overall sociological and philosophy-of-science argument is straightforward enough to follow. Technically, Woit focuses his critique primarily on the structure of the quantum physics aspect of the field.

Smolin’s book is different. First, it is better structured and you’ll get a lot deeper into it before floundering (if you are a non-physicist and you actually think you understand non-mathematical popular physics books, you are either deluded or a closet prodigy). Smolin himself is a respected physicist (now at Canada’s unique Perimeter institute) who has published extensively in both superstring theory and one of the alternatives you don’t hear about — general-relativity based quantum gravity. A deep love of physics permeates every page of the book, and Smolin tries his best to base his conclusions on a sophisticated (and classical) understanding of what legitimate science is about. By contrast, despite his best efforts, Woit’s writing reads like a thinly-veiled polemic in parts. Each book, by itself, provides a critically incomplete look at the situation, but taken together, they provide a reasonably complete assessment.

The Technical Critique

The technical critique hinges on two major elements and several minor ones. The first is the lack of experimental evidence. Superstring theory has dominated fundamental physics since around 1980. Despite more than 25 years of development, it still has no experimental support or any realistic hope of experimental support in the near future. This is apparently historically unprecedented. By itself, this wouldn’t be so worrisome if it weren’t for the fact that superstring theories don’t look verifiable or falsifiable even in principle. This forms the basis for much of the philosophical critique.

The second element has to do with general relativity, and is particularly emphasized by Smolin’s book. You have probably heard some claim to the effect that superstring theory naturally requires and predicts gravity. Not quite. Smolin argues carefully that the most mathematically and philosophically compelling way to think about gravity is through background independent theories. Theories, like Einstein’s general relativity, where the laws do not depend on a particular structure of space-time (like Euclidean or Riemannian), but which actually tell you how space-time behaves based on the distribution of mass and energy. Superstring theories, by contrast, are background-dependent. You have to assume the structure of space-time to get started. That gravitons (hypothetical particles that carry the gravity field) pop out of superstring theory doesn’t help. You want background independence.

Since superstring theory grew out of quantum field theories and the Standard Model rather than out of general relativity, there are a great many other specific technical critiques, most of which are inherited from the the concepts and techniques of the Standard Model (which itself is at least experimentally verified to unprecedented levels within its explanatory scope). Woit’s book gets into some of these at a fairly detailed level, that I am not competent to summarize. One common thread to all these critiques though, is that the successful conceptual advances and technical solutions that do exist should be interpreted as advances in mathematics, not physics. In fact the influence of superstring theory, particularly that of the field’s leading figure, Ed Witten, has led to a renaissance of sorts in mathematics, and earned Witten a Fields medal.

The Philosophical/Aesthetic Critique

The main philosophical critique of superstring/M theory is that it appears to be fundamentally unfalsifiable due to its intrinsic nature: it has enough mathematical flexibility that its parameters can be set to match any observed reality. In fact there are apparently trillions of possible “theories” that form an entire landscape of possible instance theories which could fit any observed reality. This suggests that what we are seeing is not an actual theory, but instances of the trajectories of a theory we don’t yet know (a rough, but limited analogy: if you observed various pendulums and documented each damped oscillation trajectory as a “theory” you would probably be looking for the familiar simple-harmonic-oscillator equation).

But what are we to make of a (hypothesized to exist) theory that can fit any reality? You could either toss out your methodological dogmas about falsifiability and appeal to ideas like the anthropic principle, or you could stick to your guns and decide that this is not real science. Both Smolin and Woit stick to their guns. Going in, I personally was fairly receptive to the postmodern philosophies of “ironic” science that are being used to bolster superstring theory, but I came away convinced that Smolin and Woit are probably right. It is not that postmodern philosophies, such as those of Paul Feyerabend, are bad. It is that they are being employed in bad faith in the service of flawed concepts.

The aesthetic critique is actually more interesting. Superstring/M theory proponents claim that the fundamental beauty of their mathematics is an argument in its favor. Smolin challenges this framing and cites a number of examples in history where apparent beauty turned out to be flawed and gave way to a more correct (and perhaps more beautiful) idea. Woit, on the other hand, calls the “beauty” bluff and pokes at it. He concludes that the theory is not very beautiful at all, but is extraordinarily full of technical ugliness as well as a big failure by that one metric of beauty that scientists tend to apply, Occam’s razor. Superstring theory has vastly more primitive parameters than the Standard Model and therefore, along the Occam-razor dimension, is much uglier.

The Sociological Critique

Both authors lay out the evidence to show that the entire field of physics has been taken over by a culture of intellectual bad faith that dominates discourse, prevents alternative viewpoints from getting traction (or funding), and somehow manages to perpetuate itself. Some calibration might help. Just how bad is this situation, viewed from the perspective of other fields? To a certain extent, these criticisms are usually being leveled all the time at the dominant paradigms in every field. I have heard versions of these critiques in the three academic fields I am personally most familiar with (my own home discipline of control theory, and the closely related disciplines of artificial intelligence and operations research). Calibrating against these examples and others, I have to conclude that the picture painted by Woit and Smolin does look pathologically toxic and unhealthy.

Among the things Smolin in particular points out, which I have never seen in such egregious form in my field, are the deification of a few priestly figures like Witten, the open suppression and exclusion of non-string approaches, the striking homogeneity of physics departments and the insularity of conferences in the field. Reading the anecdotes illustrating the pathologies at work, I felt for the first time that my own field is relatively healthy and open.

The Bogdanov Affair is a particularly interesting illustration of the sociological problems in superstring theory. The short version of the story is this: a prestigious journal apparently accepted clear mathematical nonsense for publication, despite peer review by competent reviewers. Woit examines this case in detail and concludes that the lack of sound foundational concepts in the field has created a world full of technicians who can’t even understand each other’s dense calculations as garbage.

So where might all this have come from. Both Woit and Smolin are careful not to blame individuals, and this seems like a sincere attempt to be fair. The one suggestion that comes through, regarding a possible cause, is the very structure of American science. After the great conceptual advances before World War II, American physics (which, by its dominance, has meant world physics until recently) somehow slid into an era where people asking foundational questions were marginalized, and a “Shut up and calculate!” technician ethic took hold, leading to a vast number of technically brilliant physicists taking over the field, leaving little room for philosophical introspection and alternate conceptual frameworks.

Book Reviews: The Trouble with Physics, Not Even Wrong

Venkatesh Rao delves into the concept of magic through the lens of the Harry Potter series, exploring why we are enchanted by imagined realities that depend on it. In this thought-provoking essay, he challenges the popular notions of magic and its association with technology, moral philosophy, and stage tricks, instead offering a powerful and imaginative definition of what magic truly is. Prepare to question your perception of magic and discover the deeper meaning behind our fascination with the extraordinary.

The upcoming end of the Harry Potter series demands piggyback attention, especially from a new blog like mine. Since I have been talking lately about concepts and definitions using toy examples from geometry, I thought I’d take on a more complex concept: magic. In this first of a series of posts aimed squarely at piggybacking the Potter phenomenon, I’ll attempt a definition of the concept of magic that explains why we delight in imagined realities that depend on it.


To understand why a conceptual analysis of ‘magic’ is interesting, consider books like The Science of Harry Potter and Harry Potter and Philosophy: If Aristotle Ran Hogwarts. These are among the more prominent books that have attempted to ride the Potter wave. There are also postmodern, moral, religious, literary and other takes on the series that I found among the 7427 results of a simple search for “Harry Potter” on Amazon.com.

At a pragmatic level, there is nothing wrong with trying to ride a popular trend to talk about the things you care about. I am doing it myself in this blog, after all. If high school kids are more receptive to Aristotle packaged in Potter-talk, so be it. Most of these dimensions of analysis cause no problems. The problem with reading science, technology or philosophy onto an imagined magical world though, is that these frameworks distract us from the core aspects of magic. These core aspects, which I’ll get to, are vastly more interesting than the peripheral aspects that are highlighted by talk of the science, technology or even (most sorts of) metaphysics of magic.

What Magic is Not

The first of the books above for instance, dives straight into examining whether broomstick flight is technologically possible. Cute (and I am an aerospace engineer after all), but besides the point. The same goes for attempts to read metaphysics or individual and social psychoanalytical lessons into a tale of magic. Possibly legitimate exercises with useful results, but besides the point if you are interested in magic qua magic. So, to clear the clutter here, before introducing my understanding of magic:

Yet-to-be-understood technology is not magic. Arthur C. Clarke is probably to blame here for starting off this dull line of thinking about imaginative reality with his quote, “Any sufficiently advanced technology is indistinguishable from magic.” Possibly true (though trite) as a statement about advanced technology. Irrelevant and distracting as a statement about magic. If you get a kick out of imagining how to technologically achieve the phenomenology of magic, the charm of Harry’s world has been lost to you. And I say this as an engineer who makes a living figuring out how to make technology do things.
Magic isn’t about moral philosophy, the nature of good and evil, or even about metaphysical thought experiments about possible worlds.
And this is hopefully trivial enough to not require stating, but magic is also not about stage magic, the performance art (though stage magic works because of the actual nature of our conceptualization of magic). If you’ve ever felt a serious sense of being let down once you’ve learned how a magic trick works, you know what I am talking about. It is not so much that you mind the magic being explained, or that you mind it being explained away, but that you mind it being explained that way.
Worth a note for the seriously conceptually challenged: magic isn’t about poetic or romantic hyperbole of the “the lovers spent a magical evening together” variety, though again, this sort of figurative speech gets at what we actually feel about magic.
The Nature of Magic

So what is magic?

Magic is an imaginative conception of the lawfulness of a universe where matter has the attributes of consciousness, and can be engaged purely through intention. It is the product of our (primarily emotional and existential rather than intellectual) yearning to connect with the physical world beyond living organisms.

Here is a thought experiment to demonstrate the point of the definition: imagine a real magical broomstick that responds to Accio! Broomstick because it is, at some level, a dimly conscious and intentional entity that likes you. Now think about a broomstick that is really a Magnalev flying machine with a high-gain directional microphone for an ear and programmed to respond to a set Latin vocabulary via speech recognition algorithms. Unless you are an impossibly dull person, the idea of the former should make you yearn while the latter should make you yawn. Note that I said consciousness, not intelligence. We are talking here about matter manifesting consciousness as a fundamental property, in the sense of, say, David Chalmers but that gets us into the AI-consciousness debates and away from magic, so we’ll leave that issue for another time. I’ll defend the “intention” bit in a second.

Rowling gets this: more than once she emphasizes that the visible phenomenology of magic is irrelevant. It shows up in her notions of Horcruxes and Mr. Weasley’s warnings not to trust any artifact that displays intelligence if you don’t know where it keeps its brain. It shows up, perhaps most brilliantly, in her idea that advanced practitioners do not need to vocalize spells, that spell-words have no significance of their own other than permitting the wizard to focus his intentions on a very specific aspect of the lawful behavior of the magical world.

This isn’t an argument against using narrative imaginings of magic as inspiration for technology. It is an argument to get at the distinction between magic and technology-that-behaves-like-magic.

The same goes for metaphysical readings. It is fine to ask if a world where Superman can fly is a metaphysically possible world. That is an entirely different exercise than understanding the role the notion of magic plays in our world.

The Lawfulness of Rowling’s Magic

Once you’ve understood the core nature of magic, you can ask, how should you imagine a magical world that is true to this concept? I have mentioned lawfulness a couple of times, and that is key. Imagined magical realities are always lawful rather than arbitrary, but what sort of lawfulness do they manifest?

Is it the sort of lawfulness that physics studies and physicists argue about? Is the law that broomsticks in Rowling’s world respond to Accio! (or a focused thought) similar to the law of gravity? I won’t argue this in detail here, but the answer is no. In no imagining of magic that I have encountered, does this interpretation hold.

Is it then, the lawfulness of collective agency? The sort of laws that emerge in interactions among conscious agents? An extension to conscious brooms of the lawfulness of the interactions among living things? This is closer to the real thing, and many imagined magical worlds do in fact fit this model. Indian mythology is full of human yogis undertaking severe penances with the intention of summoning up the gods. There is at work here a notion that the gods, though more powerful than humans, must nevertheless yield to some cosmic lawfulness that binds all conscious entities in a web of shifting power. This sort of magical reality is driven by the same type of lawfulness that many non-magical ethical universes are (say the one defined by laws like “there is no free lunch” and “what goes around comes around,” and and imperatives like “do unto others as you would have them do unto you.”). But, I argue, it still falls short of magic proper.

No, I think we get closest to our natural conception of magic if we understand it as a lawfulness that governs the connectedness/disconnectedness of a universal consciousness. When I am able to summon up that broomstick, I become one with the broomstick in some way. Evil, in this sort of magical universe, is a perverse condition of resistance to such connection. That is why Voldemort’s ultimate sin is his sin against his own consciousness, in tearing it into seven pieces, damaging the larger consciousness of the Potter universe in an effort to selfishly preserve his own. That is why the notions of connections between minds and leglimency and occlumency are so central to Rowling’s universe. In fact, the very success of Harry Potter, arguably, is driven by Rowling’s unerring sense of our native conception of magic.

And that sort of lawfulness also explains the role intention plays in my definition. In our regular world, the only thing that yields to pure intention, without any manifest action, is our own thoughts. Think of a pink elephant. You intend it, it is done. Neither physical lawfulness, nor ‘agency’ lawfulness behaves this way (the latter is more like telepathic communication, not direct intention-manifestation).

We need to imagine magic because we want the entire universe to behave this way. To be intentionally one with us.



Harry Potter and the Concept of Magic

In this intriguing essay, Venkatesh Rao delves into the concept of "silos" within organizations and examines their existence, prevalence, and potential implications. Drawing on analogies between macroeconomics and microeconomics, Rao challenges the belief in a perfectly collaborative firm and instead explores the dynamics of dissent and lack of collaboration within organizations. He argues that such dissent might actually be beneficial for the long-term viability of firms. If you're curious about the complex nature of organizations and the concept of silos, this thought-provoking piece will surely captivate your interest.

In reponse to my attempt to reconstruct the definition of a silo in a value-neutral way, Torp brings up an interesting empirical question about the relative proportions of healthy and unhealthy silos in the “wild,” and how you would add some empirical color to the discussion. It is reasonable to wonder whether any healthy silos actually exist, and ask how you might detect their existence and measure their “health.” I am going to argue here for an answer based on an analogy between macroeconomics and microeconomics that I hope you find surprising.


The Setup: A Micro-Macro Recursion Analogy

To respond to this thought, let me set up my micro-macro analogy, since that is the basis for my answer. In classical economics, the firm is supposed to be an organization that manifests an idealized form of collaboration within, while the larger embedding economy, which for this article I’ll take as synonymous with “market,” is supposed to manifest idealized competition among economic actors (which could be firms and/or individuals). The imperfection of real markets is readily recognized, but the underlying conceptual model of markets — what amounts to Darwinian competition — has remained solid. Economists from Hayek to Keynes to Friedman essentially work within the same competing-actors/invisible hand framework due to Adam Smith.

The firm though, is a different story. The ideal firm implicit in treatments in managerial economics (which I am just starting to explore beyond Economics 201 level, so don’t skewer me on this point) evokes a vision of a coordinated, collaboration machine dedicated to a single goal. Unlike the real market though, the real firm is not just a noisy, limited-information, bounded-rationality approximation of the the ideal thing. The inside of a firm (and this is easiest to see in the case of large firms) is so far from the notion of the ideal firm that the model should be discarded at a conceptual level. Why?

Firms do not (and in general, should not) have a single unambiguous goal. At any given time, there are competing pressures from different goals: Wall Street quarterly result pressures, customer pressures, pressures from “value” investors to innovate, and pressures arising from the multiple possible interpretations of these ambiguous top-level drivers (defining “customer” as “current customer” or “current and potential future customer” leads to very different results).
This shows up in the fact there has never been consensus on a single metric for even a single aspect of a company’s performance, let along a single metric for the entire performance. Performance on Wall Street, I am told, has variously been measured using a variety of metrics ranging from earnings per share (EPS) to a beast called EBITDA ( Earnings Before Interest, Taxes, Depreciation and Amortization).
Enough stories abound about companies not collaborating internally that we can dismiss the possibility of a firm coming together in idealized collaborative execution even if it miraculously did achieve consensus on the goal.
Tales of redemption and success that look like they are about collaboration towards the greater good (such as the great anecdotes in that classic on change management, The Wisdom of Teams) need to be taken with a grain of salt. These are tales of new, healthy silos dethroning old and unhealthy silos, not tales of all silos being banished on the path to the Grand Collaborative Firm.
This dynamic of dissent and lack of collaboration I will argue, is a good thing, and we shouldn’t be looking to fix it in a naive way. A highly collaborative organization marching to the tune of a single doctrine is not viable in the long term (though periods of such dictatorial discipline are probably good for short periods). The reason is that the market in which the firm operates is a complex place that sends a rich, noisy mixed signal into the innards of every firm within it. Different parts of the firm will process different data sets with different analytical methods. Marketing will highlight demographic trends, while engineering will be screaming about China, and research about missing the nanotechnology window of opportunity.

So what’s a good model for this tumult of a hundred contending voices? There’s no need to invent a new one: the same model that is used for the market at large is also a good model for the inside of a firm. Of course, we need to set the parameters differently, perhaps move closer to the nonzero sum end of the game theoretic spectrum (note that one of the “flaws” of the model of the market is that it does not fit an ideal zero-sum model). The key is that performance and growth of the company as a whole arise from internal competition among silos. This is hardly a revolutionary conclusion: the boundary between the firm and the market is a porous one, so we should expect the same sorts of models to work in both cases, albeit with different parametric characteristics (if you like the biological metaphors, think of cells emerging from a primordial chemical soup — the little bubbles of soup encased in semi-permeable membranes are still soup, even if they are stewing less vigorously).

So that allows me to propose some answers to Torp’s questions:

Who’s The “Maker”?

In the first part of his comment, Torp wonders (emphasis mine):

… if you took about 100 companies, and asked each one to recognize its top 10 silos, and then had them rate the silos as functional or dysfunctional… what the stats would be like…most functionally siloed organizations invariably have large amounts of 1 and 7 [key attributes of dysfunction in my definition], which get in the way of making them truly collaborative, and “company-first” thinking groups.

Let me get a minor point out of the way: whether you asked the question via a democratic poll or a CEO interview (or something in between) you are probably going to get an overestimate of the number and toxicity of the silos present in the organization. This is because normally functioning healthy silos per my micro-macro analogy should create a somewhat adversarial atmosphere of competition, and you should expect accounting and engineering to paint each other as “dysfunctional” silos just as Microsoft and Apple might call each other names. Bad times will only make this name-calling worse, since the economic jungle selects for survival, not truth-telling. But let us assume that our interviewer is smart enough to frame the questions right and analyze them right to get rid of this “healthy competition” bias.

The problematic part of Torp’s question is the “making them” part. Who is this maker? The CEO’s office (s/he and her/his close advisers) are as much a silo working in the internal economy to influence others as any other actor (the same is true of the market, since the government is just a particularly advantaged set of economic actors). So how do you “make” silos collaborative, “company-first” groups?

You, whether you are an individual employee or the CEO, shouldn’t even try to exercise influence with this particular intention in mind. Company-first slogans are fine for morale boosting and tapping into team spirit occasionally, and for corporate brand management, but in general, pure collaboration is a bad thing and should be resisted (call it “groupthink” and see if you agree with me now). Similarly “company-first” implies a consensual definition of “company-first” and whip-cracking around a single definition is only good for short periods of emergency recovery, but not a good practice in general. Every silo, healthy and unhealthy, will define “company first” differently. Sales will say: “company first = customer first,” engineering will say “company first = parts cost first” and research will say “company first = let’s get into nanotechnology”). In all but the most unhealthy periods, analogous to wartime or a national emergency at the level of nations, this dissent is necessary. This landscape of local definitions of the “greater good” is both inevitable and necessary. There is no such thing as a systems thinker (that’s a whole different post for later).

Aside: though the micro-macro analogy breaks down on several specific points, one bit survives. Just as money is the transactional medium for the market, mutual favors form the currency of the internal economy of a firm. A careful barter of favors through trust networks, built up one conversation at a time, is what keeps the firm humming.

And the same sort of force as in the Adam Smith model of the ideal economy directs this chaos towards growth: the invisible hand of (however imperfect) internal Darwinian competition.

Now this is probably not an appealing message to CEOs who need to believe in their greater ability to influence things in order to punch in to work every day. But a belief in an economic model of the internal behavior of a firm is not the same as fatalism. The same sorts of guided-evolution influence mechanisms work; you just need to frame your intervention in the same ways the federal government frames its approach to influencing the nation. You need to argue for your chosen level of intervention the way Keynesians argue with Friedmannians.

Silos and the art of Empirical Theology


Venkatesh Rao takes us on a mind-bending exploration of the third dimension, challenging our understanding of space and perception. From personal anecdotes to diving into spatial cognition and mathematical concepts, Rao delves into the complexities of engaging with three dimensions and how our two-dimensional minds struggle to grasp it. Get ready to question your perception of reality as you join him on a socio-cultural-mathematical-cognitive journey through the intriguing world of the third dimension. Buckle up, it's going to be a wild ride!

Ever since Einstein got us thinking about the fourth dimension and string theorists got us worried about ten and eleven dimensions, we have not really given serious thought to the mundane old third dimension. Several things, ranging from the emerging three-dimensional Internet over at Second Life, to the delightful modern religion of Parkour and the Nintendo Wii controller, have made me think seriously about the third dimension in recent weeks. It isn’t just badly-developed characters in movies and books that are two dimensional — you and I are as well, in fundamental ways.


The third dimension is not a conceptually simple extension of two dimensions. It is also not as familiar a place as you might think, once you attempt to actually engage all three dimensions at once. So here is a socio-cultural-mathematical-cognitive meditation on the third dimension. I’ll start with two anecdotes, segue into some thoughts on spatial cognition, take a detour into some math (which you can skip without much loss if you like), discuss 3dart forms like dance and Parkour, and conclude with some observations on the 3d Internet which is emerging. Take a deep breath.

Two Anecdotes

Two anecdotes should illustrate what I mean when I say the third dimension is not a familiar place for humans. The first is from when I was an enthusiastic swimmer as a kid. Occasionally I’d go swimming rather late, after dark. One such evening, I was the only person in the deep end of the pool, and I decided to do the underwater equivalent of spinning around on a swivel chair, to make myself dizzyfor fun. So I took a deep breath and turned as many underwater somersaults as I could, trying to change my spin axis in random ways as I did so. Result: I got massively disoriented several feet under water. I couldn’t tell which way was up, and for a few seconds, in a blind panic, I was actually swimming away from the surface after I ran out of breath. For those few seconds, I hada powerful sense of the third dimension — it is a scary place.

The second incident is more recent. While a postdoc at Cornell, I lived near the Finger Lakes trail, and one afternoon, I took off on a hike armed with a topographic map — the sort with contours. I somehow managed to wander off the trail and get hopelessly lost, and stupidly convinced that the trail continued on the other side of a deep gorge, with steep sides. I scrambled down one slope and began walking upstream alongside the narrow creek at the bottom, looking for a place to cross. In afew minutes the stream ended, and I’d walked into some sort of un-nameable topographic mess that looked nothing like the map I was holding. It was noon, and I had no sense of North. The map looked like it was gullies and little canyons all over. Suddenly I realized I’d have to navigate in full three dimensions with a bad two-dimensional aid. Vastly harder than navigating in an essentially two-dimensional place like a city,which involves a trivial bird’s-eye-to-worm’s-eye translation around well-behaved cuboidal things like buildings. During the few minutes of clambering up an impossibly convoluted slope, the full complexity of three dimensions hit me. At the time I was working on vanilla two-dimensional robotic navigation algorithms, and I remember a very powerful, sinking feeling of “my research is rubbish and nothing like the real world!”

So what is full 3d really about?

Our Two-Dimensional Minds

I won’t belabor this point, but our minds are essentially two-dimensional in the way they process visual sensory input. Stereoscopic vision gives us some sense of depth, but very loosely, our firmware, I suspect, is limited to some sort of projective geometry in two dimensions. The details of the visual cortex all seem to be about two-dimensional processing. (our auditory system seems more legitimately three dimensional, since it is associated with the semi-circular canals, but I have no insight to offer there at the moment.)

Almost all the time, visually, our attention is focused on one of two planes: the vertical plane in front of us, with “up” defined by gravity, or the horizontal plane (organized with a left/right/forward/backward relative frame of orientation). Call these plane V and plane H (the third, “sideways” plane doesn’t seem to get much action, since we turn around to make it V if we need to process something oriented that way). With one major exception, all human motion and activity involves information processing in one of these planes. Reading is a plane V activity, most walking is a plane H activity. Walking down a long corridor is plane V.

The major exception is tactile manipulation (a subject studied with neat elegance by CMU Roboticist Matt Mason in his under-appreciated Mechanics of Robotic Manipulation) with our famous opposable thumbs. I am convinced though, that the processing going on during tactile manipulation is fundamentally not spatial processing at all. When you solve Rubik’s cube (if you can, that is; I can’t), I suspect you are invoking a very different brain capacity that runs on finite/countable group theory and algebraic topology rather than the geometry of the three-dimensional continuum. This is why expert Rubik’s cube solvers can solve the puzzle blindfolded. They aren’t thinking spatially in a meaningful sense.

If you don’t get the mathematical speculation, this might help. Twiddling your thumbs or solving Rubik’s cube is a discrete/digital type of activity like counting, not an activity involving a sense of continuous space . There is a subtlety here though — all our two-dimensional visual processing is also discrete in non-trivial ways beyond our neural hardware being digital (for example our visual systems detect parallel lines with a special layer of neurons). But at the level of awareness, we sense our two-dimensional thinking to involve the two-dimensional continuum. So in a sense this discreteness of 2d is merely a sort of sampling.

So how do we process three dimensions when we have to? I think we can only do it in two ways: we can train ourselves to think (with low intuition) about three dimensions, with mathematics as our crutch, or we can learn limited, specialized sensori-motor three dimensional skills.

Thinking about 3d with a mathematical crutch

(You can skip this section with no loss of continuity if the mathematical references are obscure to you)

There are differences between two and three dimensions that are purely mathematical and topological, which makes three dimensions a fundamentally tougher space to think about (for instance,knots and knot theory only meaningfully exist in three dimensions). But I am interested in limitations that arise from the way we think.

maneuver automata, introduced by Emilio Frazzoli, that make three dimensional rotations “countable” in a sense). Not much of what we do in 3d is in “rich” continuous 3d. Dropping a spinning top in turbulant water is the sort of thing I mean when I say “rich 3d.” Or the instantaneous snapshot structure of a stormy ocean surface.We have mathematics to help us even when we are thinking of “continuum geometry” aspects of three-dimensional space, rather than finite/countable group-theoretic or algebraic topological aspects (which, as I said before, seem to be more innate in our circuitry). Quick example: solving the Rubik’s cube involves thinking about turns along only 3 orthonormal axes that are integer multiples of 90 degrees. Most human artifacts also limit themselves to such finite/countable group-theoretic limits (for example, agile aircraft maneuvering can be understood in terms of what are called

Continuous three-dimensional structure is something you can start to wrap your mind around using the differential geometry of curves (the Frenet-Serret equations). I recently played briefly with a problem involving these equations (think roller-coaster design). Perhaps I am stupid, but I managed to get no sense at all for how to work them at an intuitive level, even after writing some rather neat visualization code. I was reduced to groping with the actual symbolic math, which I never like to do for long. That tells me I’ll never be a mathematician, since I don’t trustsymbolic manipulation to take me where my visualization abilities fail.

mostYou don’t need to get to differential geometry. Even your more basic three-dimensional SO(3) rotational matrices can be tricky to understand (I don’t find quaternions helpful at all). Once you probe the subtleties of these creatures in the context of sufficiently complex three-dimensional phenomenology (like say, dog fighting in fighter aircraft) you will soon lose all intuition. By contrast, analogous two-dimensional problems, like planar robotic navigation, retain intuitiveness almost everywhere. I have never found atwo-dimensional result that I understood mathematically but not intuitively. In three dimensions though, of my understanding is purely technical, and I have no sense of how the stuff works. There is a lot to explore here, including spinning tops, nutation and precession, continuous (Lie) groups and the like. These are subjects I am exploring slowly and painfully at a conceptual/philosophical level, and I may blog about them as I understand more. But for now, I’ll leave you with the thought that in three dimensions, mathematics is anecessary crutch for most of us, unlike in two dimensions, where intuition easily keeps pace with mathematics. I’d like to meet someone who has true three dimensional intuition. I don’t.

Dancing in Three Dimensions

A more familiar way to get beyond our two-dimensional limitations is through training of our bodies and sensori-motor mental capabilities. Dance is the most familiar example. I have often thought, observing good dancers, that they seem to actually exploit all three dimensions in a rich way. That is what distinguishes them from us pointy-finger clowns.

But a much more dramatic example has surfaced in recent years. This is the French sport/religion of Parkour and its relative, freerunning. Two recent movies have featured the practice, the James Bond flick Casino Royale and the fourth installment of the Die Hard series, Live Free or Die Hard. The New York Times recently did a feature on this movement. There is some mildly interesting Zen-like philosophy and aesthetics the practitioners bring to their art, but it is the art itself that is simply stunning. If you get a kick out of the action scenes in the Spider-man series, you will be enthralled by live Parkour and freerunning (a Web search should yield plenty of videos if you are curious).

The 3d Internet: Second Life vs. Bumptop

It is hard to explain the premise of the 3d Internet. Let’s clear some conceptual confusion first. The current Web isn’t a two-dimensional Web. It is a symbolic Web whose basic metaphor is the document, not two-dimensional visual space. The real two-dimensional artifact in our computing lives is the two-dimensional desktop metaphor. The natural descendant of the symbolic Web is the Semantic Web. The natural descendant of the two-dimensional desktop is something like the three-dimensional Bumptop.

Second Life. (It is not yet quite clear what Google Earth is, in this framework).The 3d Internet only makes evolutionary and conceptual sense if you first assume the convergence of the Desktop and the Web (and therefore the blending of the document and desktop metaphors). Okay, having cleared that up, and assuming Web 2.0 and Google Docs culminate in the desktop-Webpage convergence, that sets the cognitive stage for a true 3d Internet. At the moment, this has been conceptualized in only one way and realized only in one way:a photo-realistic 3d world based on 3d games, the leading example of which is

Second Life. Here is a picture of my avatar in Second Life:You have to experience it to understand where it is taking us. Simply put, what happens if you take the game out of massively multiplayer online role playing games (MMPORGs)? You get a 3d virtual world where you can build stuff, fly around and really stretch your mind in unusual ways. There are social aspects that are interesting in some ways that I may talk about at some point, but it is the foundationalspatiality and underlying metaphors of the concept that intrigue me. If you haven’t yet checked it out, go log on to

Venkat in Second Life

The most curious experience I had in Second Life was building. To build stuff in Second Life you manipulate objects (in translation and rotation modes). Though the interface is extremely well-crafted, it is still extremely unintuitive and feels conceptually and metaphorically wrong. I suspect that is because it is based on an underlying continuum mathematics as a metaphoric base: you do things by moving/rotating things continuously along or around axes. At the same time you have to manipulate your viewpoint and zoom level to get the right look at what you are trying to do (without your avatar getting in the way — the animation doesn’t actually show the avatar doing tactilemanipulation, more like magical hand waving).

I suspect a truly effective three dimensional interface will somehow draw on algebraic topological or group-theoretic concepts, not 3d geometry and rotations. CAD software, and to a lesser extent, Google Sketchup, do this to some extent, but not enough(the underlying “machine shop operations” metaphor that drives these has limits). Perhaps 3d graphic arts software is where the great idea lives right now (things like Maya or the open source Blender). I haven’t used these.
But at a more fundamental level, I suspect the 3d Internet, when it arrives, will take on a much more metaphoric character than Second Life. Bumptop shows us some of the possibilities, as do the neat interface of Windows Vista and Tom Cruise’s cool hand-waving display in Minority Report.

But the true high concept of the 3d Internet still eludes us.

The Third Dimension is Not Simple


In this captivating essay, Venkatesh Rao delves into the enchanting world of Harry Potter and explores what makes it a genre-transcending phenomenon. He argues that the Potter series breaks a major fantasy convention by collocating the magical realm with the real world. Unlike other escape narratives, the Potter universe doesn't offer a refuge from reality but rather confronts it head-on, incorporating familiar social structures and challenges into the magical world. Discover the intriguing dynamics that make Harry Potter an enduring favorite for both fantasy enthusiasts and those who have never delved into the genre before.

In my first article in this Harry Potter series, I took a serious look at the foundations of the idea of magic in general, and its manifestation in the Potter universe in particular.  In this  second part, I want to ferret out that elusive aspect of the Potter series that makes it a genre-transcending hit. What explains its broad appeal beyond the  fantasy genre?  My answer is that the Potter universe is fantasy, but it is not genuine escapist fantasy. That is why people who have never heard of Robert Jordan still read J. K. Rowling. In fact, to find even a remotely similar premise in a major narrative, we have to go all the way back to Lewis Carroll. Not to Alice in Wonderland, but to his lesser known masterpiece, Sylvie and Bruno.

The Broken Golden Rule
Let’s take as a starting point the argument that to break the genre boundary, Rowling must have also broken a major genre convention. Which one did she break?
An obvious candidate is what we might call the convention of non-collocation with the real world. This convention dictates that a work of fantasy must be set in a time, place or alternate reality different from, and causally unconnected to, contemporary reality. That’s a start, but not good enough. This is a pretty weak and frequently-violated convention within the genre itself. Though major works such as the The Lord of the Rings respect the convention, there are many that don’t. Vampire fiction has the same contemporary situation as the Potter Universe, as does the X-Men series from the closely related science-fiction genre. In children’s fiction, Enid Blyton’s  Enchanted Wood and The Wishing Chair  series are also situated in contemporary reality.
But we’re getting close: the key value of the collocation device is that it allows for a an interplay between the imagined realm and the real one.  In the simplest case (as in the Enid Blyton examples), the collocation merely allows for a very close identification with the protagonist during the process of escape. While in Dune  you have to do the work of forgetting reality if you want to identify with Paul Atreides, a collocated narrative like the Enchanted Wood helps you along: just follow the children, Jo, Bessie and Fanny, into the magical forest, and you escape with them into a better place.
Vampire fiction and the X-Men stories are more complex. Like in escape narratives, there is a motive for escape from reality — usually a social marginalization. But rather than actually escaping to a better world, you take a detour through one that changes you in some way, and return to reality, hoping to re-engage it from a position of power. You then have to beat up an  unnecessary villain worthy of your newfound prowess before you realize you are still the same unpopular geek you were before. Tragic, yes, but fun. The X-Men series is probably the culmination of this hopeless line of development within the genre. Appreciate the full irony of your powers being mutations  and being identified as the cause of your original alienation.  There isn’t even room for temporary delusion that your powers might compensate for that acne that made you unpopular. Your power is the metaphoric acne. All you can do is tragically battle others like yourself and choose between staying marginalized with quiet dignity or trying to destroy the reality which alienated you. Enough said. The Potter Universe doesn’t play by these rules.
Here is the key. In the Potter universe, there is no escape motive — the real world with all its problems follows you right into the magical world via platform Nine and Three Quarters. Literally, not figuratively. Government bureaucracies, tiresome sports and their fans, boring school teachers and institutionalized oppression (think house-elves) come right with you. You even get smelly socks as a Christmas gift from your hated Uncle and Aunt, and a plot contrivance forces you to return to them every summer. No escape there.
That’s the key. The Potter universe involves a collocated, accessible, alternate reality that is neither better nor worse than regular reality. If you want to follow Harry there, you’ll need a better reason than feelings of marginalization and a yearning for redemption.
And that’s the Golden Rule that Rowling broke:
If thou collocateth, thou shalt use an escape motive from an oppressive reality and use the alternate reality to provide the means for a (preferably tragic and hopeless) reaching towards redemption.
The parallels between Harry Potter and classic English boarding school literature are not important, and neither are the parallels to  The Lord of the Rings. The true parallel is to Lewis Carroll’s Sylvie and Bruno, the only other example I can think of where the protagonist switches between (or rather, drifts between) reality and a magical world for non-escapist reasons. An examination of the deeper parallels there is beyond the scope of this piece, but trust me, that’s where you want to look for a literary history.
But this conclusion begs the question, if there is no escapism, is this fantasy at all? And why are we following Harry to Platform Nine and Three Quarters anyway? Why don’t we just watch Erin Brokovich instead if we want to retain a sense of reality?
The True Fantasy in Harry’s World

Harry does not escape to Hogwarts. What the Dursleys do to him is not oppression, it is a sly parody of the oppression premises of other works in the genre. Harry (and you and I) are not going to Hogwarts to escape the Dursleys, bureaucracies or anything else.
That is not to say the Potter Universe isn’t fantasy. It just isn’t an escapist fantasy about overcoming evil forces that oppress us at an individual level due to who we are. But neither is it a whimsical, light-hearted sort of fantasy. Though the story contains humor, it is also not a comic fantasy. There is genuine darkness in Harry Potter (which, as I’ll argue in a future post, has nothing to do with the Dark Lord Voldemort, but has to do with the loss of innocence and trust in parental authority).
The central fantasy in the Potter series is this: that we can overcome the calamitous forces the universe throws at us, even when the forces are not evil. Voldemort is not evil — his apparent evil has been psychoanalyzed away and attributed to fairly mundane social forces (and you thought the Marvolo history in the Half-Blood Prince was unnecessary exposition!). True fantasy aficionados would probably love a detailed Sauron-versus-Voldemort comparison, but to understand the logic of Harry Potter, you only need to appreciate that Voldemort would not work as a villain if he’d remained an ineffable malfeasant force symbolized by a flaming eye on a tower. The story needed Voldemort, the evil person, explained away.
Voldermort the person being rendered peripheral, we can start to understand his real role: as a symbol and reductive manifestation of all societal dysfunction.
Here’s why. There is a lot that is systemically wrong and oppressive with the Potter magical society, but this wrongness and oppression lacks both a clear locus of origin and more importantly, lacks a clear evil intent to oppress on the part of a well-defined oppressor. Unlike Sauron in the Lord of the Rings, Voldemort is a product, rather than the cause, of a flawed world.
That is why Rowling necessarily had to maintain a close analogy between the social structures of the magical and the modern real world — if this systemic oppression is to be believable, it must necessarily be democratic and modern.  The democratic Ministry of Magic is not merely an update to the  quasi-medieval, monarchic social structure of the Lord of the Rings. It is a necessary element in an unpleasant world where there should be no real evil to blame (a medieval monarchy wouldn’t work because the evil Emperor would actually be evil).
And herein lies the true fantasy in Harry Potter: that systemic dysfunction can be destroyed because it has been artificially projected onto, and reduced to, one locus: the person of Voldemort. It is a fantasy because the real-world equivalent would be to pretend that poverty, AIDS, global warming and the rest of the usual lineup of the troubles of earth, could be eliminated by destroying one specific person. Even George Bush does not believe that about Bin Laden.
The Great Leakage
The adoration of regular fantasy fans of all ages does not need explaining. So the interesting question is, “why do non-fantasy fans read Harry Potter?” Why did the book leak out of its borders?
I should probably be classified a non-fan. I was mildly entertained and objectively impressed, but not enthralled, by Tolkien. I couldn’t make it past the turgid first book of Dune. Robert Jordan and the more hard-core writers lost me on page one. I read Terry Pratchett because a friend urged me to, but that didn’t stick either.
Potter though, engaged me instantly and kept me engrossed despite the fact that in terms of craft, discipline and mastery of language, Tolkien and many other genre writers are leagues ahead of Rowling.
What works to draw in outsiders like me (and people like my wife, who’ve not read any fantasy fiction besides Harry Potter, not even Tolkien) is the fact that it isn’t genuinely escapist. Because Rowling ports the problems of our world wholesale into her world, and injects the right sort of immersion-disrupting humor into the growing darkness, we never truly forget the real world as we engage hers (incidentally, based on the similar narrative development of Catch-22, I predict that The Deathly Hallow will be unremittingly dark and humorless, like the end of Catch-22).
Frankly, I’ve had a fairly privileged life, and have never felt the need to “escape” it as comprehensively as Middle Earth allows. But even the most privileged people in this world, from Bill Gates on down, must sometimes feel a sense of hopeless frustration about global problems ranging from AIDS and heart-wrenching poverty to global warming and metastasized terror. Because of that, we occasionally need a good-humored dose of fantasizing, that we can slay the hydra (or horcrux-headed monster) that threatens us all, with a simple Avada Kedavra. And then we return to the real world where all most of us can do is recycle.

Harry Potter and the Leaky Genre

Ribbonfarm

Venkatesh Rao's captivating exploration of meetings and collective decision-making unveils the hidden dynamics and powers at play in these seemingly mundane gatherings. Rao dissects the naivety of believing in the intelligence of large groups and reveals the influence of those who control the obvious. From framing discussions to determining common knowledge, he reveals the surprising power wielded by those who dominate the conversation. Discover the intriguing "rules of power" that Rao has discovered through his extensive experience in various meetings. Get ready to view meetings in a completely new light and understand how to navigate them with finesse.

The 15 Laws of Meeting Power


Sapir-Whorf, Lakoff, Metaphor and Thought

This piece was written in Ithaca, in 2005, and is as accurate a phenomenological report of an actual mental response to real events as I am capable of.  At the time I thought — and still do — that a very careful observation of your own thoughts as you react to sensory input is a very useful thing. Not quite meditation. Call it meditative observation. Stylistically, it is inspired by Camus.

-1-
From my window table on the second floor of the coffee shop, looking down at the Commons – the determinedly medieval, pedestrians-only town square of Ithaca – I saw the parrot arrive. It was large and a slightly dirty white. Its owner carefully set a chair on top of a table and the parrot hopped from his finger onto the back of the chair and perched there comfortably. I suppose the owner wanted to keep it out of the reach of any dogs. He gave it a quick second glance, and stepped inside a restaurant. The parrot ruffled its feathers a bit, looked around, preened a little (showing off some unexpected pink plumage on the back of its neck, hidden in the dirty white), and then settled down
-2-
The Ithaca Commons is a ring of shops and restaurants around an open courtyard, occupying the city block between Green and Seneca streets. The shops are an artfully arranged sequence of mildly unexpected experiences. Tacky used clothing and dollar stores sit next to upscale kitchen stores, craft shops, art galleries and expensive restaurants. The central promise of the Commons is that of the Spectacle. Street musicians, hippies meditatively kicking hackeysacks, the occasional juggler – they all make their appearance in the Commons. A visibly political Tibetan store and experiential restaurants such as the Moosewood and Just a Taste complete the tableau. The Commons is crafted for the American liberal, a cocoon that gently reinforces her self-image as a more evolved, aware, and thoughtful creature than her parochial suburban, beer-guzzling, football-fan cousin.
But in any world, the presence of a large, dirty-white parrot is a definite non sequitur. Wall Street, Hollywood, sell-out Suburbia (and Exurbia), Southern Baptist congregations and the liberal Ithaca Commons, are all equally at a loss to accommodate the parrot. The grab-bag of varied oppressed Others that mill about University towns, I suspect, would also be at a loss to handle the parrot. Those of us who claim to be governed by eclectic, deeply considered and original world views – and I count myself among these – are also forced to admit that for all our treasured iconoclasm, we cannot accommodate the parrot. We are therefore forced, out of sheer necessity, to look at it.
-3-
I am no deep observer of real life. When I work in public areas, it is for the steady supply of low-intensity human contact. The mass of unremarkable humanity does not register, except as a pleasant backdrop. Pretty girls, babies, dogs and notably ugly people do register, leaving a gentle and piquant trail of unexamined visual flavor. I am not a true people watcher.
I didn’t quite know what to do with a parrot though, so I was forced to look at it. It triggered no runaway train of thought, so for a while it was just me and the parrot, separated by a pane of glass, and about fifty yards. The impression of “parrot,” did not fade, get filtered away or get overwhelmed by free association. It lingered long enough that I began to watch. The parrot seemed happy. It sat there, awake, but not alert or wary. It looked straight ahead. Presumably it did not find the scene interesting enough to strain its neck.
-4-
I wonder how Hegel would have reacted to the parrot. Would it have triggered, through some improbable sequence of dominoes, a fresh insight concerning the Self and the Other? Would he have gazed inattentively at the parrot and chased gleefully after some new thought (“bird… freedom …”)? Would it be just another little nudge powering the inexorable progress of his snowballing philosophy of everything? Would it occur to him that whatever lofty abstractions it triggered, the parrot qua parrot would not make an appearance in the edifice he was building? Sadly, I must suspect that the thought would not have occurred to him.
To be fair, I must also suspect that the existentialists would have done no better, despite their protestations to the contrary. I must conclude that Camus would have looked at the parrot and instantly exulted, “There it is, the Absurd manifest!” The parrot would again have been lost, subsumed here by the Absurd. As far as the parrot is concerned, Camus and Hegel differ little.
-5-
The parrot, without its owner, was sitting there, qua parrot, indifferent to its impact on passersby. Most people looked at it. Some did a double take. One man stopped, turned to face it squarely and stared at it for a minute, as if waiting for it to acquire some significance. A decrepit old man in a wheelchair rolled by, glancing at it with a painfully slow motion, before letting his head sink again to his chest, weighed down, I suppose, by illness and unseen burdens. A black mother, pushing a stroller, walked by, glancing at the parrot without interest. I wonder why black registered.
A pretty girl in faded red pants stepped out of a shop, talking on a cell phone. She took in the parrot in the absent-mindedly, absorbed several network hops away. She exited my field of vision, stage right, but returned a few minutes later. This time she stopped and genuinely stared at the parrot before heading back into a shop.
A hippie, dread-locked and tie-dyed, stopped and grinned delightedly at it. There was no discernible transition from “see” to “grin,” and something about that bothered me. There was something scripted about the response; her engagement of the parrot was not authentic.
-6-
You know you have are a slave to the life of the mind if a phrase like “her engagement of the parrot was not authentic” crosses your mind quite naturally, and it takes you more than a minute to laugh.
But consider what it means if your response to the parrot is measured, seemingly scripted, or otherwise deliberate in any way. A mind with ‘parrot’ on it should not look like anything recognizable. A frown might mean you are trying to rapidly assimilate the parrot – but in that case, the process of assimilation, rather than the parrot itself, must be occupying your mind. You cannot, at the same time, think “parrot” and engage in the task of wrapping up the parrot in a bundle of associations and channeling it to the right areas of long-term memory. The hippie’s grin is equally symptomatic of a non-parrot awareness. The hippie is probably self-indulgently enjoying a validated feeling of “one must be one with nature” or something along those lines.
So an authentic engagement of the parrot must have an element of the unscripted in it. It can neither be deliberative, nor reactive. Furious and active thinking will not do. Nor the “Awww!” you might direct at a puppy. A puppy is a punch you can roll with.
-7-
Two moms with three babies wandered onto the scene. It being a nice day, the babies were visible, one squirming in the arms of its mother and the others poking their snouts out of the stroller. The mom carrying the baby stopped immediately upon spotting the parrot and approached it (she was the first to do so). As is the wont of moms, she immediately began trying to direct her infant’s attention to the parrot, shoving its face within a foot of the parrot. Mothers are too engaged in scripting the experiences of their babies to experience anything other than the baby themselves. The parrot obliged with a display of orange (I suspect it was stretching, disturbed from its contemplative reverie). The baby, however, seemed entirely uninterested in the parrot. Perhaps the parrot was unclear to its myopic eyes, or perhaps it was simply no more worthy of note than any of other exciting blobs of visual experience all around. At any rate, the mom stopped trying after a few moments, and the five of them rolled on.
The pretty girl in faded red pants was back. This time, she had two waitress friends along, and took a picture of the parrot with her cell phone. The three girls (the other two were rather dumpy looking, but I suppose it was the aprons) chattered for a bit and then stared at the parrot some more. Two more pretty girls walked past, and though the parrot clearly registered, walked past without a perceptible turning of their heads. Something about that worried me. They were of the indistinguishable dressed-in-season species of young college girl that swarm all over American university towns. These could have been either Ithaca College or Cornell; I can’t tell them apart. Two more of the breed walked by, again with the same non-reaction.
A black-guy-white-girl couple walked by. The girl turned to look at the bird as they walked past, while the guy looked at it very briefly. Shortly after, an absorbed black teenager walked by. She looked at it as she walked past, with no change in her expression. The parrot was clearly on Track Two. Track One continued thinking about whatever it was she was thinking about. I suppose ‘parrot’ might have consciously registered with her a few minutes later, but she did not walk by again. Something about black responses to the parrot was sticking in my mind. The owner came back out of the store, carrying a cup of coffee.
-8-
Now, a parrot is not an arresting sort of bird. It does not have the ostentation of the peacock, the imposing presence of the ostrich or the latent lethality of a falcon or hawk. Even in context, at a zoo, a typical white parrot is not remarkable in the company of its more gaudy relatives. Any of these more dramatic creatures would, I suppose, instantly draw a big gawking crowd, perhaps even calls to the police. Undivided attention, active curiosity and action would certainly be merited (“try to feed him some of your bagel”).
The parrot though, had neither the domesticated presence of a dog, nor the demanding presence of a truly unexpected creature. A dog elicits smiles, pats or studied avoidance, while an ostrich would certainly call for a cascade of conversation into activity, culminating in the arrival of a legitimate authority (though, I suppose, most communities would be hard pressed to generate a legitimate response to an ostrich. Cornell though, is an agricultural university, so I suppose eventually one of the many animal experts would arrive on the scene).
So a dog elicits a conventional ripple of cognitive activity as it progresses through the town square, soon displaced by other preoccupations. An ostrich presumably triggers a flurry deliberation, followed by actual activity. So what does the parrot cause, living as it does in the twilight zone between conventionally expected and actionably unexpected? You cannot have the comfort of either action or practiced thoughts, with a parrot in your field of view. Yet, the parrot is not a threat, so you clearly cannot panic or be overwhelmed. The parrot, I think lives in the realm of pure contemplation. The parrot is rare in adult life. For the child, everything is a parrot.
-9-
The return of the owner annoyed me briefly. With his return, the non sequitur instantly became an instance of the signature of the Commons: a spectacle. The owner was clearly used to handling his parrot. He had it hop on his hand again and swung it up and down. The parrot spread its wings and did various interesting things with its feathers which I do not have the vocabulary to describe. With the owner, the context of a small bubble-zoo had arrived. The owner chatted with the girl in faded red pants, who had come out again. Fewer adults stared. The ensemble was now clearly within the realm of the expected. Most people walked on without a glance, while some, emboldened by the new legitimacy of the situation, stopped and watched with interest. The owner tired of active display and set the parrot back on its perch, and turned his attention to the girl.
For a minute, I was sorry, but then a girl, about six years old, walked by with her mother. It was a classic little girl, in orange pants and ice cream cone. She stopped and stared at the bird very carefully. It was not a curious probing look, or the purposeful look that kids sometimes get when they are looking about for a way to play with a new object. This little girl did not look like she would be going home and looking up parrots on the Discovery channel website. She did not look like she was gathering up courage to pet it or imagining it in the role of a chase-able dog or cat. She was just looking at it. Clearly her powers of abstraction had yet to mature to the point where she could see the bubble circus.
A pair of middle-aged women stopped by the parrot. After an initial look at the parrot, they turned and started chatting with the owner. I expect the conversation began, “Does he talk?” or “Doesn’t he fly away?” Shortly after, I saw them wander off a little to the side, where there was a fountain. One woman took a picture of the other, standing next to the fountain, with a disposable camera. Local resident showing visiting Cousin Amy the town, I guessed. All is legitimate on a vacation, including a parrot.
-10-
I don’t think children are necessarily curious when presented with a new experience. The little girl presented a clearer display of authentic engagement of the parrot than all the adults. It was what I have been describing all along as a stare. But stare doesn’t quite cover it. “Stare” does not have the implicit cognitive content of the hippy’s “grin.” Happy, bemused, smiling, frowning, eager curiosity – these are visible manifestations of minds occupied by the workings of deliberative or reactive responses to the parrot. Parrot flits too quickly the face to be noticed, and is replaced by more normal cognitions.
So, here is a question: what is the expression on the face of a person who has authentically engaged a parrot? I must propose, in all seriousness, the ridiculous answer, “it looks like the face of a person who has seen a parrot.”
-11-
The people talking to the owner had left. He now sat reading a book, while the parrot ate seeds of some sort off the table. Three teenage skateboarders wandered to a spot about a dozen yards away. One of them nudged the others and pointed to the parrot. They looked at it in appreciation. It wasn’t quite clear what they were appreciating, but they clearly approved of the parrot. That made me happy.
Now, a large brood of little black children came by, herded by two young women who might have been nannies, I suppose. The black kids all stopped and stared intently at the parrot. The nannies chatted with the owner, who looked on approvingly at the children while he talked. The conversation looked left-brained from fifty feet away. Some tentative petting ensued. As the nannies led the children away, after allowing them a decent amount of time to engage the parrot, one little boy had to be dragged away; he managed to turn his head full circle, Exorcist style, to look at the bird.
Now, five young black men, perhaps eighteen to twenty, walked by. Theirs was clearly a presence to rival that of the parrot-owner duo as a spectacle. Their carefully layered oversized sports clothes and reversed baseball hats demanded attention. I suppose spectacles, be they man-parrots or a group of swaggering young black men, do not supply attention, but demand it. But you cannot really compete with a parrot. The parrot is entirely unaware that it is competing. The black group almost rolled past, but suddenly one of them stopped and turned around to look at the parrot. He looked like he’d suddenly reconsidered the studied indifference that I suppose was his response to competing spectacles. A visible recalibration of response played across his face, and suddenly, he was authentically engaging the parrot in a demanding, direct way. The other stopped and looked to. The first man then pulled out his cell phone, still staring at the parrot, and took a picture. He then briefly interrogated the owner about the parrot, and the group rolled on.
-13-
I wonder now, why are black responses to the parrot more noteworthy than generic white responses? And while I mull that, why have the responses of one other group – pretty young girls – stuck in my mind (besides the fact that I notice them more)?
Now, for an authentic engagement of the parrot, there must be parrot on your mind. Your face must look like the face of a person who has seen a parrot. This is not an ambiguous face, or a face marked visibly by the presence of other thoughts or a subtext. A parrot-mind may wrestle briefly with cell phone mind or preoccupied-with-race-and-oppression mind, but the outcome is all or nothing. There is no useful way a constantly active subtext of race can inform your engagement of a parrot.
I suppose I was looking for evidence that there is room in the black mind for at least a small period of unambiguous engagement with the parrot. If your preoccupation with race and injustice occupies you so completely that even the parrot cannot dislodge it, then it must be a sad life. In a very real sense, your mind is not free, and therefore neither are you, if there is not even temporary room for the parrot. The parrot can only occupy a free mind. To my list of profundities, I will add the following: a free mind is one which the parrot can occupy easily, and stay in as long as it chooses.
Now, the little black children engaged the parrot as completely as the little white girl. So if the little kids are born free and demonstrably remain free until at least age six, as demonstrated by the parrot, why and when do they choose to give away their freedom to a pre-occupation with the subtext of race, which makes those happy six-year-old faces sad? Or is it that the mist of preoccupation descends on them, whether they want it or not?
-14-
I suppose enough actual watching eventually teaches you to observe better. It suddenly occurred to me that the neck-language of parrot-engagement said a lot.
The clearest response is the snap, or double-take. It signals computation. A slight glance on the other hand, no different from the casual scanning of everyday scenery, with no special attention, must mean filtering. I refuse to believe that everybody has a nontrivial scripted response to parrot, so it must mean that the scripted response simply treats the parrot as noise to be filtered. In the casual glance, there is no parrot on the mind.
Now, a more complex response, one signalled by a snap, is one where there is a perceptible pause or break in stride, followed by a turning away.  That is a response that is looking for an explanation. The sort of response that might be hooked by a lone parrot, but would ignore the contextually appropriate owned parrot. Most of the time, when we look for an explanation, we can only see an explanation. Sometimes, when the mind hiccups on the path to the explanation, we see the parrot.
 Viktor Frankl said, “between stimulus and response there is a space. In that space is our power to choose our response. In our response lies our growth and our freedom.” Self-improvement gurus like to use that  quote  to preach,  but to me, it seems that this space is primarily interesting because the parrot can live there for a bit, so your mind can be parrot for a bit.
You might hesitate and never visit that space. You might react so fast you leave the space before it registers on your awareness. Or you might dwell there awhile.

The Parrot

Get ahead of the crowd and dive into the world of analytics with this eye-opening book review by Venkatesh Rao. "Competing on Analytics" challenges conventional wisdom about data-driven decision-making and showcases examples of companies that have mastered the art of using analytics to gain a competitive advantage. Even if you think you understand analytics, this review will expose you to new insights and reveal your own biases. Discover why this book has become a must-read for professionals in the field and why it's generating buzz in workplaces everywhere. Don't miss out on this opportunity to expand your knowledge and improve your approach to business intelligence.

I read Competing on Analytics because my boss began swearing by it, and my conversations with her were starting to get seriously confusing. So I bought a copy, and was plowing diligently through it at a local Rochester coffee shop, when a friendly woman — your inevitable next-table laptop warrior — noticed the book, came up to me, and struck up what turned out to be a very interesting conversation (which ended with her heading off to the nearest book store, to buy herself a copy). Since I’ve only ever struck up conversations over a book with random strangers twice before in my life, that struck me as an important piece of evidence in favor of the book. So here is my review-slash-summary.

The short version: well worth a read even if you think you know what analytics is about. And that’s coming from a resolutely non-data-driven guy who normally wouldn’t touch such a bean-counter-ish book with a ten-foot pole. Grudgingly, I have to admit I learned a lot, and saw more than I liked of my own flaws revealed, in the pattern of my resistance to the book’s ideas.
The Premise
You should read this book if you don’t have a ready and clear answer to the question: “what are the differences among the concepts of business intelligence, data mining, analytics and six sigma?” That’s actually also a pretty good interview question for the hordes of job-seekers who are undoubtedly going to repackage themselves as analytics professionals following this book (my idea of a good answer is provided later in this review).
 
 Competing on Analytics, by Thomas H. Davenport and Jeanne G. Harris
There are two good reasons to read this book. First, you are going to hear a lot about it wherever you work, and it is likely going to figure in your company’s next effort at introspection and change, so you might as well get ahead of the crowd. Second, there is actually a lot of good stuff in this book, whether or not you are part of the “data-driven” choir (I am not; though I work closely, kicking and screaming, with many people who are).
Here is the authors’ own definition of the term analytics (which should really be ’empiritics’):
“By analytics we mean the extensive use of data, statistical and quantitative analysis, explanatory and predictive models, and fact-based management to drive decisions and actions…analytics  are part of what has come to be called business intelligence: a set of technologies and processes that use data to understand and analyze business performance.” (Page 7)
That definition is mostly justified through the book, except for the “explanatory” bit (which I’ll critique in a bit). The key premise of this book is that you can compete on analytics defined in this way; that it can be the basis of a sustainable competitive advantage and market differentiation.
For those who get ideas better through examples, here are a few examples whose nature as ‘analytics’ competitors is obvious: Amazon, Google, Netflix, and Progressive Insurance. A few not-so-obvious ones: Capital One, Anheuser-Busch (aka “Budweiser”), Harrah’s gaming and John Deere. Each of these companies differentiates itself from its competitors through mastery of the data flows in its operations.
What is (and isn’t) New 
Analytics is about corporations learning to drink from the fire hose of cheap data that the modern IT systems wrapped around their operations can generate. So what is new? People have been championing a variety of data-driven approaches to management since Frederick Taylor, so what difference does, say, a modern point-of-sales (POS) or RFID system make? Here’s what’s new, per the authors:

Unlike data mining, analytics is about operational interpretation and visualization, not collecting or reporting. This distinction can range from significant to irrelevant depending on the case you are talking about.
Analytics is to be viewed as a subset of business intelligence (BI), within which it lives next to its older, stupider sibling, reporting. I am not sure this contextualization helps anybody, since BI is a massively overloaded term.
Unlike ideas like lean six sigma and its predecessor, total quality management, analytics (when mature) is i) truly global in scope rather than globally-local, ii) about an enabling information infrastructure for all processes and functions (managed at an enterprise level) and most importantly, iii) about responding opportunistically in real time to some sort of systemic variability via feedback, rather than about reducing process variability via episodic process measurement and re-engineering.
Unlike most management doctrines, analytics needs an enabling technology, since it relies on continuous data flows to support high-frequency operational decisional making, rather than low-frequency interventionist decision-making. Without some sort of technology that provides a breakthrough cost structure for data generation (such as remote diagnostics, RFID, retail POS data aggregation or Web services), the data flows required for competing on analytics would simply be prohibitively expensive. That explains why Web-based businesses lead in best practices.
Unlike related technology paradigms like “Service Oriented Architecture” and “Software as a Service,” analytics is about a business competence that has its locus in people, not (just) software.
Perhaps most important: unlike previous data-driven management paradigms, analytics can be a competitive differentiator. This is such a critical claim that it deserves some probing, so I give it its own section.

So if you are championing analytics-based competition, and someone asks you how it is different from what failed before, your canned answer: “Continuous data flows, cheap, operational measurement systems, enterprise-level presence, differentiating capability.” Keep examples handy for those untalented in the area of business abstractions (a.k.a the “all buzzwords=bullshit” crowd).
Why Competing?
The most non-trivial claim in the book is that analytics can serve as a differentiator in the market. That means that it can provide a sustainable strategic advantage that won’t just be copied in 6 months and be reduced to yet another cost of doing business, whereby everybody in the stadium stands up, nobody gets a better view, and everybody has aching legs. Take, for example, something like six sigma, which is widely viewed as (and applied as) a cost discipline paradigm. Once everybody in your industry adopts it, it provides no differentiation (the quality differentiation that can appear as a result of TQM or LSS is not sustainable). Is this claim of analytics as a differentiator credible?
I spent a lot of time scratching my head, trying to shoot down this claim, and finally concluded that it was actually credible for one really good reason: there are domains and business models where rich data analysis can create a one-of-a-kind growing mass of digested information that cannot be duplicated. An example is Amazon’s recommendation system. You could easily reverse engineer the algorithms and start to compete, but you’ll never catch up unless you stole the actual data. In fact, analytical competitive advantage is grown, not built overnight. This isn’t true of all domains, despite the author’s claims, and if you aren’t sitting on such a growing pile of information with cumulative value, analytics may not provide differentiation for you.
In fact the only way I could think of an analytical competitor being overtaken is by an equally analytical new competitor with a better idea on how to crunch the data. Besides the obvious example of Google vs. Altavista and other early search dinosaurs, you might mull a more recent one: Stumble Upon versus the other social bookmarking sites like Digg and del.icio.us.  In these examples, the competitor managed to squeeze more intelligence out of the same data as the competition. Examples from non-Web domains are harder to come by (post any you know of please!).
The Missing Pieces: Modeling, Metaphor and Deduction
Competing on Analytics is not a fatally flawed book. It does have one moderately serious flaw that you need to be aware of. The dominant dichotomy in the book is between “intuitive” management and analytical management.  While winning over old “gut feel” die hards in your organization might be of enormous practical importance,to  buttress the case for analytics by beating up intuition is to tilt at windmills (a particularly pointless windmill charge is a critique of Malcolm Gladwell’s Blink early int he book).
Intuition, understood in the usual sense of “gut feel,” is important whether or not you are swimming in high-quality digested data. The real dichotomy that ought to be addressed, and isn’t, is between empiricist and deductive modes of thought. Analytics (despite its name) is an empiricist doctrine. It’s antithesis isn’t navigating from the gut, it is first-principles deductive (rather than inductive) reasoning.
That is not a fatal flaw though, since induction has its place, and the book does a sound job of what it sets out to do, which is to show that analytics can work and describe a reasonable road map. You don’t have to do an either-or between deduction and induction (you should be using both). That said, however, it is definitely an incomplete book if you are looking for an idea that you can hang an entire business model on. So long as you read the book with a view to filling three gaping holes from other sources, you should be able to keep yourself out of trouble.
Modeling
Unless you are the sort of person who, after a ritual incantation of “correlation is not causation,” goes on to equate R-square with “percent of variability explained”, you will find each of the analytics case studies lacking in a clear underlying conceptual model of the business. Amazon recommendations are cool, but does that explain the book industry? Capital One can crunch data to find profitable customers in pools previously dismissed as unprofitable and high risk, but does that explain the nature of credit risk? Bounce rates help you manage Web sites, but do they tell you how to architect Websites? No, No and No.
For all its talk of “models,” the blunt reality is that analytics, as understood in this book, is about smart people asking the right questions, and trying to guide their thinking with the right sorts of glorified and opaque black-boxes fitted around data. Analytics by themselves will not provide you a clear and sound way to tell signals apart from noises. Conceptually powerful models (like, say, classical mechanics) do that. Conceptually weak models (like regressions) don’t.
All you have underlying most analytics approaches cited in the books then, is some first-order digestion of raw data, some serendipitous metrics that are at once easy to compute and useful, and a few pretty pictures. The number-portrait of your company is nowhere near being a complete (qualitative or mathematical) model you can manage with. The key weakness of such approaches is captured by Robert Pirsig’s  rather clever assertion that “data without generalization is just gossip.” All the familiar criticisms about being fooled by randomness apply. Empiricist models are prey to all sorts of human failings (confirmation bias, self-selection bias, survivorship bias). They encourage prioritization on the basis of data-availability rather than importance, and encourage risk aversion. They are susceptible to rare events (both rare worst-case scenarios and rare huge opportunities), and are prone to bad-faith manipulation.
But most importantly, analytics alone won’t help you think more clearly about your business. They will only help you if you first start out with a clear idea about the nature of your business. The right foundational questions won’t pop out of the data.
Can this gap be filled by borrowing other sorts of business management ideas? Since real-time feedback-driven decision making is central to analytics, the only framework that even comes close is system dynamics, and that will not do (despite an initial infatuation with system dynamics, based on the field being a descendant of my own field of control theory, I am not currently a fan). So that is an interesting research challenge for an ambitious academic: what conceptual modeling framework and organizational theory can provide a sound foundation for analytics?
Until somebody comes up with a good answer, you’ll need to make up a good qualitative mental model of your business before you apply analytics.
Metaphor 
A good metaphor is key to insight, understanding and tasteful application for just about any major idea. Without a guiding metaphor, you are operating blind, applying cookie-cutter solutions, and setting yourself up for failure creeping up on you via serious blind spots. You have little real sense of priority or proportion, and nothing approaching “understanding.” The book screams for a couple of evocative metaphors.
One perception (possibly incorrect) that I couldn’t suppress, was that the authors (and apparently most of the practitioners of the field) are so massively left-brained they should be falling down sideways.  That might be why they don’t supply any metaphors (even though pattern-recognition in data is, paradoxically, primarily a right-brained function)
So let me try to help: a business competing on analytics is to one that isn’t as a plant is to an animal. Animals have a central nervous system that pipes massive amounts of data to and from a very sophisticated (and usually central) processor, to drive action on the same time scales as key opportunities and threats. A tree, on the other hand, has no way of reacting to a chain saw.
This is not a particularly original idea, but it is a massively useful one. IBM has been talking about the biological “autonomic computing” metaphor for years, and I am surprised the authors of CoA didn’t make some use of the idea.
Deduction
As I said, the key debate isn’t between intuition and data. It is between deduction and induction. The failings of induction are treated well enough by entire books (like the delightful Fooled by Randomness by Nassim Nicholas Taleb, which I referenced before). But beyond that, there are places deduction can go where induction cannot. A deductive process can figure out the general logic of a given “checkmate in two moves” situation from one example, while an inductive process (like, say, reinforcement learning) never will, even given all possible examples of the situation. This isn’t to say that deduction and induction are incompatible. They are merely necessary complements, and no treatment of one is complete without a compatible treatment of the other. CoA lacks this, so you will need to supply it yourself. Remember, applying deductive analysis to the foundations of a data-driven inductive process is not the same as applying inferences to the data itself (which assumes that the induction model is valid a priori).
Book Summary
The book is a matrix of running examples (primarily Harrah’s gaming, Progressive insurance and Capital One credit cards, with  plenty of more isolated ones) woven through a set of themed chapters divided into a “fundamental concepts” part and a “roadmap” part. Here is a summary:
Part I: The Nature of Analytical Competition
Chapter 1: The Nature of Analytical Competition
This chapter introduces the key examples and sets up the premise of analytics-as-competitive advantage, provides a list of best practice leaders and distinguishes analytics from business intelligence in general via a neat little graphic of “Degree of Intelligence versus Competitive Advantage.” There is also a bit of a historical overview walking you through notions of enterprise resource planning, decision support systems and other buzz phrases to lead you away from the temptations of conflation. There is also some well-intentioned hedging about when analytics is appropriate. Finally, there is also a suspiciously  extended (and not particularly useful) discussion of analytics in sports which suggests the authors might be sports fans.
Chapter 2: What Makes an Analytical Competitor?
Chapter 2 identifies the key attributes of analytical competitors (page 23), three of which are substantive and one of which is trite. Pop quiz, spot the trite one:

Analytics supported a strategic, distinctive capability
The approach to, and management of, analytics was enterprise-wide
Senior management was committed to the use of analytics
The company made a significant strategic bet on analytics.

The chapter is based on a survey of 371 medium to large firms, conducted in 2005. The results are synthesized in a nice graphic of a 5-level pyramid. The five stages are: analytically impaired, localized analytics, analytical aspirations, analytical companies and analytical competitors. The authors estimate that no more than 5% of all companies belong in the final, enlightened stage.
Chapter 3: Analytics and Business Performance
This is largely a mish-mash of anecdotes and unstructured notes, but with some good meaty check-off lists for your use, that get at what turns mere floods of data into a basis for competition.
Chapters 4-5: Competing on Analytics with Internal/External Processes
These chapters form the heart of the first part of the book. A variety of examples, organized by business function (ranging from R&D to sales and marketing) should give you all the illuminating examples you need, no matter where you are thinking of applying analytics.
Part II: Building an Analytical Capability
This part is much weaker than the first part, so it deserves much less attention. In Chapter 6, “A Roadmap to Enhanced Analytical Capabilities,” we get a fairly unsurprising map, which might, nevertheless, save you some time and fill in some of your personal blind spots  if you are championing analytics. Chapter 7, “Managing Analytical People” is probably the most useful and nontrivial chapter, making, in particular, several useful points about levels of training you might need in your workforce to compete on analytics.
Chapter 8, “The Architecture of Business Intelligence” is worth a skim, but is largely forgettable, and you are better of reading a book on BI to really understand the problem this chapter addresses. Finally, Chapter 9 is titled “Future of Analytical Competition,” but is mostly a summary and conclusion with some lip service to extrapolation.
Championing Analytics
I’ll add one final note that you might find useful if you decide to get hold of this book and use it as ammunition in some sort of daring charge: you can easily string together a very complete and educational presentation to champion analytics by using the large number of diagrams, tables and bullet lists throughout the book. In fact, a good 2 hour short course, based on the book, could be very effectively created using the figures alone, with some cherry-picked anecdotes to make the key points associated with each picture. A shorter version, buttressed by some proof-case data from within your company, should probably win a few small battles.

Book Review: Competing on Analytics

Venkatesh Rao mercilessly dissects the widely popular management book "Blue Ocean Strategy" in this scathing review. Rao argues that despite its success and influence on strategy formulation, the book is inherently flawed and potentially dangerous. He critiques its use of the blue ocean metaphor, deeming it incoherent and lacking in real value. Rao's review takes the form of a list of seven deadly sins committed by the book, further highlighting its flaws. If you're curious about why this bestseller might not be all it's cracked up to be, this review is a must-read.

I don’t usually write strongly negative reviews of books, because most of the time, when you encounter a bad book, it is usually obviously bad from page 1 on. Intelligent readers don’t need help figuring that out. Blue Ocean Strategy is a bad book, but it is not your usual bad book. It is a dangerous bad book because it takes some thinking to figure out why it is bad, despite its success, and despite the fact that its key metaphor of blue ocean vs. red ocean has made it into the business lexicon and titles of innovation projects. This means that harried workers and managers, even really smart ones, who digest this on a short plane ride,  might very well be led down very dangerous paths by it. So here is the first ever skewering of a book on ribbonfarm. Call it a ritual sacrifice. If you already read the book and didn’t find any serious flaws, read on for a detox course.  If you haven’t read it, I hope I am able to turn you off. If you still insist on buying it, please use the affiliate link on this page, so at least some good comes from it. A better use of your hard-earned money might be buying me a cappuccino when you are done reading the review. Or buying one of the alternate recommendations at the end.

 
Blue Ocean Strategy, by W. Chan Kim and Renee Mauborgne
Though I am late to market with my review, it is only now that the book (published in 2005) has seeped into companies to the point where it is influencing strategy, so I might still be in time to save some lives.
Here is a quick summary of the main message of the book: “do something really new.” That’s it. The message is that trite, and it takes the authors 216 pages to deliver it. Wait! You might argue. Doesn’t the book introduce a compelling metaphor and frame a valuable new concept? Isn’t the road map reasonable? Isn’t it at least useful for the collection of anecdotes it contains? No, no, no and no. And I’ll tell you why. Better still, I’ll write this review in the form of a 7 deadly sins list.  I’ll tell you why, even if you follow the book’s road map and succeed, that does not validate the book. You succeeded in spite of the book, not because of it. Finally, I’ll point you to some better stuff to read. So here are the sins of the book, in order of increasing seriousness.
Sin #1: The Blue Ocean Squatter Metaphor
The basic metaphor of the book is this: uncontested market space (which of course you can easily get to using the “roadmap” and “tools” prescribed) is like a blue ocean, where other sharks aren’t swimming around competing with you for prey. Regular markets are red oceans, where the water is bloodied by combat with other sharks. The metaphor is incoherent and confusing. Among other things, blue ocean is a naval term to be contrasted with littoral,  and if you are a normal person, the term probably conjures up an image of deep oceans, far away from land, not a bloody/non-bloody contrast. Even accepting their framing, the ocean might be blue because there’s no krill out there, as a colleague pointed out (a criticism that is set up in strawman form and peremptorily dismissed in the book), unless you are an oceanic white tip shark (aside: I’ve been watching the Discovery Channel “Shark Week”). But these are not the main problems with the metaphor.
It’s main problem is that it gets you nowhere, and actually needs more explanation than the abstract phrase uncontested market space or even just new market. A good conceptual metaphor should have a lot of implications that really enrich your thinking, highlight the pattern of key features in the domain, and get you farther than the raw abstractions or literal examples would. Hobbes Leviathan, for instance, is a great conceptual metaphor, because it conveys the subsuming organic nature of a complex society. The blue ocean metaphor says nothing more than leave the competition behind.
So what’s so bad about a bad metaphor? Well, it causes a squatting problem. Like somebody squatting on an Internet domain name. The idea that you should try to create and dominate uncontested market space is an obvious one, and every manager in the business world understands it. And yes, they could all use a really good metaphor to understand it better. Now unfortunately, there is an incumbent bad metaphor to displace. I suspect we’ll be using this metaphor for a while, and then throw out the baby (of focusing on uncontested market space) with the bath water of a metaphor that dampens and distracts rather than energizing thinking. I hope somebody comes up with a better metaphor and kicks this squatter off prime intellectual real estate.
Sin #2: Conceptual Emptiness
It is my practice, when reading a business book, to ask, “what would be left of this book if I took all the anecdotes out?” The answer is an assessment of the conceptual framework and model of the book. For this book, the answer is “nothing.” Worse, many of the anecdotes are rehashed, so you probably saw half of them before anyway (like Southwest).
Early in the book, the authors hint at a potentially powerful line of development, when they assert that the strategic move rather than the great company is to be the unit of analysis. I perked up and said, “Aha! Now we’re getting somewhere.” The idea that there are no permanently great companies has been around for a while (for instance, in Clockspeed as pointed out by a comment on my review of Competing on Analytics). But what do we find?
“We studied more than one hundred fifty strategic moves made from 1880 to 2000 in more than thirty industries… we searched for convergence among the group that created blue oceans…we tried to discover the common factors leading to the creation of blue oceans and the key differences separating those winners from the mere survivors and the losers adrift in the red oceans” (page 11)
Hmm… alright, we’re with you so far, though you’d better watch out for the Lincoln-Kennedy-assassination common factors effect. So what did you find?
“The creators of blue oceans, surprisingly, didn’t use the competition as their benchmark. Instead they followed a different strategic logic that we call value innovation. We call it value innovation because instead of focusing on beating the competition, you focus on making the competition irrelevant by creating a leap in value for buyers and your company.” (page 12)

Oh my God! How did we all fail to realize this? This is almost as brilliant as the harried bureaucrat in an R. K. Laxman cartoon in the Times of India decades ago, who is saying to his ministerial boss from behind a huge pile of paper, “Minister, I’ve completed a thorough study of the grave flood situation in Kerala. It shows that the flood situation in Kerala is grave.” It gets worse:
“Value innovation is a new way of thinking about and executing strategy that results in the creation of a blue ocean and a break from the competition.”
That’s it right there. They claim to have the secret sauce, and even give it a name. Hard work, inspired visions, blood and sweat, luck and uncertainty not needed. Follow the cookbook and you’ll get there. There is more gobbledygook that I won’t bother to quote.
The conceptual framework really is that trite. There appears to be more, but as I’ll explain in a minute, the more substantial stuff actually came from elsewhere (from sources like Clayton Christenson and Malcolm Gladwell). If somebody uses the phrase value innovation at me, I will throw a stapler at him/her.
Sin #3: Fooled by Anecdotal Randomness
If the authors couldn’t invent a good theory, maybe they at least did their multiple regressions right? Nope. Think about the hard problem they’ve set themselves: analyzing dramatic (and successful) strategic moves that turned companies around. You’d think that sudden, discontinuous changes in the trajectories of high inertia entities like companies would merit more respectful treatment. You don’t want to pretend you can predict, retrospectively explain, or produce earthquakes when perhaps the best you can do is identify some sort of coarse pattern, like the power law that Gutenberg and Richter found.
The problem with analyzing a set of 150 short-sharp-shock type events is that there are a huge number of variables, and you are almost certain to find commonality patterns (among successes and failures) and discriminating factors (between successes and failures) if you look hard enough at any large enough subset of variables. They do that all the time in the stock market. Tired though the complaint is, people still don’t seem to get it, even sophisticated Minitab users. Correlation ain’t causality.
Given this, we need a whole lot more explanation than we get. Did they think about whether their variables were the right ones? Did they account for randomness and path dependence effects? Did they come up with an explanation and necessity/sufficiency arguments? Sorry, go elsewhere for those.
On second thoughts, don’t bother. Their conclusions about the pattern they found aren’t interesting enough to need explanation: reconstruct market boundaries; focus on the big picture, not the numbers; reach beyond existing demand; overcome key organizational hurdles; get the strategic sequence right. 
The only one that is not 120% obvious to most managers I’ve known is the last one, and that’s obvious to about 80%.
Anecdotes are among the most powerful weapons of the business writers. They provide easy wins and easily packaged value. That makes it all the more important that they are analyzed thoughtfully and tastefully. At least hindsight ought to be 20/20.
Sin #4: The Vaguely Plausible Roadmap and Proof Case

Yes,  the book does provide a “road map” and a bunch of visualization “tools.” By themselves, the tools are neither brilliant nor particularly bad.  You and I could invent similar diagramming techniques. Chances are, since you and I are honest enough to not create pointless fluff, where possible, we’d avoid inventing new “techniques” unless faced with a clear inadequacy in current tools and terminology.
But no, we are subjected to the torture of chapter after chapter of new “tools” and useless conceptual clutter (since, per Sin #2, there are no real concepts in the book). We have, among other profundities, the “Eliminate-Reduce-Raise-Create” grid and the “Strategy Canvas” and the “Six Paths Framework.” Call me Stuart Smalley and give me my daily affirmations please. Excellent for retrospective analysis and grabbing credit. Not much help if you are actually sitting over coffee, nursing a splitting headache from having thought for hours, with reams of crumpled paper and acres of scrawled white boards, about  a tricky business situation.
And is there any humility and acknowledgment that other techniques might work as well or (gasp!) better, or that hard creative work can’t be bottled up? No, we are informed that: “Chapter 2 introduces the analytical tools and frameworks that are essential for creating and capturing blue oceans” (emphasis mine). Clearly, the players behind the 150 moves from 1880 to 2000 that they studied just got incredibly lucky, since they lacked these “essential” tools.
What about their proof cases, like the minor leap made by  Yellowtail wine (a fairly trivial innovation that the authors glibly classify with major revolutions like the Model T Ford or Southwest Airlines). Here we see confirmation bias and bad experimental design at its worst. It would take too long (and too much work, given their caginess about details) to point out the problems with the way they claim success for their methods. But here’s the honest experiment. Give a bunch of equally creative teams the same clear directive (say, “find us some new billion-dollar nascent growth options where there is no competition”), and send them off on a 2-day retreat, letting them pick their own tools and methods. You’d find, I suspect that the team using the Blue Ocean methods would do no better than the average team using any reasonable method. Quite possibly (see Sin #6), it will do worse. Which would tell you that it is the creativity and chemistry of the team, and the ideas that are already floating around, that matter. Not the diagramming techniques they choose to use. Of course, doing the honest experiment is close to impossible, which makes their theory unfalsifiable. For an honest look that gives credit where credit is due, and honestly admits what cannot be bottled up, take a look at the narratives and analyses in Katzenbach’s The Wisdom of Teams.

The few experiments described in the book scream “staged!” All the more compelling examples are instances of the authors retrofitting their tools and visualizations onto historical examples.
Sin #5: Not Giving Credit Where Due
The normal practice in business books is to cite other authors and books inline, except for academic references. The authors do that liberally — to call out books they think get it wrong (these include “the company is great” premise books like In Search of Excellence and Built to Last). But when it comes to giving credit where credit is due, we get inconspicuous  endnote references.
Clayton Christenson’s idea of disruptive innovation,  which the authors glibly repurpose and repackage (many of the examples in the book are far better explained by The Innovator’s Dilemma than by Blue Oceans), is not explicitly mentioned or discussed at all (it probably deserves a whole chapter in a book like this). I kept searching for a mention, since the ideas were obviously borrowed from Christenson. Finally I gave up and looked in the bibliography, where I did find it. I never found the place in the text where it was cited. That’s buried for you. In fact, if you take away the stuff explained by disruption, and take the authors at their word (page 13) that they are talking about something other than “technology innovation” and “market pioneering,” you are left with no examples that map to “blue oceans.” Much of what remains is in fact minor differentiation that could rapidly be copied and therefore belongs in the Red Ocean.
Malcolm Gladwell, likewise, is an obscure endnote.  At least Gladwell gets indirect credit, since the authors decided to coin the term “tipping point leadership” (thank God for small graces; we aren’t subjected to “Tiger Shark Leadership.”)
If you haven’t read many other business books, you could be forgiven for thinking the authors are geniuses. They invent so many new terms for old ones, and minor tweaks of old definitions, you can easily get awed.
Sin #6: Promoting a False Sense of Security
We’re getting to the last two, most grievous sins.  If you read the book, use its tools, its “six paths” and walk its road map, you might very well succeed, sing its praises and recommend it. In that case, congratulations. Luck and your own natural abilities to work with any non-random framework won out.
The problem is that the book might very well give you a false sense of security, and seriously derail you. You’d scuttle your effort and, thanks to survivorship bias, not live to tell your tale.
The book might actually delude you into thinking you are on track dealing with a phenomenally hard problem. The hardest business problem in existence in fact: reinventing the business model of your company. You might well convince yourself that it takes just a couple of weeks worth of drawing “strategy canvas” diagrams. I hope you are smarter than that.
This is serious. An honest business writer acknowledges the risks of the market, the complexity of business models, the true nature of the hard work required, and the limitations of any set of tools. In fact, this sin might well be considered an ethical/moral sin as opposed to sloppy thinking or intellectual sinning.  As a colleague who shares my opinion of this book pointed out, it is fundamentally disrespectful to the hard work and dedication of people working diligently to change companies, to reduce their labors to trite retrofitted “strategy canvas” diagrams.
Sin #7: Ignoring the Real Problem
Here is the real problem. How do you reinvent a business? How do you come up with that truly great new idea, that next big thing that will return your company to growth and away from brutal competition? How do you create the passion around great ideas, tempered by a pragmatic acceptance of market uncertainties? How do you create the courage it takes to face them?
Blue Oceans has no answers. The one tiny glimpse of the idea that creativity and courage might be required is reduced to a shoddy discussion of the third of their “three characteristics of a good strategy” — focus, divergence and compelling tag-line. The first two are trite. The last is where the meat is: what they think is just effective advertising is actually the high concept of a clear and elegant value proposition. You don’t just come up with any old idea and look for a “compelling tag-line.” Only truly great ideas will actually admit a powerful “high concept” description. I’ll point out a more modest and more effective book that helps you get there, in a bit.
Unfortunately, there are no complete answers. These questions are as deep as the deepest questions about life, and it is simply dishonest to pretend you know the answer. At best you have some techniques that improve part of the process, reduce some of the uncertainty.
Ultimately, the problem Blue Oceans attempts, and spectacularly fails to address, is the problem of creativity itself. They are not to be blamed for failing. They ARE to be blamed for pretending they know the answer.
What to Buy Instead

Try Katzenbach’s  The Wisdom of Teams to see some real stories of looking for uncontested market space.
Try Christenson’s The Innovator’s Dilemma to understand at least one class of such ideas
Try Daniel Pink’s A Whole New Mind: Why Right-Brainers Will Rule the Future to understand the true creative thought and work that needs to go into powerful new ideas. You’ll see the vast amount of depth behind what Blue Oceans dismisses as “compelling tag-line” and the lumps into the  “create” bucket of its  “Eliminate-Reduce-Raise-Create” framework.
And finally, try Paul Feyerabend’s Against Method to understand the intricacies of trying to explain revolutionary change with retrospective models.

Book Review: Blue Ocean Strategy

Note: the ideas in this post have been significantly refined and turned into a book. The treatment here is somewhat obsolete as a result, but the spirit of my revised arguments remain the same.
Suppose a job candidate walks into your office and hands you a resume. It proclaims, “strategic, systems thinker.” You wince, and almost throw her out right there, but since other parts of her resume look promising, you decide to give her a chance and proceed with the interview. Now ask yourself, how would you actually probe if there is any substance behind the candidate’s claim to strategic abilities? Here is a very good answer: ask the candidate to tell a story. Not any old story, but a relevant one, like how she views the history of development of her field. Or how she views her own personal trajectory. If you can’t figure out why this is an excellent question, read on.

(Credit: many of the ideas in this piece came out of discussions with Mike Allers, ex-US Navy, in 2003, now a philosopher. More on that story later in this piece.)
I react badly, very badly, to words like strategy. Even when people apply the term to me and mean it as a compliment, I cringe. This is because self-descriptions like strategic thinker (and dismissive insults like mere tactician) are at once irresistibly attractive, functionally unique (when you need them, very few other words will do) and so ambiguous that when they are used, the result is usually the worst sort of content-free hand-waving. Unfortunately though, you cannot dismiss every instance of the use of such terms as self-important gas from superior hand-steepling armchair generals. Occasionally, you’ll actually be part of critically important conversations where terms like strategy and tactics are used with sophistication, and consequential matters hang in the balance. Occasionally, even idiots using these terms clumsily will be saying things you need to decipher.
But hopefully, through this piece, I’ll be able to arm you with a cleaner instinct for this language. Note that my treatment is both normative and descriptive. Normative because I’ll try to convince you that you should use such language my way (and because I’ve never seen a comparable set of definitions anywhere). Descriptive because the most sophisticated users of such language I have met do in fact use the words in these senses.
The Triangle and Waves Model 
Chances are, your mental model of complex decision-making, based on just the terms strategy and tactics, looks something like this picture, which is simple, elegant and wrong. I call this the ‘triangle and waves’ model because it conceptualizes the structure of decision-making with a triangle (implying hierarchy) and a set of time scales with shorter and longer cadences.

I won’t tackle all parts of the diagram right away, but here is a sketch of why it is the wrong mental model to have in your head.

Relativism: Draw a bigger triangle subsuming this one, and/or longer/shorter time horizons.  Everything changes, making your definition relative. You gossip that your annoying co-worker is “tactical” and doesn’t see beyond his product to the whole product line. One floor above, some managers are shaking their heads and labeling you as tactical because you don’t consider succession planning issues or morale, and are focused on just the product-line.
Reversal: The captain of a military unit is sent on the tactical mission of securing a bridge. “Private!” he barks at one of his men, “Place those charges at strategic points on the bridge.” (I made this up, but my military friends have assured me its not too unrealistic). Isn’t strategy supposed to be on “top” of tactics?
Arbitrariness: Why three levels? Why not 2, 7 or 19? If it is just a matter of zoom levels, why only 2?
Over-specificity: Do strategic and tactical mindsets only exist in hierarchical decision-making contexts? Can strategists and tacticians exist among a set of nominal peers? Can an individual cook be strategic as opposed to tactical? Can a writer or comic-book artist? Can people trade roles, with tacticians temporarily doing strategy and strategists doing tactics?

So why do so many people seem to have this sort of picture in mind when they talk about complex decision-making? One reason is that it can function as being practically true under specific local conditions (like a stopped clock being right twice a day).  We can get away with the vague notion that strategy is about the ‘big picture’ while tactics is about the ‘details’ because, most of the time, believing this doesn’t hurt us.
But the problem is, it doesn’t help us either. You cannot compute with this mental model, making it effectively useless. You can rationalize, but not reason, with this model. Here is an example. Suppose you are the chairman of the board of a company on the brink of bankruptcy. You’ve just fired your incompetent CEO and are scrambling to find a turnaround artist. What do you need? A master strategist or a brilliant tactician? Turns out, most of the time, you will need the latter, and that’s often the type of person hired (in fact there are firms and executives who specialize in turnarounds).  The triangle-and-waves model can be used to rationalize this. You could say turnarounds are “short-term” quick changes and therefore require “tacticians.” But what is the reference scale? Is 6 months short-term or 3 years? You can tell this is a retrofitted explanation because the alternate (“To turn a company around, you need long-term focus on the rebuilding the fundamentals”) sounds equally plausible.
I haven’t completely demolished the triangle-and-waves model (that would take a longer essay) but I hope I’ve poked enough holes in it that you are ready for something better. I’ll move on, but after providing you with a final question to undermine your confidence in the triangle-and-waves model. Which do you think is the more abstract construct? Strategy or tactic? If you answered “strategy” you’ve been misled by the model. Along the dimensions that actually matter, tactics are the abstract constructs. Strategies are the more concrete ones. I’ll tell you why in a bit.
Four Definitions
To get at why the triangle-and-waves models leads you astray, let me introduce to you two terms that are part of military discourses but unlike strategy and tactics, haven’t made as successful a leap to civilian language. These are operations (sometimes termed logistics) and doctrine. A popular cliche in the military goes: an amateur focuses on strategy and tactics, a professional focuses on operations (an observation validated by the fact that most video games have no operational elements).  Operations is what the naive dismiss as bureaucratic middle management. Doctrine usually doesn’t even appear in the lexicon of an inexperienced decision-maker. Here are my four definitions:

Doctrine: Doctrine is the set of assertions we accept as true in an action domain.
Strategy: A strategy is a set of action and sequencing commitments, consistent with doctrine, and driven by the unique features of an action domain that constrain, but do not define, plans and schedules.
Tactic: A tactic is an abstract action that can be applied in any of a large class of situations that conform to set criteria.
Operations:  Operations is the discipline of realizing strategy in the context of a background of infrastructure systems, resources and processes using a vocabulary of tactics.

I don’t (yet) have a clean visualization or metaphor, but here is how the concepts relate:
Strategies are imagined stories about possible worlds, whose constraints are determined by elements of doctrine, and whose vocabulary is determined by available tactics. Converting those stories into reality through appropriate mixes of deliberative, reactive and opportunistic planning, scheduling, resource allocation and risk management, in the fog of action, is the discipline of operations.
Quick example to illustrate the relationships. In World War II, the Germans invented the tactic of moving infantry on vehicles at the achievable speed of mobile artillery rather than keeping mobile artillery at the speed of soldiers marching, as was done in World War I. This lead to the doctrine of blitzkreig, which in turn constrained the specific strategies in the invasions of Poland and France.
Notice how, in this example, there is a complex circular relationship where the apparently low-level element (tactics) drives the apparent highest level (doctrine) which in turn rewires strategy. Hardly the simplistic case of picking a strategy and then fleshing it out with tactics.
Examples
Here are some examples demonstrating the use of these definitions in classification and short fragments of analysis.

 The idea in the US military that one must control the tempo of a military engagement is an element of doctrine (hence “shock and awe”). So is the distinction between air superiority and air supremacy. In the corporate world, mission statements articulate doctrine. Pop quiz: what do vision statements articulate (hint: NOT strategy).
Strategic nuclear warheads are strategic because they can take out enemy cities and other assets that play unique roles in enemy social systems. New York, Moscow, New Delhi and Baghdad are not just large cities. They also play unique roles in their respective cultures.
Blowing up a bridge or securing a building is about tactics. Running a meeting effectively is tactics. Choosing not to describe yourself as a “strategic thinker” on your resume or in an interview is a tactic (based on the nugget of learned Groucho Marxian wisdom that people who call themselves “strategic” are perceived as NOT strategic).
A unique context can turn a tactical element into a strategic one. Blowing up any bridge is a matter of tactics. Blowing up the only bridge leading into your enemy’s capital becomes a matter of strategy. A tactical nuclear warhead can be used to strategic effect if you use it to blow up your enemy commander-in-chief’s bunker. In the business world, a crucial meeting about a merger can turn an everyday tactical skill into a unique, once-in-the-history-of-the-company strategic element.
The notion of “Information Superiority” in today’s US military  is a matter of doctrine. The related concept of “Network Centric Warfare” is a matter of operations. Business analytics is a tactical element, the idea of Open Innovation is doctrine, not strategy. Lean Six Sigma is about operations. IBM moving to services is strategy.
Placing a charge to blow up a bridge (a tactical objective in general) can be a strategic matter because you must identify the unique parts of the bridge structure that, if blown up, will cause the bridge to collapse most efficiently.
Tactics are abstract: they can be described in manuals with toy situations (for running a meeting or blowing up a bridge). Strategies are concrete: they are best learned through case studies of specific real histories about real people and places, with names (ever wonder why military officers and MBAs train on case studies rather than game theory?)

Practice Questions

Can you analyze the “turnaround artist” situation described earlier and propose an argument why this person must be a tactician rather than a strategist?
Apply these definitions to a different domain. How would you tell a tactical chef apart from a strategic one? What elements of a chef’s approach to cooking constitute doctrine? Is Iron Chef a game of strategy or tactics? What about Dinner Impossible?

Try art. What parts of making a portrait map to doctrine, strategy and tactics?

The Foundations
So why are these definitions justified? I’ll provide a hint, but defer a full development of the underlying theory (I promise you, I have one!) for later. Here is the hint: think about the complexity of a decision problem in terms of the number of variables involved. Now think about the processing limits of a decision maker (7 plus or minus 2 items in short term memory, and similar constraints). Now think about how many repetitions it takes to master a skill (about 2000 for a motor skill), and the time it takes to get one repetition under your belt in a given domain. How many generals do you think will have a chance to become good at leading alliances in world wars through 2000 instances? If they don’t have time to learn the “leading world war alliances” skill through repetition and practice, how do the Eisenhowers and Genghis Khans of the world come about?
To make it even simpler, consider toy examples. Why is Go, a game with a 19×19 board, and indistinguishable pieces considered “strategic” while Chess (6 unique pieces, 8×8 board) considered tactical? Why is cricket considered a strategic game with respect to baseball? Why is American football considered more strategic than soccer?
I’ll address all these interesting foundational questions in a future piece (bookmark and check back), but for now, let me leave you with some simple rules of thumb to operate with, and a little history of my own interest in these matters.
The Cheat Sheet
I’ll explain some of this in a later article (bookmark this and revisit, I’ll add links), and leave some as homework exercises. (How’s that for arrogance?) Keep in mind that individuals may display all four of these skills in varying amounts.
How to tell a real strategic thinker apart from a pretender

Strategic abilities are developed by studying case studies, which pick out unique features of histories of actions, highlight the important and ignore the irrelevant. If you tell a story incoherently, you are not a strategic thinker. If you tell a story dutifully and boringly well, respecting the canonical telling (or genre conventions), you might be a competent strategist, but are not a talented one. If you are able to tell a story in an exciting way that picks out the right elements to highlight, you might be a talented strategist. These are necessary conditions; by no means sufficient. This is one reason why ‘a sense of history’ is a symptom of strategic ability.
How to spot a doctrine artist

A doctrine artist — a priest — is a decision-maker who thinks about the consequences of asserting or denying various existing beliefs about actions, and adding new beliefs to the set of axioms. To spot talent at doctrine-setting, ask questions like “What are the fundamental assumptions of our industry today? Which of these do you think is changing?”
How to spot a tactician
Look for practiced, non-disciplinary skills. Most people have mastered at least one or two professional skills requiring specialized training, so looking for that doesn’t help. But natural tacticians tend to have developed skills that aren’t taught, but nevertheless require practice. These include conversation skills and meeting skills. Look in particular for things someone could have said or done, but chose not to.
How to spot an operationalist
Operationalists are usually the easiest to spot. They usually display a high degree of comfort with the design and maintenance of systems and processes and “getting things done.”
Despite what many people think of me, strategy is not my strongest suit. Doctrine is. I tell interesting but not striking stories. Like Vonnegut’s Kilgore Trout, I am better at making up clever premises for stories than actual stories. My tactical abilities are patchy, with pronounced weaknesses and strengths, as are my operational abilities. I’ll deploy a tactic right now and not tell you what my strengths and weaknesses are.
Where this is coming from
Someday, I’ll do the full version, but here is the short version. Back around 1994, I sat next to an Indian Air Force pilot (he flew Mig 29s) during a train ride, and had a long conversation about the differences in warfare driven by the differences between Soviet and American military hardware. Though I didn’t realize it at the time, that was the first conversation I ever had about doctrine.  I was an undergraduate at the time, and still not over my infatuation with encyclopedias of military aircraft.
Skip ahead to about 2000. I was in graduate school at the University of Michigan, and playing a small part in a US Air Force research proposal our department was submitting. We talked a lot about some very complex problems with very loose language. We used, and drew diagrams of, nebulous ideas like “strategy is about what you want to do, tactics are about how you do it.” We proposed fluffy “three layer architecture” diagrams based on such foggy arguments. No wonder we lost the contract. Substituting one pair of ambiguous terms (“what” and “how”) for another (“strategy” and “tactics”) just makes things worse.
I survived and found other ways to fund my PhD, but was left with an uncomfortable sensitivity to clumsy language in this domain and wrote up some notes. In 2003, I met Mike Allers, who had just quit the US Navy to pursue a PhD in the philosophy of language. After a series of excited brainstorming sessions, we actually co-authored a proposal to study the semantics of the terms “strategy” and “tactics.” We didn’t get funded, but the ideas stayed with me and fermented. Between 2004 and 2006, I spent my time doing more Air Force research as a postdoc at Cornell, this time on things like command and control systems and notions of “situation awareness.” In the process, I plowed through reams of military literature (mostly around the concepts of network centric warfare) and had some interesting conversations with Fred Zeitz, a retired USAF Lieutenant Colonel. The ideas finally started coming together.
Somewhere along this long journey, two simple words, “strategy” and “tactics” become  a 3-ring binder full of half-digested notes and thoughts on this sort of thing. You’ve just read the first of my attempts to finally process that 15 year line of thinking (and hopefully be done with it).
It’ll take me some time to work out all these ideas in beta, so apologies for any incoherence. Any ideas you can add to flesh out my snowballing theories (Clausewitz, Sun Tzu, Machiavelli and Kautilya beware) appreciated.

Strategy, Tactics, Operations and Doctrine: A decision-language tutorial

A few months ago, I read a thoroughly depressing book by V. Raghunathan, Games Indians Play: Why We Are the Way We Are. That book is a game-theoretic exploration of Indian weaknesses. Being a strengths-oriented guy, I am offering up  a much more energizing look at real Indian games and what they reveal about us. I’ll talk about three games — Kabbadi, Kho-Kho and Lagori — and tell you how these games, viewed as business metaphors, help explain some widely-recognized Indian strengths, particularly in the area of management thinking. I hope it provides some introspective fun for my Indian readers, and some insight into the Indian psyche for my non-Indian readers worrying about outsourcing decisions.
  

warning: this article is chock-full of what might seem like blatant stereotyping, cultural essentialism and even gender-bias. See my end-note if you insist on reading those intents into the piece. I am making the best-faith assumption that you are capable of reading this with a sophisticated eye! 
Get Outta My Way Raghunathan!
Let’s first get Raghunathan’s approach out of the way. The book applies game theory (along the lines of Axelrod’s The Evolution of Cooperation) and behavioral economics (along the lines of Freakonomics ) to “explain” the many (and there are many) ugly social behaviors that are characteristically Indian, ranging from not boarding planes in orderly ways, to jumping queues, to littering. I put scare quotes around explain because the book doesn’t really offer fundamental explanations so much as it builds a posteriori phenomenological models. And that’s why the book is depressing — it paints Indians as culturally hopelessly-flawed and impossible to reform. The book is a classic example of  S. Gurumurthy’s tongue-in-cheek acronym GURU — “great at understanding, relatively useless.” Full disclosure: my middle name is also Guru. Really.
So my summary opinion of  VR’s GIP is GURU. I am not entirely critical — I do think the book asks the wrong questions, but they are interesting ones nevertheless, and the book is a pleasant one for a medium-longish flight. But ultimately Raghunathan is playing yet another weakness-game Indians like: the impossible-problem game. We even have an idiomatic Indian-English phrase for this self-identity: we are like this only, mind it (if you can’t parse that English, never mind, long story). Now let’s move on to VGR’s OGIP, which I hope, is less GURUish and more useful.
Other Games Indians Play
You don’t normally put “games” and “Indians” in the same sentence. The stereotype, largely justified, is that we are beyond embarrassingly bad at most modern international sports. We will undoubtedly embarrass ourselves yet again in 2008 up North, at our friendly neighbor’s (in China, for the slow). The board games that history associates in some way or or other with India — Chess, Parchisi and the like — are usually used as evidence of our tactical, pointlessly and self-indulgently cerebral,  metaphoric and metaphysical approach to life.
But this is not about these “usual” themes that come up in conversations about Indian sports. This is about 3 games that, as far as I know, nobody else in the world plays but us. There may be related games in other cultures, but key to this article is the fact that these are popular Indian games. We (at least my generation, I don’t know about the Indian Gen Y) grew up playing these games.
Look for these six themes as I describe the games. I’ll elaborate later.

Individual-Group Tension: First, you’ll notice a common thread of tension between individuals and groups in all these games.
Creative-Destruction: Second, you’ll notice an element of creative-destruction, especially in the third game, Lagori.
Breathlessness: Third, you’ll notice that all three games are breathlessly fast. They go beyond ‘think on your feet’ fast — they are what I think of as sub-tactically fast, even faster than basketball. You have to play almost on pure instinct, interspersed with occasional conscious decisions that must be made in almost no time at all.
Burst-Reversal-Energetics: Fourth, you’ll notice that the games have an odd cadence of very short periods of relaxation punctuating fast context switches between opposed bursts of action.
Attrition: Fifth, all three games have a last-man-standing element to them
Leaderlessness: Sixth, none of the games have captain-roles, or indeed, any fixed roles. All roles are situational, not formal.

Kabbadi
Kabbadi is a territorial defense game. Sides alternate in sending an invader (Red team in the picture below) into enemy territory. The invader must touch (‘tag’) as many opponents as possible and escape back into his territory. If he succeeds in getting back, everybody he tagged is out. If the opponent team manages to tackle and pin him in their territory, he is out. The catch: the invader must carry out his entire raid while holding his breath. As evidence of holding his breath, the invader must chant something rhythmic out aloud very rapidly. The usual is the word kabbadi. The game has alternating raids going on in rapid succession, with barely a pause to take a breath. Eventually attrition of tagged or pinned players leaves one person standing, and his team is then the winner.

Kabbadi can be brutally violent, more so than Rugby or American Football even, in some ways. It is a full-contact wrestling sport (often played in sandy courts, much like traditional Indian akhada wrestling). It also involves very rapid teamwork on the defending team’s side, since they must avoid getting tagged until they’ve created a formation (a linked semi-circular ring of defenders circling warily around the attacker, attempting to get between him and the center line, is a common formation). Once they see an opportunity, they tackle. The raider, on the other hand, tends to dart around to avoid getting hemmed in, while trying to pick off the defenders one by one, often using long-range touch-kicks. This is a very different sort of tackling element than in American Football or Rugby.
A fun fact. Many kids like to chant little rhymes and repeat the last phrase, instead of repeating a single word. I grew up with this one:
Chal kabbadi aan me
Joota mare kaan me
Gir gaye maidan mein, maidan me, maidan me…
That translates to “Let’s go play kabaddi in the courtyard, let’s kick each other on the ear, let’s fall down on the playing field, on the playing field, on the playing field…”
The little ditty captures the ethos of the game quite well.
Kho-Kho
Kho-kho is like reverse tag on crack. “It” (blue in the cartoon) runs around a line of other players, who squat in runners-start positions, alternating the directions they face. One of these players is chasing “It” at any given time. The rules: “It” can run in either direction,while staying within a boundary. The chasers on the other hand, can only run in one direction (anti-clockwise at the moment in the picture). Their advantage: the current chaser can tag a squatter on the back with the word kho and have him take up the chase, and the new chaser has an opportunity to switch directions, but only once for his sprint, until the next kho (in the picture, a kho is in progress between the yellow dot and green dot guys in the picture). Effectively, chasers chase in a line-crossing relay. I grew up playing it as a one-versus-many game. When “It” is eventually caught (as is inevitable), the tagger becomes the new “It” and you play till you are exhausted. In more organized forms, it is played as a team sport, with one team sending a sequence of “It” runners and then taking their turn at being the chaser team.

One dynamic is worth mentioning. A just tagged chaser will try to defer his direction decision as much as possible (a few yards before he runs out of bounds). Much of the elegantly-minimal level of strategic thinking in the game hinges on your choice of direction on being kho’ed.
Lagori
Called saat-pathar (seven stones), pittu and several other names, Lagori is the most complex  popular  children’s game in India, and is rather like Dodgeball, but more aggressive. A pile of 7 stones is arranged in the center of the court. The defending team has its players taking turns attempting to knock down the pile with a tennis ball, from a throwing crease line. While there are variations, the version I played had each person getting three chances. If you knock down the pile, but somebody in the opposing team catches the ball, your entire team is out and play switches sides. But if you knock it down and they don’t catch it, then the real game begins. The defending team, which brought down the pile now goes about trying to stack it back up, while the attacking team attempts to hit the rebuilders with the tennis ball. The dynamic is very weird, as the rebuilders dart around near the scattered pile, trying to rebuild, and repeatedly scattering as the opposing team takes shots at them. Both attackers and rebuilders tend to coordinate in emergent ways. There are no formal leaders. If the rebuilders manage to restack all 7, they win. If the attackers manage to hit every one of the rebuilders with the tennis ball before the restacking is complete, they win. Then they get their turn at attempting creative destruction.

The point to note about Lagori strategy is that you should attempt to disrupt the stack of 7 as little as possible. Preferably, just the top stone being knocked relatively close by, with a glancing blow from the ball. That will mean your rebuilding will be trivial. Blast all 7 all over the place with a  powerful direct hit, and you are in for trouble rebuilding.
An aside: Lagori can turn brutally Lord-of-the-Flies-ish among tween boys, with the focus being on hurting with the tennis ball rather than the game strategy. We occasionally played a raw, bloody game called maar-peet (loosely: hit-and-beat-up) which dispensed with the complexities altogether and just had us attempting to hit each other with the tennis ball.
The Analysis
Let me elaborate on my themes from before, in the context of an Indians-in-the-workplace business metaphor, now that you have understood the games.

 Individual-Group Tension: Indians are staggeringly good at very rapidly “reading” organizations. The one-vs.-many isn’t, as you might think, a David-Goliath oppression narrative. We fit neither the collectivist ethos of the Sino-Japanese cultures, nor the teamwork-among-individualists culture of the West. Ours is a culture driven by a constant tension between sheathed-claws individual and wary collective, and we wear both hats at the same time. Our games are homage to a (culturally-natural I think) ability to appreciate that groups have more power, but with more constraints, while individuals can be more agile, but will eventually run out of energy. Kho-kho in particular, epitomizes this spirit. This same point is what struck Clifford Geertz about cockfighting in the culturally-somewhat-Indian Bali; the cockfighting is largely about allowing individuals to read an apparently static culture. Geertz’s famous article, Deep Play, explores this.  The tag-line for ribbonfarm is “deep play for disruption.” This strength also manifests itself as a weakness — one that Raghunathan explores at length. We like figuring out how to beat a superficially “fair” system, just for the hell of it. When we play for a system or group, we tend to have a natural ability to see individuals through the eyes of the organization if necessary, a good managerial capability. In pathological form, there is a reason the Indian babu was the heart of the British Raj — we make for some of the most creatively-obstructive petty bureaucrats.
Creative-Destruction: I have written before about the metaphysics of creative-destruction, but you don’t have to get to the metaphysics to appreciate this point. More than any other culture, Indians truly and instinctively get the idea that there can be no growth without pain and acceptance of true destruction of some form. Not nice, nominal destruction, but real destruction, with hurt people and broken systems. We do not toe the line like Germans or Japanese (or seethe internally). We do not have a Gentleman’s code of conduct like the British, or the locker-room backslapping camaraderie of the Americans. We are a collective of utterly primal individuals who maintain a facade of polite cooperation but are always driven by an itch to break and remake the system to our advantage. Nurtured right, this leads to good business-model innovation skills. As a pathology, it can turn into purely disruptive tendencies.
Breathlessness: This is one of the least appreciated Indian strengths, and is a product of an extraordinarily powerful tradition of live, public, think-on-your-feet argumentation, honed through centuries of debate among dueling scholar-warrior metaphysicians of Vedantic and Buddhist varieties. Amartya Sen explores the theme in The Argumentative Indian but I haven’t read it, so I won’t comment more on that. But the result is that Indians, in a business environment, have a natural inclination towards rapid, conceptual sparring and debate. We are rarely the best doers in most meeting rooms, but rapid and nimble mental gymnastics and reframing is a characteristic of strong Indian managers (as a pathology, it shows up as a tendency towards just-so analysis and sophistry). We don’t need to go away and think things through before reacting to information.
Burst-Reversal-Energetics: We — at least the privileged, dalit-exploiting upper castes — have a much-deserved reputation for being a lethargic culture. But this is coupled with an aptitude for watchful opportunism and a sense of leverage and timing. Indians are (in a good way) naturally opportunistic workers. We try to minimize the effort required to achieve a desired outcome by just waiting and watching for the right opportunity. We are also amazingly good at switching contexts on a dime and suddenly going from sheer resistance to sheer support for something. Perhaps that’s why we manage to run the world’s largest, most unruly democracy, venal politicians and all. That’s also perhaps why (besides English skills) we managed to bootstrap our economy into the global economy using Y2K as the opportunity.
Attrition: This is an attitude more than a strength — Indians are perhaps a culture that is very much at home, philosophically, with natural processes of death and decay. Yes, in our games we count points upwards, but we like the fundamental dynamic to be a last-man-standing dynamic. There is also significant overlap between this trait and an Absurdist, Sisyphean and existentialist worldview. Why does it make for good management strengths? Because we accept disruption, change, Sisyphean rock-rolling and other business inevitabilities — all attitudes that make for pragmatic management styles. At more immediate, day-to-day levels, we are probably far more willing and able to engage in bloody-minded conference-room brawling, take-no-prisoners debating, and zero-sum warfare. Not the pleasantest of attitudes. Today, most Indians in Western-style workplaces suppress any such tendencies, but the naturally cultural training is there in the background, and when Steven Covey and his bromides leave the room, we are always open to the possibility that the current situation is not a win-win one, and that there may be a need to fight and fight hard.
Leaderlessness: This might seem very odd as a comment about a culture which invented the most explicit caste system on the planet, but we are actually not very status conscious within caste or class boundaries, which is what matters since historically there wasn’t much interpersonal interaction across boundaries anyway. Power flows fluidly (though not openly, except among kids) among individuals of the same nominal class in very situational and context-sensitive ways. Children talk back at parents, young men challenge the old. Labels such as CEO or VP matter little (though there may be as much ritualistic deference to nominal superiors as in China, this does not run deep. There is a lot more back-room insubordination). Students smart-mouth teachers. There is a whole cat-may-look-at-a-king quality to our approach to interpersonal relations.

I’ll conclude (barring that pesky endnote) with a quote from an obituary of P. V. Narasimha Rao, a quintessentially Indian Prime Minister, by M. J. Akbar, a leading Indian journalist:
[PVN’s attitude] was the loneliness of a long-distance Brahmin runner, for the
intellectual in him was also the Brahmin in him. He did not advertise his
innate superiority of insight and scholarship. There was no need to. It
was obvious. But he could never be one of the boys, if you see what I
mean. Is that because he was always one of the adults?
— M. J. Akbar, writing on P. V. Narasimha Rao, Dec, 2004.
I’ll leave you to interpret that as you will. Now for that endnote so I don’t get skewered for my blunt opinions.
Endnote

My generalizations aren’t intended as stereotypes, but as shorthand. I’ve known Indians who don’t fit this mold and many non-Indians of all sorts who do. But our culture does tend to bring out these aspects of our personalities if we have them.
You might suspect that this is a Brahminical view masquerading as a pan-Indian view. The short answer: No. The reasons would take another 10,000 words, but this analysis applies to non-Hindu Indians and Dalits as much as it applies to the historically oppressive upper castes.
Yes, my language is gendered, because these are quintessentially boys’ games in India. Girls do play them, but Indian women turn out to be the way they are due to other forces, that merit yet another essay.
No, I don’t mean to be cultural-essentialist. The socialization/enculturation that Indians grow through is simply one among many forces that make us who we are, to the extent that we have certain characteristic cultural traits.
Yeah, we could talk about Cricket too. Another time.

The Other Games Indians Play

At the straw market in Nassau, in the Bahamas, — famous for stuff like the straw handbags below — I recently encountered a distinctive culture of bargaining that made me stop and ponder the subject (on a beach, aided by rum). The pondering resulted in a neat little flash of insight that allowed me to synthesize everything I know about the subject in a way that surprised me. The short version: game-theoretic and information-theoretic approaches to the subject are something between irrelevant and secondary. What drives bargaining behaviors and outcomes is story-telling skill. Here’s how you can learn the skills that really matter in being a successful bargainer.


The Simple, Elegant and Wrong Answer
Buyer A and seller B are haggling over product P. Neither knows what price the other is willing to settle for, but A wants P at the lowest price B might sell, and B wants to sell at the highest price A might be willing to pay. You model the situation as a sequential move game, with each move being, at its simplest, a price. You might represent the progress of the game thus, as pairs of B-A call-response moves:
($200, $100), ($180, $120), ($160, $140), ($150, SOLD!)
At each turn, each player uses the history of previous offers/counter-offers to estimate the true limit of the other party, and makes an offer that induces the other to move his price-point as much as possible while revealing as little as possible of his own limit. Throw in some bounded rationality and a value on time, and you have the sort of framing economists like.
You could mathematically model this (no doubt, somebody already has analyzed this to death and proved all sorts of convergence results). One example of such an analysis is in the classic game-theoretic decision analysis book, Thinking Strategically (an excellent read, by the way, for what it sets out to do).
But neat though this mathematical formulation is,  it is fundamentally wrong-headed. Bargaining isn’t primarily driven by parties attempting to (bounded) rationally guess each other’s limit points by doing (non-trivial) real-time analysis of number sequences. Yes, the alternating sequence of numbers does carry information, but in most real situations, information is primarily conveyed in other, non-quantitative ways, and the relevant information isn’t even about price. Let’s examine two examples before I present my alternate model of bargaining.
Real-World Examples
Example 1: The Bahamian Turtle
In Nassau, I bought this coconut-shell toy turtle for my nephew, for $5:

The seller, a pretty young woman, came up to us and engaged us with, not one, but three moves all at once, and persuaded me to close, before I had a chance to make a counter-offer:
Seller: You like this turtle? Nice toy for child! Eight dollars!
Seller (conspiratorial whisper, eyes darting left and right): Tell you what, I make you a deal, only six dollars.
Me (doubtfully): Hmm…
My Wife (enthusiastically): oh, it’s so cute; we should get one for Arjun [my nephew]!
Seller: Alright, I give it to you for five dollars. What color bead you want lady? [the thing has a bead on the string that is not visible in this picture].
This one is, in a sense, a non-starter example, since I got played before we got to bargaining. Let’s take a longer example where I acquitted myself better.
Example 2: The Jaipur Good-Cop/Bad-Cop
In Jaipur, India, on a vacation recently, my wife wanted to buy herself a traditional Rajasthani kurta, or shirt. Translating from Hindi, the exchange went roughly as follows (I may be misremembering the prices, but the gist is accurate):
Me: How much?
Seller: 300 rupees
Me: That’s too much, how about 175?
Seller: Come on sir, at that price, I won’t even recover my costs!
Me: No, 175 is the reasonable price for this kind of item.
Seller: Arrey, come on sir! Just look at this fine needlework; you may have seen similar stuff for less in other shops, but if you look closely, the work isn’t as delicate!
Me: Of course I can see the quality of the work, that’s why we want to buy it, now come on, quote me the right price.
Seller: Okay sir, for you,  I’ll let it go for 250 (starts folding up the kurta).
Me: No no, this lady may not be Indian, but I am; be reasonable [my wife is Korean, and since I hadn’t mentioned that she was my wife, the shopkeeper had almost certainly assumed I was her local guide — many other shopkeepers had in fact called out to me to bring her into their stores, offering me a commission!]
Seller: But I did quote the price for you sir, for foreigners, we normally ask for at least 4-500!
Me: Fine, tell you what, I’ll give you 190.
Seller: Come on sir, at that price, I don’t even make a profit of 10 rupees!
Me: Fine, let’s do this deal. 200; final offer.
Seller (looking upset): But…
At this point, the seller’s boss, probably the store owner, who’d been poring over a ledger in the background, looked up, interrupted and said shortly, “Can’t you see the lady wants it? Just give it to them for 200, let’s cut this short!”
I have several other examples I could offer (in the US, bargaining tends to be restricted to larger purchases like cars), but these two examples suffice to illustrate the points I want to pick out.
The Phenomenology
There are several features of interest here. Here is a round dozen:


Fake moves: In the Bahamian example, consider the rapid series of three prices offered with a very quick change of subject to the color of the bead at the first sign that I wanted to buy. This bargaining is clearly fake, the numbers being part of the initial courtship ritual rather than the actual price negotiations, which were short-circuited.
Bargaining as bait: The sellers in the Nassau marketplace  promote their wares with a curious mix of American retail rhetoric (“C’mon honey! Everything 50% off today”) and more traditional bargain-hunter bait (“You want a handbag sir, for the pretty lady? C’mon I make you a deal!”). I suspect very little serious bargaining actually takes place, since the customers are largely American cruise ship tourists, who are not used to bargaining for the small amounts in play in these transactions.
Qualitative Re-valuation: Consider the variety of non-quantitative moves in the Jaipur example. In the ‘fine needlework’ move, the seller attempted to change my valuation of the object, rather than move the price point. I accepted the point, but indicated I’d already factored that in.
Narrative: A narrative also developed, inviting me to cast myself as the knowledgeable insider who was being offered the smart Indian deal, as opposed to the high-mark-up offered to clueless foreigners. This is a key point that I will return to.
Deal-breaker feints: Twice, the seller attempted to convince me that I was offering him a price he could not accept. These are rhetorical feints. A similar move on the customer’s part is to pretend to walk away (that old saw about the key to negotiation being the willingness to walk away isn’t much use in practice, but pretending to walk away is very useful).
Closure Bluffs: another interesting feature of the Indian example is the closure bluff; a non-serious price accompanied by closure moves (such as starting to package the item), on the off-chance that the other party may panic and fold early.
Good Cop: Finally, note the second individual stepping in to make the deal towards the end (this dynamic is particularly common in traditional retail stores in India, where the owner, or seth and accountant, or munim, will often be watching the salesmen at work, stepping in at the right moment. The psychological key to this is an implicit sense of escalation and respect: “forget that unimportant lackey, clearly you’re a smart customer and I, the boss, will deal with you personally and cut you a special deal that my lackey isn’t authorized to offer.”)
Ritual: In most cultures  of bargaining, there are also  moves of ritual significance. In India, the best-known one is the bohni,  or first sale of the day. Sellers will often plead with customers to close the deal, since it would constitute the bohni and also assert that the price being offered is a really good deal because of the seller’s anxiety to finish his bohni. This is not entirely a pragmatic sort of move — the bohni does matter to traditional merchants, who will often actually take a bit of a hit on the first transaction for luck, to get the cash flow started. Curiously, I also encountered a much more open version of this in the Bahamas, where one shopkeeper said it plainly, “C’mon honey, first customer gets best deal!” I suspect anthropologists would find an equivalent in every culture besides modern fixed-price retail.
Knowledge Bluffs: Though I didn’t use them, knowledge bluffs are common in bargaining (“I saw that same thing in Delhi for half the price!”) . These are bluffs because if the buyer really knows something about the cost structure of the seller, that information is usually quickly deployed and factored out, reducing the bargaining to a matter of “what is the convenience-value to me of buying here rather than in that other place?” Why do they still work? I’ll tell you in a moment.
The Justice Bluff: Surprisingly, in the form of a bluff, a notion of fair price often enters even the purest free-market exchanges. This  is usually brought into play via displays of mock anger at being treated unfairly. Surprisingly, even sellers will do this, attempting to convey a clear sense of disgust by putting away the items under discussion. But there are boundaries: sellers will rarely plead to make a sale on the basis of personal need, since this subtly moves the situation from a peer-transaction to a (morally unacceptable) charity transaction. My mom once bought some vaseline from a tearful door-to-door saleswoman who apparently genuinely broke down and said, “buy some just out of sympathy please!” My mom caved.
Boundary Blurring: If a full transaction has three parts (discovery/selection, negotiations, closing), elements of bargaining often creep out of their nominal home in the middle section. For example, in traditional Indian full-service retail, the sales staff will often pull out a vast selection — more than necessary or asked for — visibly doing a lot of work, creating a clear sense of pressure. On the other side, towards the end of a transaction, sometimes the customer will throw in surprise in-kind requests after the deal seems closed. “Alright, I am buying this car from you for more than I wanted, how about you throw in some floor mats?”
Non-influence of actual knowledge: Though there is a lot of bluffing, there is not much actual information about price limits in play. In every example I’ve encountered, at least for small amounts, buyers and sellers do not attempt to guess limits directly (beyond having a ballpark “reasonable” figure in mind that may be utterly irrational). It is obvious that the buyer may not even have a “walk away” limit price in his/her head, but what is not so obvious is that even the seller may not have such a price in mind. Cost structures can be murky even to sellers, and other hard-to-value elements may be in play, like prospects for future sales, desire to clear slow-moving inventory, and the like. Actual price information usually enters the picture only when the amounts are significant, in which case the parties generally do their research beforehand, and attempt to factor that information out of the bargaining stage altogether. Bargaining is primarily about ownership of the unknown factors in pricing, and is a high-cost process, and it is in the interest of both parties not to bother bargaining about mutually-acknowledged certainties.


The Right Model
So that’s all very well. There are lots of psychological subtleties involved. But is this all really important? Is it possible to just cut the Gordian knot and become really good at some sort of game-theoretic model? Would all the bluffing and qualtiative nuances vanish under the right sort of time-series modeling?
The answers are yes and no respectively. Yes, you do need to work with the full thing; game theory won’t cut the Gordian knot for you. And no, you will not be able to subsume all the bluffing and complexity no matter how much you crunch the numbers. So you do need to appreciate the qualitative sound-track of the bargaining, but no, don’t be discouraged — I am not suggesting that the only meaningful model is a localized sui generis ethnography. Universal models and approaches to bargaining are possible.
What actually happens in a bargaining transaction is the co-construction of a storyline that both sides end up committing to. Every move has a qualitative and a quantitative part. The prototypical transaction pair can be modeled roughly as:
((p1, v1) , (p2, v2))
Where the p’s are qualitative statements and the v’s are price statements. The key is that the qualitative parts constitute a language game (in the sense of people like Stalnaker). Each assertion is either accepted or challenged by subsequent assertions. The set of mutually accepted assertions serves to build up a narrative of increasing inertia, since every new statement must be consistent with previous ones to maintain credibility, even if it is only the credibility of a ritual rather than literal storyline.
This is the real reason why there is apparent spinning-of-wheels where the price point may not move for several iterations. For example in the Indian kurta case, I rejected the seller’s assertion that 175 would represent a loss, but acknowledged (but successfully factored out) the “this is fine needlework” assertion. Though the price point wasn’t moving, the narrative was. At a more abstract level, a full narrative with characters and plot may develop. This is also the reason why knowledge bluffs work — even if the seller knows the buyer cannot have seen the same item for half the price in another store, he cannot call out the bluff in an obvious way since that would challenge the (always positive) role in which the buyer is cast.
The key conclusion from all this? The transaction moves to closure when the emerging logic of the narrative becomes overwhelming, not when price transparency has been achieved. To bargain successfully, you must be able to control the pace and direction of the development of the narrative. At a point of narrative critical mass, something snaps and either a new narrative must displace the old one (rare), or there must be a movement towards closure.
Becoming a Right-Brained Bargainer
So here is my magic solution: become good at story-telling based conversations.

Walk in, not with a full-fledged plan/story, but a sense of what roles you can comfortably fill (straight-dealer? cynic? know-it-all? Innocent student without much to spend?)
As the conversation progresses, try to sense what roles the other party is trying on for size, and suggest ones favorable to you (“Look, I try to buy only from local merchants, and you guys do are doing a great job for the economy of our town, but…”). Say things that move towards a locked-in role on both sides that favors you. In the example above, I got “locked in” into the role of “knowledgeable local” on disadvantageous terms.
Look out for the narrative logic as it develops. For example, I successfully resisted an attempt to bring “fine needlework” assertion into play, which would have moved the story from “guy looking for cheap deal” to “connoisseur transaction” and a premium-value storyline.
There are critical/climactic points where you can move decisively for closure; watch and grab. In my case, I thought I had one when the seller offered the “Not even 10 rupees” move, but the owner cutting in for the kill and accepting was a clue that I could have pushed lower.
Be aware of the symbolic/narrative significance of your numerical moves. If the seller moves from 200 to 180, and you move from 100 to 120, the very symmetry of your move shows that you have no information at all to use for leverage, and the transaction is likely to proceed dully to a bisection unless you do something creative. If the seller offers 500 and you say 250, that reveals that you may be using a “start at half” heuristic, which might create an opening for the seller to launch a storyline of “really, 500 is a fair price, here’s why.” Offering “275” instead creates the right sort of ambiguity. If you do want to drive towards a symmetric-bisection storyline, make sure you pick an irrational starting point, but not one so irrational that it reveals you know nothing about the price (irrationally low opening offers can work, but you need a 201 level bargaining course to learn why).

Now, this isn’t easy. You have to become a storyteller. But I never said I was going to offer an easy answer; merely a better one than a misguided attempt to do real-time game-theoretic computations.

Bargaining with your Right Brain

A private sense of humor has been key to my sanity for several years now. It has helped me deal with the label Idea Guy that I acquired as a kid, and have never since shaken off. Once I realized that it was a (very) mixed blessing, I went from downplaying it, to being alternately defensive or assertive about it, to resisting it, to trying hopelessly to “fix” it, to finally finding a way to accept and manage it. Acceptance, for me, has involved a private joke of a self-image  that is a mix of Camus’ Sisyphus and the Max Fischer character in Rushmore, coupled with a model of the ‘Idea Guy’ mental style as a chronic medical condition. I don’t know if you’ll find the medical metaphor in this piece funny, but you might find it useful.  This is not a piece about becoming an idea person. That is not something you choose. It is a condition you have to manage, like diabetes, once you recognize  it.
A well-known, highly-productive professor (measured in terms of number of PhDs supervised and papers published), once confided an opinion  about a colleague to me: “He’s a visionary and very smart, but he hasn’t graduated any students yet, though he’s been here as long as I have. At some point, if visionaries don’t produce any output, people just stop listening.”
The anecdote teases apart the notion of the idea person, the personality type, from the professional ideator, a (more common) socially-defined role that can be learned, and performance within which can be measured. Even if you are in a research lab, like I am, where everybody is supposed to be an ideator, generally the idea people will be easily identifiable (hint: follow the derisive laughter).
The professor’s attitude is a pragmatic, statistically reasonable attitude towards idea people.  Most of the time, the attitude will turn out to have been justified.  If you are an idea person, chances are extraordinarily high that you will die without having had any impact whatsoever on the world, because of the demand/supply economics of ideas that I’ll cover later. So you have to develop the right philosophical attitude (hint: Sisyphean), which I’ll cover in the treatment/management section.
Diagnosis
Idea-personhood is caused by an underactive piece of neural machinery coupled with an overactive piece in another part of the brain. The overactive part generates more what-ifs about the world than are necessary for survival.  The underactive part fails to stop the runaway train of what-if thinking efficiently. Someday neuroscientists will locate these pieces in Latin. The result is a mental condition of near-constant ideation. A flow of ideas, from trivial to substantial, that far exceeds the amount of statistically necessary brainstorming in any socially useful role. Far more than necessary to fuel those deluded rational innovation processes like TRIZ.
The condition also creates a voracious appetite for a very unhealthy kind of intellectual diet: heavy on concepts, a little lighter on facts (favoring  qualitative and historical rather than quantitative and current) and rather light on how, or skills/procedural knowledge (though we are quick skill-learners as a breed, we rarely get beyond ‘amateurish’ at anything).  That’s like having a congenital inclination to overeat fats and simple carbs with too little complex carbs and protein, and no exercise to integrate the protein into muscle and maintain it. Again, sufferers don’t choose that mix of cravings.
So an idea person is a concept-heavy, story-loving, reluctant-to-face-data amateurish dabbler in a lot of different things.
If you’ve only ever tried brainstorming when told to do so by some coach or teacher in a classroom exercise, and found it unnatural, you don’t have the Idea-personhood condition. For an idea person, it feels forced and unnatural to stop the process. We suffer chronic or episodic insomnia as a result. For the idea person, the idea of starting the process is  not separable from ‘waking up.’ It can even intrude on dreaming.
So if that sounds like you, try to confirm the diagnosis.
Confirmation
There are no good personality tests that I know of, that unerringly nail Idea personhood. But honest autobiographical analysis can help. Here are the signs you should look for, based on what I see as evidence along my own trajectory.

Lemonade-stand entrepreneurial instincts:  I was always the guy in my class who came up with the idea for, and led, the class booth at the school fair. In fifth grade it was a coconut shy, next came a maze navigation game with remote-controlled cars. Then an instant-noodle booth. Finally, an air-gun balloon shooting booth. My team turned a profit each time, but I have no idea where we ranked in the money-making stakes. Probably not very high. Idea people get in the game for the fun of building something out of pure conceptual hot air. While I have been involved in “real” entrepreneurship by way of an Internet startup, that’s not as reliable a sign, since lots of Idea People never have the opportunity, and lots of non-Idea people are attracted to adult entrepreneurship for reasons besides having ideas (like “making money”).
Execution Failures: You should be able to spot incidents when you disappointed people by producing output that was wimpy compared to your big talk. I’ll cleverly not list mine, but trust me, I have several. None really big though. I wised up quickly here.
Narrative/Communication instincts: I scripted several class amateur theatricals, founded our school science magazine (the Loyola Probe; I wonder if it is still alive), privately wrote dozens of short stories, lots of bad poetry and even one embarrassing unfinished Hardy-Boysesque novel. Tellingly, I was fairly good at extempore public speaking, but pretty terrible at what Indian schools call ‘elocution’ (memorized speeches) competitions. In more pathological contexts involving brain-damaged patients, skill at extemporaneous public speaking becomes confabulation. If I ever go mad, that’s the likely route. People sometimes ask me how I blog so much both here and at work. The right answer: it is an effort to stop myself.
Organizational instincts: Idea people like to start (but not run) organizations. Preferably around novel processes. In school, I co-founded an airplane enthusiast club and a computer club (neither succeeded). I’ve also started several mailing lists and online communities of varying success levels. We like flowcharts. But these instincts nearly always end up badly without good support from people with complementary strengths, because we are not process people. We like to invent them (usually in an overcomplicated first-draft form that we then have to simplify), but we have trouble following them ourselves because we are easily distracted, and attracted to improvisation and short cuts.
College Dark Ages: Higher education tends to be a bad time for idea people. College is where amateur talents become professional, money-making skills. It is where the truly skilled competitors forge ahead, and coasters, even if successful in grade school, drift to rankings-mediocrity. I stayed near the top of my class with hardly any effort through high school, but “hardly any effort” got me a comfortably obscure middle-of-the-class ranking in college. Even the  Max-Fischerish extra-curricular dabbling doesn’t work out, because there are people around who turn that into a professional-quality competitive game as well. Graduate school gets a little better, and with an understanding adviser, a PhD program can be downright comfortable, but the rank/numbers-competition based student environment is never really friendly to idea people.
Can’t-Help-Myself Disruptive Work Behavior: Posturing by a few jerks aside, most idea people don’t actually set out to be disruptive. They don’t toe the line because they have a lot of difficulty seeing it. Others by contrast, seem to see brick walls whose existence idea people can only infer in an abstract sense. This means unless they’ve learned to manage around their boundary-blindness and get the right kind of support, idea people will tend to get fired, sidelined, underutilized, ridiculed into obscurity or otherwise end up in unproductive places. Think cat-personalities versus dog-personalities; hierarchy sensitivity versus hierarchy obliviousness.

Prognosis if Untreated

I’ve know good idea people, who’ve gotten the condition under control and found themselves the right environment to be productive members of society. But far more often, I’ve encountered idea people who’ve left their conditions untreated and unmanaged (remember my unhealthy diet metaphor? think mental obesity). The outlook is grim, and proceeds through several clear stages. You will progress as follows:

The Evangelical Phase: You will spend some years talking too much and trying to make others do things through force of argument and vision-painting alone.
The Bumbling Phase: You will recognize the need for proof belatedly, and for some project, over-estimate personal abilities and try to do too much all by yourself, with weak, amateurish execution skills, leading to ridicule-worthy output that even you can see looks nothing like the starter visions.
The Loss of Credibility Phase: You’ll give up execution, but continue talking. At some point people will stop listening. Credibility will become a distant memory. If people around you are sufficiently lacking in compassion, you will also endure overt ridicule.
Descent into Madness: I have seen one truly sad real example of this (my friends from undergraduate years can guess who I am talking about).

Treatment
Idea-personhood requires lifelong management. It is unlikely to get you anything of value, but it still needs to be managed to prevent madness.  Here are the things you need to do.

Learn the UnSkills: Craftsmanlike people will always beat idea people at anything where skill and virtuosity can be evaluated and indefinitely improved. There is a set of what I call “Unskills” that, however, craftspeople ignore because they don’t think of them as learnable, improvable skills. These include a sense of narrative, a sense of metaphor and a sense of design. You already have these senses. Develop them through play, not practice. This blog is it for me. They are nicely described in Dan Pink’s Whole New Mind.
Learn the “Braking” Tricks: You’ll never get anything done until you learn a few tactics to shut off your brain. I use TV, walking, getting drunk, cooking, and eating. Anything based on distraction tends to work. Driving is dangerous though — I’ve had two accidents due to driving while thinking too hard. If you are thinking furiously and uncontrollably, don’t drive. One subtle trick I learned was ‘give the runaway train of thought enough rope to hang itself.’ If your idea stream seems unstoppable, do everything you can to accelerate it. At some point it will just run out of steam out of sheer neural exhaustion, and you can get some sleep.
Accept Bipolarity: You are borderline (or full-blown) manic-depressive. Which means your energy swings are fairly wild. Learn to live with it rather than attempting to mimic the steady energy flows of the virtuoso doers. Get your work done in your manic bursts, stop apologizing for long periods of apparent sloth, and tackle swings towards the depression end with busywork and mindless chores that you have to do at some point anyway. You might as well use your otherwise useless lows.
Adopt Sisyphus as your Model: For reasons I’ll outline in a minute, there is a good chance you will never have an impact. Your life will likely be one long series of episodes of rolling rocks up hills, only to see them roll down again on the wrong side. Learn to accept and enjoy that process. Camus’ Myth of Sisyphus should be your bible. Existentialism should be in your attitude toolkit.
Learn to Manage Perceptions: More than for almost any other personality type, Idea people must actively manage perceptions, because they don’t have the luxury of having strong, silent actions speak for themselves. If you ever have an impact, chances are it will happen through more skillful and intelligent people believing in your ideas — Idea person-hood is not a form of intelligence. You are the sum total of your ideas, not your actions. Ideas, by their vary nature, are objects of ridicule and derision until proved sound. If you don’t manage the way you are perceived, you will default to ‘ridiculous.’
Get out of Mutual Admiration Circles: Idea people, often beaten into submission by powerful and skillful doers around them, can retreat into co-dependent circles with other idea people. Don’t do this. Just don’t.
Get into Complementary Teams: More than almost any other type of person, Idea People need others. We’d die in Robinson Crusoe mode. In all but a small collection of fields, we are incapable of adding much value by ourselves. Recognize that, mourn for a bit for your loss, but then get started trying to find people with the strengths that make up for your significant failings. Win their trust, and hope something good comes out of it.
Get your timing/positioning instincts honed: Idea people depend on being in the right place at the right time to see their ideas gain traction and go somewhere. Develop your restlessness and keep moving (your personality will propel you to motion anyway, but only conscious will can direct you towards the right opportunities). The specific time/place where you might get a win will be an accident. That you eventually will get at least a small win will not be, so long as you keep moving.
Be Maze-Bright: You aren’t naturally good at numbers’ games beyond a certain level. You are neither motivated by, nor skilled at, winning numbers games at professional levels (papers, patents, number of novels, column inches of journalistic output). But you need to play these maze games just well enough to get the passports you need to access opportunity spaces. Don’t look for your wins there though. You’ll either succeed by causing a couple of major/minor earthquakes, or by hitting a rich vein as a first-explorer where you’ll turn into a hyper-productive winner in a numbers game that YOU define.
Develop Gap Sensitivity: Understand that you are relatively amateurish at everything you think you are good at. You will never be a virtuoso anything. Look around and learn to recognize the people who play your amateur games at the pro level, where things happen with an operational sophistication you can’t even begin to appreciate. I learned this the hard way. I always knew I sucked at chess, but one day in college, a friend talked me into playing. Just a few moves in, another skilled player walked in, took one look and asked, “My God! Who’s getting f***ed?” It struck me forcefully that these skilled players were seeing the game at a level I wasn’t aware existed. I now realize that the same holds true of any high-skill game, from writing to theorem proving to programming (all of which I am thankfully somewhat better at than chess, but by no means a pro). You need these pros working with you if you want to stand a chance at winning any seriously worthwhile game in life.

The Economics of Ideas
So why are Idea People such a besieged and troubled species? It is a mix of probabilistic pragmatism on the part of non-Idea people and unalterable over-supply. From the point of view of a skilled doer, the wisest course through life is to become extraordinarly good at something and get paid a premium to apply that skill. For somebody with a valuable skill, it is possible to completely ignore the idea people strewn like beggars around the landscape, and have a good life just working on validated, low-risk mature ideas. The demand for good, big ideas isn’t as high as people think, because of the simple constraint of execution bandwidth. One Einstein (a classic idea person) can occupy a couple of generations of more-skilled peers. This means a few idea people can basically meet the demand for interesting work from vast numbers of more talented and skilled virtuouso execution types.
Unfortunately, the medical condition is more common than economics demands. There are about 10 times as many idea people as the world needs, and even among those — say 3/10 — who manage their conditions soberly, only 1 will get anywhere. Nine will need to live with a life of zero impact.I think I manage my condition about as well as I can, but I still think my odds of having an impact-ful life are just 1 in 3, and to a large extent, the outcome is going to be random. A Sisyphean sense of humor is necessary. Fries help too.

Read Ribbonfarm

Choose Your Reading Path

Full Backlog

Best Hits

You'll start with:

Book Review: Wikinomics

Concepts and Prototypes

The Silo Reconsidered

About Ribbonfarm