Robin Hanson outlines the content of his future blog. He suggests readers will learn how to overcome biases and become more critical thinkers. He also encourages them to join the discussion on common patterns of bias and self-deception, statistical analysis tools, and social institutions that encourage bias correction.

How can we better believe what is true?  While it is of course useful to seek and study relevant information, our minds are full of natural tendencies to bias our beliefs via overconfidence, wishful thinking, and so on.  Worse, our minds seem to have a natural tendency to convince us that we are aware of and have adequately corrected for such biases, when we have done no such thing.

In this forum we discuss whether and how we might avoid this fate, by spending a bit less effort on each specific topic, and a bit more effort on the general topic of how to be less biased.  Here we discuss common patterns of bias and self-deception, statistical and other formal analysis tools, computational and data-gathering aids, and social institutions which may discourage bias and encourage its correction.  Other topics may be discussed to the extent they exemplify important biases and correction issues.

This forum is brought to you by the Future of Humanity Institute at Oxford University, and consists of posts and comments on posts.  We allow open comments on most posts, but are selective about who can post, and moderate the posts.  To be considered as a poster, please send a link to your thoughtful essay on this topic (published elsewhere is fine) to rhanson@gmu.edu.  Comments should be polite, on topic, and not very long.

Added: See also Contributors: Be Half Accessible.

Disclaimer: The views and opinions expressed in this forum are those of individuals only; they are not endorsed by the FHI or Oxford University.  Copyright is retained by each author.

How To Join

Robin Hanson explores the bias in our education system towards promoting undiscriminating altruism and discouraging the teaching of self-interest. Drawing on the insights of renowned primatologist Sarah Hrdy and gene theorist David Haig, the author questions whether we should approach the teaching of sociobiology (or evolutionary psychology) with the same caution as we do sex education. As the author challenges the prevailing notion of humans as innately self-interested beings, they delve into the price individuals and society may pay for overcoming bias. 

Self-interest goes a very long way to explaining human behavior.  Yet when we educate our young, we prefer to bias them, focusing their attention on the virtues of undiscriminate altruism.  Why?

In Friday’s issue of The Chronicle of Higher Education David Barash accepted (temporarily un-gated version here) Nietzsche’s claim that “society encourages self-sacrifice because the unselfish sucker is an asset to others,” and Freud’s complaint that

[Education] does not prepare [children] for the aggressiveness of which they are destined to become the objects.  In sending the young into life with such a false psychological orientation, education is behaving as though one were to equip people starting on a Polar expedition with summer clothing and maps of the Italian Lakes.

Nevertheless, Barash prefers that we discuss self-interest as we do sex: too little too late:

The teaching of sociobiology (or evolutionary psychology) should be undertaken only with great caution. The renowned primatologist Sarah Hrdy … questioned "whether sociobiology should be taught at the high-school level … … The whole message of sociobiology is oriented toward the success of the individual. … Unless a student has a moral framework already in place, we could be producing social monsters by teaching this." … Preferable, I submit, is to structure the teaching of sociobiology along the lines of sex education: Teach what we know, but do so in age-appropriate stages. Just as we would not bombard kindergartners with the details of condom use, we probably ought not instruct preteens in the finer points of sociobiology.

What reason does Barash give?

There is an altruistic as well as a selfish side to the evolutionary coin. A half-baked introduction to the discipline, which pointed only to the latter, would therefore do students a substantial disservice. … Writes the gene theorist David Haig, "… even though genes may cajole, deceive, cheat, swindle, or steal, … this does not mean that people must be similarly self-interested." … The real test of our humanity might be whether we are willing, at least on occasion, to say no to our "natural" inclinations, thereby refusing go along with our selfish genes. To my knowledge, no other animal species is capable of doing that.

I’m skeptical that humans are special in this way.  But more important, Barash fails to say why the possibility of such a special ability suggests that we should wait to teach children about self-interest.   It seems to me that Barash has failed here to otherwise rationalize his own apparently self-interested behavior in encouraging others to be “unselfish suckers.”

Nevertheless, the key question remains: what price do we pay, individually and socially, for overcoming bias?

Hat tip to Keith Henson.

Addendum:  I’ve fixed the Freud quote, which was completely wrong.  Sigh.

Hide Sociobiology Like Sex?

Robin Hanson explores the fascinating correlation between cognitive styles and prediction accuracy. Drawing from Philip E. Tetlock's book "Expert Political Judgement," Hanson delves into the distinction between two cognitive styles: the hedgehog and the fox. Which one is more successful in predicting outcomes? Take the quiz provided by Tetlock to determine whether you are a fox or a hedgehog, and stay tuned for Hanson's upcoming analysis of Tetlock's findings. This article promises to shed light on the nature of bias and error in our own predictions and analyses.

I read a great book a few months ago that provides insight into the nature of bias and error.  It is Expert Political Judgement, by Philip E. Tetlock. One of Tetlock’s discoveries is that there is a significant correlation between expert prediction accuracy and a cognitive style measure introduced by Isaiah Berlin called "fox" vs "hedgehog". Below the fold I go into more detail on this distinction and present a quiz you can take to determine whether you are Fox or Hedgehog.

Tetlock is Professor of Leadership at UC Berkeley, with a background in psychology and political science.  He ran an experiment for many years from the 80s through the early 2000s in which he invited experts on foreign affairs and political science to make predictions about political and economic trends in countries around the world.  Tetlock tracked and evaluated those predictions, developing a large database of both accurate and inaccurate predictions.  He then analyzed it in various ways to find out how accurate the experts were overall, how they compared to other kinds of predictions, and what psychological and social factors influenced expert accuracy.  These results are what make up the book.

There’s meat here for people coming to the book with a variety of interests and needs, but I will focus on the pragmatic question of what factors influence accuracy, with the hope that learning more about these influences can help us to improve the accuracy of our own predictions and analyses.

One of the psychological measures or metrics which Tetlock found was well correlated with expert accuracy goes back to a distinction introduced by Isaiah Berlin in his book, The Hedgehog and the Fox. I haven’t read that book, but based on Tetlock’s presentation, Berlin distinguished between two cognitive styles to which he gave these colorful names.  The hedgehog is said to know one thing and know it well.  He sees events and trends in terms of his big idea, and aggressively extends it into new realms.  Hedgehogs tend to be confident in the applicability of their fundamental concepts and impatient with those who "do not get it".

Foxes in contrast know many small things which they bring to bear in their analyses in a dynamical and flexible way.  They tend to be uncertain and flexible, "on the other hand" types who are skeptical about their own predictive ability and in fact about the whole enterprise of making predictions in such an intractable realm.

To build suspense, I will not say, yet, which cognitive style was found to be more successful in Tetlock’s experiments.  Instead I will offer Tetlock’s test, from the appendix, along with his scoring methodology used to measure where people stand on the Fox-Hedgehog continuum.  You can take it yourself to decide whether you are Fox or Hedgehog.  In a day or two I will post some of Tetlock’s results as to how the two kinds of experts differed in the success of their predictions.

Here are Tetlock’s questions, which I have converted to a point value, rescaling his weightings which came from factor analysis.  If you agree, give yourself that many points, and if you disagree, give yourself the negative of that many points.  Note that some of the questions have a negative point value, so for those you subtract from your score if you agree, and add to your score (make it more positive) if you disagree.

(7 points) Isaiah Berlin classified intellectuals as hedgehogs or foxes.  The hedgehog knows one bit thing and tries to explain as much as possible within that conceptual framework, whereas the fox knows many small things and is content to improvise explanations on a case-by-case basis.  I place myself towards the hedgehog or fox end of this scale. [Note, I think Tetlock meant to give yourself the +7 points if you think you are a fox, and the -7 if you think you are a hedgehog.]

(-3 points) Scholars are usually at greater risk of exaggerating how complex the world is than they are of underestimating how complex it is.

(-5 points) We are closer than many think to achieving parsimonious explanations of politics

(4 points) I think politics is more cloudlike than clocklike ("cloudlike" meaning inherently unpredictable; "clocklike" meaning perfectly predictable if we have adequate knowledge).

(-5 points) The more common error in decision making is to abandon good ideas too quickly, not to stick with bad ideas too long.

(-2 points) Having clear rules and order at work is essential for success.

(5 points) Even after I have made up my mind about something, I am always eager to consider a different opinion.

(-6 points) I dislike questions that can be answered in many different ways.

(-4 points) I usually make important decisions quickly and confidently.

(5 points) When considering most conflict situations, I can usually see how both sides could be right.

(-3 points) It is annoying to listen to someone who cannot seem to make up his or her mind.

(4 points) I prefer interacting with people whose opinions are very different from my own.

(1 point) When trying to solve a problem I often see so many options that it is confusing.

As noted above, compute your score by adding the point values for the questions where you agree, and subtracting the points (keeping in mind that subtracting a negative means adding) for the questions where you disagree.  Possible scores range from -54 to 54, with negative scores indicating that you are a hedgehog while positive scores mean you are a fox.  The farther the score from zero, the more clearly you fall into the category and the more applicable results based on this cognitive analysis are likely to be for you.

Go ahead and take the test, see if you are a fox or a hedgehog, and I’ll post more shortly describing how these cognitive styles fared in Tetlock’s data.

Quiz: Fox or Hedgehog?


Robin Hanson explores the concept of bias and the effectiveness of social disapproval in discouraging it. Drawing inspiration from the movie "Click" and other works, they question the reality of bias and delve into the conflict between work and personal life. They also analyze the accuracy of expert predictions and discuss the cognitive styles of hedgehogs and foxes. With a focus on improving prediction accuracy, this article promises to challenge your beliefs and shed light on the complexities of bias.

One way to discourage bias is via social disapproval of standard biases.   That is, if when A argues with B, A can show that B’s claim fits a standard bias scenario, then observers could believe A more and respect B less, encouraging B to avoid such claims.   For example if B claims that he is in the top 5% of drivers, and A points out that most people overestimate their driving ability,  observers might believe B less.

There are several problems with this approach.   One such problem is that scearnios widely accepted as identifying bias may do no such thing. 

Which brings me to the recent movie "Click," whose moral is similar to "It’s A Wonderful Life," and "A Christmas Carol," and the song "Cats in the Cradle."  These all suggest that successful men tend to emphasize work too much over family and friends, and will live to regret it.   "No one on their deathbed ever wished they had spent more time in the office," the saying goes.

Is this bias real?   I wish someone would actually survey people at their deathbeds about their regrets, but until then I have my doubts.   Successful men have surely considered this advice before rejecting it, and deathbed regrets could just reflect a conflict between the interests of younger and older selves.   It sounds too much like lobbying from friends and family, and its advocates seem a bit too smug and uncritical to be believed.   That all said though, I really don’t know.

What seems clearer is that if people can reasonably doubt whether widely described biases are real, it will be a lot harder to use social disapproval to discourage real biases. 
Tetlock is Professor of Leadership at UC Berkeley, with a background in psychology and political science.  He ran an experiment for many years from the 80s through the early 2000s in which he invited experts on foreign affairs and political science to make predictions about political and economic trends in countries around the world.  Tetlock tracked and evaluated those predictions, developing a large database of both accurate and inaccurate predictions.  He then analyzed it in various ways to find out how accurate the experts were overall, how they compared to other kinds of predictions, and what psychological and social factors influenced expert accuracy.  These results are what make up the book.

There’s meat here for people coming to the book with a variety of interests and needs, but I will focus on the pragmatic question of what factors influence accuracy, with the hope that learning more about these influences can help us to improve the accuracy of our own predictions and analyses.

One of the psychological measures or metrics which Tetlock found was well correlated with expert accuracy goes back to a distinction introduced by Isaiah Berlin in his book, The Hedgehog and the Fox. I haven’t read that book, but based on Tetlock’s presentation, Berlin distinguished between two cognitive styles to which he gave these colorful names.  The hedgehog is said to know one thing and know it well.  He sees events and trends in terms of his big idea, and aggressively extends it into new realms.  Hedgehogs tend to be confident in the applicability of their fundamental concepts and impatient with those who "do not get it".

Foxes in contrast know many small things which they bring to bear in their analyses in a dynamical and flexible way.  They tend to be uncertain and flexible, "on the other hand" types who are skeptical about their own predictive ability and in fact about the whole enterprise of making predictions in such an intractable realm.

To build suspense, I will not say, yet, which cognitive style was found to be more successful in Tetlock’s experiments.  Instead I will offer Tetlock’s test, from the appendix, along with his scoring methodology used to measure where people stand on the Fox-Hedgehog continuum.  You can take it yourself to decide whether you are Fox or Hedgehog.  In a day or two I will post some of Tetlock’s results as to how the two kinds of experts differed in the success of their predictions.

Here are Tetlock’s questions, which I have converted to a point value, rescaling his weightings which came from factor analysis.  If you agree, give yourself that many points, and if you disagree, give yourself the negative of that many points.  Note that some of the questions have a negative point value, so for those you subtract from your score if you agree, and add to your score (make it more positive) if you disagree.

(7 points) Isaiah Berlin classified intellectuals as hedgehogs or foxes.  The hedgehog knows one bit thing and tries to explain as much as possible within that conceptual framework, whereas the fox knows many small things and is content to improvise explanations on a case-by-case basis.  I place myself towards the hedgehog or fox end of this scale. [Note, I think Tetlock meant to give yourself the +7 points if you think you are a fox, and the -7 if you think you are a hedgehog.]

(-3 points) Scholars are usually at greater risk of exaggerating how complex the world is than they are of underestimating how complex it is.

(-5 points) We are closer than many think to achieving parsimonious explanations of politics

(4 points) I think politics is more cloudlike than clocklike ("cloudlike" meaning inherently unpredictable; "clocklike" meaning perfectly predictable if we have adequate knowledge).

(-5 points) The more common error in decision making is to abandon good ideas too quickly, not to stick with bad ideas too long.

(-2 points) Having clear rules and order at work is essential for success.

(5 points) Even after I have made up my mind about something, I am always eager to consider a different opinion.

(-6 points) I dislike questions that can be answered in many different ways.

(-4 points) I usually make important decisions quickly and confidently.

(5 points) When considering most conflict situations, I can usually see how both sides could be right.

(-3 points) It is annoying to listen to someone who cannot seem to make up his or her mind.

(4 points) I prefer interacting with people whose opinions are very different from my own.

(1 point) When trying to solve a problem I often see so many options that it is confusing.

As noted above, compute your score by adding the point values for the questions where you agree, and subtracting the points (keeping in mind that subtracting a negative means adding) for the questions where you disagree.  Possible scores range from -54 to 54, with negative scores indicating that you are a hedgehog while positive scores mean you are a fox.  The farther the score from zero, the more clearly you fall into the category and the more applicable results based on this cognitive analysis are likely to be for you.

Go ahead and take the test, see if you are a fox or a hedgehog, and I’ll post more shortly describing how these cognitive styles fared in Tetlock’s data.

The Movie “Click”


Robin Hanson delves into the function of bromides, those catchy little sayings that offer nuggets of wisdom like "Look before you leap!" and "He who hesitates is lost!" In this thought-provoking essay, Hanson explores how these bromides work to draw attention to common failure modes and biases, such as being too rash or too indecisive. By dissecting the function of these sayings, Hanson reveals insights into biases that individuals may possess, even if these biases average out on a population level. Prepare to challenge your thinking and gain a deeper understanding of how bromides can shape our behavior.

Apropos Robin’s recent remark that "’No one on their deathbed ever wished they had spent more time in the office,’ the saying goes." and his wondering whether we are really biased to spending too much time in the office.

This makes me wonder about the function of bromides. Consider:

"Look before you leap!"

"He who hesitates is lost!"

Both of these seem sensible, yet they may appear contradictory. Are we biased to be too rash, or too indecisive?

I think these little pearls of wisdom function differently. Compare:

"Beware of high blood pressure!"

"Beware of low blood pressure!"

Both of these express sensible advice, and it does not depend on there being an average tendency for blood pressure to be too high or too low.

Perhaps the function of a bromide is to draw attention to a common failure mode. Thus with blood pressure. Likewise with being too rash or too indecisive. Maybe lives frequently go wrong because we spend too much time in the office. This is consistent with lives also frequently going wrong because we don’t work hard enough.

There is no commonplace to the effect that eating the left wing of a chicken may cause foodpoisoning, and none for eating the right wing of a chicken. This is because it is usually not worth paying attention to which side of a chicken a wing comes from. But it is worth paying attention to whether your blood pressure is too high or too low;  whether you are too rash or too indecisive; and whetheryou might be working too hard or not hard enough.There is no commonplace to the effect that eating the left wing of a chicken may cause foodpoisoning, and none for eating the right wing of a chicken. This is because it is usually not worth paying attention to which side of a chicken a wing comes from. But it is worth paying attention to whether your blood pressure is too high or too low;  whether you are too rash or too indecisive; and whether you might be working too hard or not hard enough.

Relevance to the topic of bias: Perhaps most biases are specific to individuals. On average, a population may be unbiased, but individuals are biased in one direction or another. There is wisdom in being aware of common biases (and errors) that individuals may have, even if on a population level these cancel out. 

The Wisdom of Bromides

Robin Hanson delves into the fascinating and controversial topic of heritable beliefs in his latest thought-provoking essay. Drawing from a study on attitudes of twins, Hanson explores the genetic influences on our beliefs regarding life, intellect, equality, and punishment. He challenges readers to consider whether our beliefs on certain topics are actually rooted in truth or simply a product of our genes. Are there plausible explanations for why genetically-influenced beliefs may be more accurate? Or should we acknowledge the random nature of our beliefs and question their validity? Dive into this intriguing exploration of belief origins and biases.

Some of the differences in our beliefs seem to be heritable.   "The Heritability of Attitutes: A Study of Twins," in Journal of Personality and Social Psychology in 2001, asked 339 twin pairs for their attitudes on 30 topics.  These attitudes had seven common factors, four of which moderate categories of beliefs:   

Life: voluntary euthanasia, abortion on demand, birth control, and organized religion.

Intellect: books, chess, education, and capitalism

Equality: open-door immigration, distinct gender roles, racial discrimination, and getting along with others

Punishment: death penalty for murder,  and castration for sex crimes

Genetic differences explained most of differences in attitudes to life and equality (66% and 55% of the variance respectively), but none (0%) of the attitudes to intellect and punishment.   

Thus your differing attitudes on  abortion, birth control, immigrants, gender roles, and race are mostly due to your genes, while your attitudes toward education, capitalism and punishment are due to your life experiences.

Is there a plausible story whereby those with genes encouraging your sort of beliefs on life and equality tend to have more accurate beliefs?   A correlation between IQ and such attitudes might be one such story, but I know of no data supporting this.

If you can’t find such a story, you should admit that the process that produced these beliefs of yours was random and uncorrelated with the truth on those subjects.   You should thus reject those beliefs as biased.   For more on the subject of rejecting arbitrary belief origins see my Theory and Decision paper "Uncommon Priors Require Origin Disputes.

Beware Heritable Beliefs

Robin Hanson proposes that rationality is the martial art of the mind, a skill that can be learned and honed just like any physical discipline. In this thought-provoking essay, Hanson delves into the complexities of rational thought and the challenges of training our brains to think critically and avoid common cognitive biases. Drawing on insights from psychology and probability theory, he explores the potential for synthesizing a shared vocabulary and systematizing techniques of personal rationality. Discover how the pursuit of rationality can help us navigate the complexities of our own minds and make more informed decisions.

I often use the metaphor that rationality is the martial art of mind. You don’t need huge, bulging muscles to learn martial arts—there’s a tendency toward more athletic people being more likely to learn martial arts, but that may be a matter of enjoyment as much as anything else. If you have a hand, with tendons and muscles in the appropriate places, then you can learn to make a fist.

Similarly, if you have a brain, with cortical and subcortical areas in the appropriate places, you might be able to learn to use it properly. If you’re a fast learner, you might learn faster—but the art of rationality isn’t about that; it’s about training brain machinery we all have in common. And where there are systematic errors human brains tend to make—like an insensitivity to scope—rationality is about fixing those mistakes, or finding work-arounds.

Alas, our minds respond less readily to our will than our hands. Our ability to control our muscles is evolutionarily ancient; our ability to reason about our own reasoning processes is a much more recent innovation. We shouldn’t be surprised, then, that muscles are easier to use than brains. But it is not wise to neglect the latter training because it is more difficult. It is not by bigger muscles that the human species rose to prominence upon Earth.

If you live in an urban area, you probably don’t need to walk very far to find a martial arts dojo. Why aren’t there dojos that teach rationality?

One reason, perhaps, is that it’s harder to verify skill. To rise a level in Tae Kwon Do, you might need to break a board of a certain width. If you succeed, all the onlookers can see and applaud. If you fail, your teacher can watch how you shape a fist, and check if you shape it correctly. If not, the teacher holds out a hand and makes a fist correctly, so that you can observe how to do so.

Within martial arts schools, techniques of muscle have been refined and elaborated over generations. Techniques of rationality are harder to pass on, even to the most willing student.

Very recently—in just the last few decades—the human species has acquired a great deal of new knowledge about human rationality. The most salient example would be the heuristics and biases program in experimental psychology. There is also the Bayesian systematization of probability theory and statistics; evolutionary psychology; social psychology. Experimental investigations of empirical human psychology; and theoretical probability theory to interpret what our experiments tell us; and evolutionary theory to explain the conclusions. These fields give us new focusing lenses through which to view the landscape of our own minds. With their aid, we may be able to see more clearly the muscles of our brains, the fingers of thought as they move. We have a shared vocabulary in which to describe problems and solutions. Humanity may finally be ready to synthesize the martial art of mind: to refine, share, systematize, and pass on techniques of personal rationality.

Such understanding as I have of rationality, I acquired in the course of wrestling with the challenge of artificial general intelligence (an endeavor which, to actually succeed, would require sufficient mastery of rationality to build a complete working rationalist out of toothpicks and rubber bands). In most ways the AI problem is enormously more demanding than the personal art of rationality, but in some ways it is actually easier. In the martial art of mind, we need to acquire the realtime procedural skill of pulling the right levers at the right time on a large, pre-existing thinking machine whose innards are not end-user-modifiable. Some of the machinery is optimized for evolutionary selection pressures that run directly counter to our declared goals in using it. Deliberately we decide that we want to seek only the truth; but our brains have hardwired support for rationalizing falsehoods. We can try to compensate for what we choose to regard as flaws of the machinery; but we can’t actually rewire the neural circuitry. Nor may martial artists plate titanium over their bones—not today, at any rate.

Trying to synthesize a personal art of rationality, using the science of rationality, may prove awkward: One imagines trying to invent a martial art using an abstract theory of physics, game theory, and human anatomy.

But humans arent reflectively blind. We do have a native instinct for introspection. The inner eye isnt sightless, though it sees blurrily, with systematic distortions. We need, then, to apply the science to our intuitions, to use the abstract knowledge to correct our mental movements and augment our metacognitive skills.

We aren't writing a computer program to make a string puppet execute martial arts forms; it is our own mental limbs that we must move. Therefore we must connect theory to practice. We must come to see what the science means, for ourselves, for our daily inner life.

The Martial Art of Rationality

Robin Hanson takes us back to the beginning of speculative markets created by non-traders in this fascinating essay. Exploring a 1990 prediction market created at Xanadu, Inc., Hanson shares how these markets were used to gain information by believing in market prices. Discover the story of the first ever market where employees and consultants bet on scientific claims and later shifted to betting on a product delivery before a political event. With insights into the dynamics of these markets and their potential to reduce bias, this essay sheds light on the origins of a revolutionary approach to information gathering.

Betting markets have been around for a long time, and as far as I know, until recently they were all created to help traders achieve goals such as hedging, gambling, proving themselves, and so on.  What appears to be new is that non-traders are now creating and subsidizing some markets, in order to gain information by believing the market prices.  Such price estimates are remarkably robust against biases, making this a promising approach to reducing bias.

The earliest such market that I know of was one I helped create at Xanadu, Inc. in 1990. 

Xanadu, then owned by Autodesk, was working on what would have been a different kind of world wide web.   I consulted for them, and had been telling them about my idea futures concept.   In April 1989 the "cold fusion" controversy erupted, and I created a market for Xanadu employees and consultants to bet on this claim:

By 1/1/91 a <1 liter device will have generated over 1 watt of power output more than input from room-T fusion, including amortized power to create/separate components.

This market used a paper-based mechanical market-maker (which loses money on average) placed on a wall of the main common room.  18 people participated, and everyones current stakes were posted for all to see.   The price slowly fell, reaching 7% in May 1990.   At this time the fusion market was replaced with a market on this claim: 

Xanadu will deliver its product before Premier Deng of China dies. 

(They hoped that their product could help China through a post Deng-transition to democracy.)  This market started at a price around 70%, and continued until Xanadu lost its Autodesk support in 1992 (the same year Tim Berners-Lee introduced his version of the web).   Deng died before the product was delivered. 

Support from the Xanadu manager, Marc Stiegler, was crucial.  His goal was to focus employee attention on the risk that Xanadu might not deliver its product soon.  Does anyone know of an earlier example of a speculative market which existed primarily because non-traders wanted to use the information in its prices?

Thanks to Roger Gregory for archiving his email. 

Addendum: The Deng market was Marc Stiegler’s idea; he also authored a 1999 science fiction novel, EarthWeb, wherein prediction markets help humans to fend off aliens. 

A 1990 Corporate Prediction Market


Robin Hanson delves into the fascinating topic of why academics, especially those in humanities and social sciences, tend to be more liberal and Democratic compared to the general public. Exploring possible explanations such as intelligence, social pressure, and discrimination, Hanson raises thought-provoking questions about the impact this disparity may have on the beliefs expressed in academic articles. As an economist in a department with a different ratio, Hanson brings a unique perspective to the discussion. Join the exploration of this intriguing issue and discover potential ways to distinguish between theories.

In the US at least, academics are more liberal and Democratic than ordinary people.   While among ordinary people the ratio of Democrats to Republicans is about 1:1, academia as a whole has a ratio of 5:1, and the humanities and social sciences have a ratio of 8:1.   These ratios have roughly doubled over the last forty years.  See this 2005 Critical Review paper by my colleague Dan Klein, but also this 2006 Public Opinion Quarterly reply, and this further response.   

Does this difference between academia and the public produce or reflect a bias in the beliefs expressed in academic articles?  

To judge this, we need to better understand the cause of this difference.  It might be due to academics being more intelligent and informed, or it might be due to social pressure and conformity.   This 2005 BE Press Forum paper suggests that discrimination against conservatives, women, and religious folks is at least part of the explanation, though see this reply and this response.

This issue is of particular interest to me, since economists have a 3:1 ratio, making us "right wing" relative to the rest of academia.   My department has about a 1:1 ratio, making us "right-wing nuts," even though we roughly mirror the public. 

It is interesting to note that when you look at higher dimensional opinion spaces what you see is a tight liberal clump in one corner, and a broad spread of opinions across the rest of the opinion space; there is no conservative clump to speak of.   Of course this is consistent with both the information and the conformity theories mentioned above.    

What we need are clever ideas for data that might better distinguish between these theories.

Why Are Academics Liberal?


Robin Hanson challenges the notion of moral superiority in his insightful essay. He draws attention to staggering statistics that reveal high levels of cheating, lying, and stealing among high school students, while simultaneously expressing satisfaction with their ethics. Hanson argues that moral overconfidence is a more plausible explanation for these results. He also explores a thought-provoking set of experiments that highlight how people tend to act less fairly when given even minor justifications for their unfair behavior. Prepare to question your own moral standing as you dive into this eye-opening piece.

A Washington Post article from from last Saturday says:

In the 2006 survey of more than 36,000 high school students, 60 percent said they cheated on a test, 82 percent said they lied to their parents about something significant and 28 percent said they stole something from a store. … 92 percent said they were "satisfied with my own ethics and character." About 74 percent said that "when it comes to doing right, I am better than most people I know." … the percentageof students who lie, cheat or steal could be higher than the survey found.   When asked, 27 percent of the students admitted that they lied on at least one survey question.In the 2006 survey of more than 36,000 high school students, 60 percent said they cheated on a test, 82 percent said they lied to their parents about something significant and 28 percent said they stole something from a store. … 92 percent said they were "satisfied with my own ethics and character." About 74 percent said that "when it comes to doing right, I am better than most people I know." … the percentage of students who lie, cheat or steal could be higher than the survey found.   When asked, 27 percent of the students admitted that they lied on at least one survey question.

It is hard to see how 3/4 of people could be better than most people they know about anything; moral overconfidence is a far more plausible explanation.

On a similar topic, a clever set of experiments, described in "Exploiting Moral Wiggle Room" (and discussed at Marginal Revolution) shows that people act to appear fair, but are less fair when given even feeble excuses for their unfair behavior.

Are you willing to admit that you are about as moral as most people you know?  I am. 

Moral Overconfidence


Overcoming Bias

The Modesty Argument states that when two or more human beings have common knowledge that they disagree about a question of simple fact, they should each adjust their probability estimates in the direction of the others'.  (For example, they might adopt the common mean of their probability distributions.  If we use the logarithmic scoring rule, then the score of the average of a set of probability distributions is better than the average of the scores of the individual distributions, by Jensen's inequality.)

Put more simply:  When you disagree with someone, even after talking over your reasons, the Modesty Argument claims that you should each adjust your probability estimates toward the other's, and keep doing this until you agree.  The Modesty Argument is inspired by Aumann's Agreement Theorem, a very famous and oft-generalized result which shows that genuine Bayesians literally cannot agree to disagree; if genuine Bayesians have common knowledge of their individual probability estimates, they must all have the same probability estimate.  ("Common knowledge" means that I know you disagree, you know I know you disagree, etc.)

I've always been suspicious of the Modesty Argument.  It's been a long-running debate between myself and Robin Hanson.


Robin seems to endorse the Modesty Argument in papers such as Are Disagreements Honest?  I, on the other hand, have held that it can be rational for an individual to not adjust their own probability estimate in the direction of someone else who disagrees with them.

How can I maintain this position in the face of Aumann's Agreement Theorem, which proves that genuine Bayesians cannot have common knowledge of a dispute about probability estimates?  If genunie Bayesians will always agree with each other once they've exchanged probability estimates, shouldn't we Bayesian wannabes do the same?

To explain my reply, I begin with a metaphor:  If I have five different accurate maps of a city, they will all be consistent with each other.  Some philosophers, inspired by this, have held that "rationality" consists of having beliefs that are consistent among themselves.  But, although accuracy necessarily implies consistency, consistency does not necessarily imply accuracy.  If I sit in my living room with the curtains drawn, and make up five maps that are consistent with each other, but I don't actually walk around the city and make lines on paper that correspond to what I see, then my maps will be consistent but not accurate.  When genuine Bayesians agree in their probability estimates, it's not because they're trying to be consistent - Aumann's Agreement Theorem doesn't invoke any explicit drive on the Bayesians' part to be consistent.  That's what makes AAT surprising!  Bayesians only try to be accurate; in the course of seeking to be accurate, they end up consistent.  The Modesty Argument, that we can end up accurate in the course of seeking to be consistent, does not necessarily follow.

How can I maintain my position in the face of my admission that disputants will always improve their average score if they average together their individual probability distributions?

Suppose a creationist comes to me and offers:  "You believe that natural selection is true, and I believe that it is false.  Let us both agree to assign 50% probability to the proposition."  And suppose that by drugs or hypnosis it was actually possible for both of us to contract to adjust our probability estimates in this way.  This unquestionably improves our combined log-score, and our combined squared error.  If as a matter of altruism, I value the creationist's accuracy as much as my own - if my loss function is symmetrical around the two of us - then I should agree.  But what if I'm trying to maximize only my own individual accuracy?  In the former case, the question is absolutely clear, and in the latter case it is not absolutely clear, to me at least, which opens up the possibility that they are different questions.

If I agree to a contract with the creationist in which we both use drugs or hypnosis to adjust our probability estimates, because I know that the group estimate must be improved thereby, I regard that as pursuing the goal of social altruism.  It doesn't make creationism actually true, and it doesn't mean that I think creationism is true when I agree to the contract.  If I thought creationism was 50% probable, I wouldn't need to sign a contract - I would have already updated my beliefs!  It is tempting but false to regard adopting someone else's beliefs as a favor to them, and rationality as a matter of fairness, of equal compromise.  Therefore it is written:  "Do not believe you do others a favor if you accept their arguments; the favor is to you."  Am I really doing myself a favor by agreeing with the creationist to take the average of our probability distributions?

I regard rationality in its purest form as an individual thing - not because rationalists have only selfish interests, but because of the form of the only admissible question:  "Is is actually true?"  Other considerations, such as the collective accuracy of a group that includes yourself, may be legitimate goals, and an important part of human existence - but they differ from that single pure question.

In Aumann's Agreement Theorem, all the individual Bayesians are trying to be accurate as individuals.  If their explicit goal was to maximize group accuracy, AAT would not be surprising.  So the improvement of group score is not a knockdown argument as to what an individual should do if they are trying purely to maximize their own accuracy, and it is that last quest which I identify as rationality.  It is written:  "Every step of your reasoning must cut through to the correct answer in the same movement.  More than anything, you must think of carrying your map through to reflecting the territory.  If you fail to achieve a correct answer, it is futile to protest that you acted with propriety."  From the standpoint of social altruism, someone may wish to be Modest, and enter a drug-or-hypnosis-enforced contract of Modesty, even if they fail to achieve a correct answer thereby.

The central argument for Modesty proposes something like a Rawlsian veil of ignorance - how can you know which of you is the honest truthseeker, and which the stubborn self-deceiver?  The creationist believes that he is the sane one and you are the fool.  Doesn't this make the situation symmetric around the two of you?  If you average your estimates together, one of you must gain, and one of you must lose, since the shifts are in opposite directions; but by Jensen's inequality it is a positive-sum game.  And since, by something like a Rawlsian veil of ignorance, you don't know which of you is really the fool, you ought to take the gamble.  This argues that the socially altruistic move is also always the individually rational move.

And there's also the obvious reply:  "But I know perfectly well who the fool is.  It's the other guy.  It doesn't matter that he says the same thing - he's still the fool."

This reply sounds bald and unconvincing when you consider it abstractly.  But if you actually face a creationist, then it certainly feels like the correct answer - you're right, he's wrong, and you have valid evidence to know that, even if the creationist can recite exactly the same claim in front of a TV audience.

Robin Hanson sides with symmetry - this is clearest in his paper Uncommon Priors Require Origin Disputes - and therefore endorses the Modesty Argument.  (Though I haven't seen him analyze the particular case of the creationist.)

I respond:  Those who dream do not know they dream; but when you wake you know you are awake.  Dreaming, you may think you are awake.  You may even be convinced of it.  But right now, when you really are awake, there isn't any doubt in your mind - nor should there be.  If you, persuaded by the clever argument, decided to start doubting right now that you're really awake, then your Bayesian score would go down and you'd become that much less accurate.  If you seriously tried to make yourself doubt that you were awake - in the sense of wondering if you might be in the midst of an ordinary human REM cycle - then you would probably do so because you wished to appear to yourself as rational, or because it was how you conceived of "rationality" as a matter of moral duty.  Because you wanted to act with propriety.  Not because you felt genuinely curious as to whether you were awake or asleep.  Not because you felt you might really and truly be asleep.  But because you didn't have an answer to the clever argument, just an (ahem) incommunicable insight that you were awake.

Russell Wallace put it thusly:  "That we can postulate a mind of sufficiently low (dreaming) or distorted (insane) consciousness as to genuinely not know whether it's Russell or Napoleon doesn't mean I (the entity currently thinking these thoughts) could have been Napoleon, any more than the number 3 could have been the number 7. If you doubt this, consider the extreme case: a rock doesn't know whether it's me or a rock. That doesn't mean I could have been a rock."

There are other problems I see with the Modesty Argument, pragmatic matters of human rationality - if a fallible human tries to follow the Modesty Argument in practice, does this improve or disimprove personal rationality?  To me it seems that the adherents of the Modesty Argument tend to profess Modesty but not actually practice it.

For example, let's say you're a scientist with a controversial belief - like the Modesty Argument itself, which is hardly a matter of common accord - and you spend some substantial amount of time and effort trying to prove, argue, examine, and generally forward this belief.  Then one day you encounter the Modesty Argument, and it occurs to you that you should adjust your belief toward the modal belief of the scientific field.  But then you'd have to give up your cherished hypothesis.  So you do the obvious thing - I've seen at least two people do this on two different occasions - and say:  "Pursuing my personal hypothesis has a net expected utility to Science.  Even if I don't really believe that my theory is correct, I can still pursue it because of the categorical imperative: Science as a whole will be better off if scientists go on pursuing their own hypotheses."  And then they continue exactly as before.

I am skeptical to say the least.  Integrating the Modesty Argument as new evidence ought to produce a large effect on someone's life and plans.  If it's being really integrated, that is, rather than flushed down a black hole.  Your personal anticipation of success, the bright emotion with which you anticipate the confirmation of your theory, should diminish by literally orders of magnitude after accepting the Modesty Argument.  The reason people buy lottery tickets is that the bright anticipation of winning ten million dollars, the dancing visions of speedboats and mansions, is not sufficiently diminished - as a strength of emotion - by the probability factor, the odds of a hundred million to one.  The ticket buyer may even profess that the odds are a hundred million to one, but they don't anticipate it properly - they haven't integrated the mere verbal phrase "hundred million to one" on an emotional level.

So, when a scientist integrates the Modesty Argument as new evidence, should the resulting nearly total loss of hope have no effect on real-world plans originally formed in blessed ignorance and joyous anticipation of triumph?  Especially when you consider that the scientist knew about the social utility to start with, while making the original plans?  I think that's around as plausible as maintaining your exact original investment profile after the expected returns on some stocks change by a factor of a hundred.  What's actually happening, one naturally suspects, is that the scientist finds that the Modesty Argument has uncomfortable implications; so they reach for an excuse, and invent on-the-fly the argument from social utility as a way of exactly cancelling out the Modesty Argument and preserving all their original plans.

But of course if I say that this is an argument against the Modesty Argument, that is pure ad hominem tu quoque.  If its adherents fail to use the Modesty Argument properly, that does not imply it has any less force as logic.

Rather than go into more detail on the manifold ramifications of the Modesty Argument, I'm going to close with the thought experiment that initially convinced me of the falsity of the Modesty Argument.  In the beginning it seemed to me reasonable that if feelings of 99% certainty were associated with a 70% frequency of true statements, on average across the global population, then the state of 99% certainty was like a "pointer" to 70% probability.  But at one point I thought:  "What should an (AI) superintelligence say in the same situation?  Should it treat its 99% probability estimates as 70% probability estimates because so many human beings make the same mistake?"  In particular, it occurred to me that, on the day the first true superintelligence was born, it would be undeniably true that - across the whole of Earth's history - the enormously vast majority of entities who had believed themselves superintelligent would be wrong.  The majority of the referents of the pointer "I am a superintelligence" would be schizophrenics who believed they were God.

A superintelligence doesn't just believe the bald statement that it is a superintelligence - it presumably possesses a very detailed, very accurate self-model of its own cognitive systems, tracks in detail its own calibration, and so on.  But if you tell this to a mental patient, the mental patient can immediately respond:  "Ah, but I too possess a very detailed, very accurate self-model!"  The mental patient may even come to sincerely believe this, in the moment of the reply.  Does that mean the superintelligence should wonder if it is a mental patient?  This is the opposite extreme of Russell Wallace asking if a rock could have been you, since it doesn't know if it's you or the rock.

One obvious reply is that human beings and superintelligences occupy different classes - we do not have the same ur-priors, or we are not part of the same anthropic reference class; some sharp distinction renders it impossible to group together superintelligences and schizophrenics in probability arguments.  But one would then like to know exactly what this "sharp distinction" is, and how it is justified relative to the Modesty Argument.  Can an evolutionist and a creationist also occupy different reference classes?  It sounds astoundingly arrogant; but when I consider the actual, pragmatic situation, it seems to me that this is genuinely the case.

Or here's a more recent example - one that inspired me to write today's blog post, in fact.  It's the true story of a customer struggling through five levels of Verizon customer support, all the way up to floor manager, in an ultimately futile quest to find someone who could understand the difference between .002 dollars per kilobyte and .002 cents per kilobyte.  Audio [27 minutes], Transcript.  It has to be heard to be believed.  Sample of conversation:  "Do you recognize that there's a difference between point zero zero two dollars and point zero zero two cents?"  "No."

The key phrase that caught my attention and inspired me to write today's blog post is from the floor manager:  "You already talked to a few different people here, and they've all explained to you that you're being billed .002 cents, and if you take it and put it on your calculator... we take the .002 as everybody has told you that you've called in and spoken to, and as our system bills accordingly, is correct."

Should George - the customer - have started doubting his arithmetic, because five levels of Verizon customer support, some of whom cited multiple years of experience, told him he was wrong?  Should he have adjusted his probability estimate in their direction?  A straightforward extension of Aumann's Agreement Theorem to impossible possible worlds, that is, uncertainty about the results of computations, proves that, had all parties been genuine Bayesians with common knowledge of each other's estimates, they would have had the same estimate.  Jensen's inequality proves even more straightforwardly that, if George and the five levels of tech support had averaged together their probability estimates, they would have improved their average log score.  If such arguments fail in this case, why do they succeed in other cases?  And if you claim the Modesty Argument carries in this case, are you really telling me that if George had wanted only to find the truth for himself, he would have been wise to adjust his estimate in Verizon's direction?  I know this is an argument from personal incredulity, but I think it's a good one.

On the whole, and in practice, it seems to me like Modesty is sometimes a good idea, and sometimes not.  I exercise my individual discretion and judgment to decide, even knowing that I might be biased or self-favoring in doing so, because the alternative of being Modest in every case seems to me much worse.

But the question also seems to have a definite anthropic flavor.  Anthropic probabilities still confuse me; I've read arguments but I have been unable to resolve them to my own satisfaction.  Therefore, I confess, I am not able to give a full account of how the Modesty Argument is resolved.

Modest, aren't I?

The Modesty Argument

An edited transcript of a long instant-messenger conversation that took place regarding the phrase, "I don't know", sparked by Robin Hanson's previous post, "You Are Never Entitled to Your Opinion."


[08:50] Eliezer: http://www.overcomingbias.com/2006/12/you_are_never_e.html

[09:01] X: it still seems that saying "i don't know" in some situations is better than giving your best guess

[09:01] X: especially if you are dealing with people who will take you at your word who are not rationalists

[09:02] Eliezer: in real life, you have to choose, and bet, at some betting odds

[09:02] Eliezer: i.e., people who want to say "I don't know" for cryonics still have to sign up or not sign up, and they'll probably do the latter

[09:03] Eliezer: "I don't know" is usually just a screen that people think is defensible and unarguable before they go on to do whatever they feel like, and it's usually the wrong thing because they refused to admit to themselves what their guess was, or examine their justifications, or even realize that they're guessing

[09:02] X: how many apples are in a tree outside?

[09:02] X: i've never seen it and neither have you

[09:02] Eliezer: 10 to 1000

[09:04] Eliezer: if you offer to bet me a million dollars against one dollar that the tree outside has fewer than 20 apples, when neither of us have seen it, I will take your bet

[09:04] X: is it better to say "maybe 10 to 1000" to make it clear that you are guessing when talking to people

[09:04] Eliezer: therefore I have assigned a nonzero and significant probability to apples < 20 whether I admit it or not

[09:05] Eliezer: what you *say* is another issue, especially when speaking to nonrationalists, and then it is well to bear in mind that words don't have fixed meanings; the meaning of the sounds that issue from your lips is whatever occurs in the mind of the listener. If they're going to misinterpret something then you shouldn't say it to them no matter what the words mean inside your own head

[09:06] Eliezer: often you are just screwed unless you want to go back and teach them rationality from scratch, and in a case like that, all you can do is say whatever creates the least inaccurate image

[09:06] X: 10 to 1000 is misleading when you say it to a nonrationalist?

[09:06] Eliezer: "I don't know" is a good way to duck when you say it to someone who doesn't know about probability distributions

[09:07] Eliezer: if they thought I was certain, or that my statement implied actual knowledge of the tree

[09:07] Eliezer: then the statement would mislead them

[09:07] Eliezer: and if I knew this, and did it anyway for my own purposes, it would be a lie

[09:08] Eliezer: if I just couldn't think of anything better to say, then it would be honest but not true, if you can see the distinction

[09:08] Eliezer: honest for me, but the statement that formed in their minds would still not be true

[09:09] X: most people will say to you.... but you said....10-1000 apples

[09:09] Eliezer: then you're just screwed

[09:10] Eliezer: nothing you can do will create in their minds a true understanding, not even "I don't know"

[09:10] X: why bother, why not say i don't know?

[09:10] Eliezer: honesty therefore consists of misleading them the least and telling them the most

[09:10] X: it's better than misleading them with 10-1000

[09:10] Eliezer: as for "why bother", well, if you're going to ask that question, just don't reply to their email or whatever

[09:11] Eliezer: what if you're dealing with someone who thinks my saying "I don't know" is a license for them to make up their own ideas, which will be a lot worse?

[09:11] X: they may act on your guess, and then say "but you said...." and lose money or get in trouble or have less respect for you

[09:11] Eliezer: then you choose to wave them off

[09:11] Eliezer: with "I don't know"

[09:11] Eliezer: but it's for your own sake, not for their sake

[09:12] X: [09:11] Eliezer: what if you're dealing with someone who thinks my saying "I don't know" is a license for them to make up their own ideas, which will be a lot worse?

[09:12] X: here i could see why

[09:12] X: but it's difficult working with typical people in the real world

[09:13] Eliezer: the first thing to decide is, are you trying to accomplish something for yourself (like not getting in trouble) or are you trying to improve someone else's picture of reality

[09:13] Eliezer: "I don't know" is often a good way of not getting in trouble

[09:13] Eliezer: as for it being difficult to work with people in the real world, well, yeah

[09:13] X: if you say...10-1000, and you are wrong, and they are mad, then you say, i don't know, they will be even madder

[09:13] Eliezer: are you trying not to get in trouble?

[09:14] Eliezer: or are you trying to improve their picture of reality?

[09:14] Eliezer: these are two different tasks

[09:14] X: especially if they have lost money or have been proven wrong by someone else

[09:14] Eliezer: if they intersect you have to decide what your tradeoff is

[09:14] Eliezer: which is more important to you

[09:14] Eliezer: then decide whether to explain for their benefit or say "I don't know" for yours

[09:15] X: well, if it was my job, i would say i don't know rather than be wrong, because who knows what your boss will do after he loses money listening to you

[09:15] Eliezer: okay

[09:16] Eliezer: just be clear that this is not because "I don't know" is the rational judgment, but because "I don't know" is the political utterance

[09:16] X: he may take your guess, and try to turn it into an actual anwser because no one around you has a better plan

[09:17] Eliezer: you can easily see this by looking at your stated reason: you didn't talk about evidence and reality and truth, but, how you might get in trouble based on someone's reaction

[09:17] X: yes

[09:17] X: that's what you have to put up with in the real world

[09:17] Eliezer: if you're really worried about your boss's welfare then you should consider that if you say "I don't know" he must do something anyway - refusing to choose is also a choice, and refusing to act is like refusing to let time pass - and he will construct that plan based on some information, which doesn't include your information

[09:18] Eliezer: if your life isn't worth more than someone else's, neither it is worth any less, and it is often proper to let fools make their own mistakes

[09:18] Eliezer: you can only throw yourself in front of so many bullets before you run out of flesh to stop them with

[09:19] X: ?

[09:19] Eliezer: in other words, you cannot always save people from themselves

[09:23] Eliezer: but all of this is wandering away from the original point, which is true and correct, that no one is ever entitled to their own opinion

[09:26] X: what is his name?

[09:26] Eliezer: ?

[09:26] X: a man outside

[09:26] X: random guy

[09:26] Eliezer: It's probably not "Xpchtl Vaaaaaarax"

[09:26] X: probably not

[09:27] Eliezer: I suppose I could construct a second-order Markov transition diagram for the letters in names expressed in English, weighted by their frequency

[09:27] Eliezer: but that would be a lot of work

[09:28] Eliezer: so I could say "I don't know" as shorthand for the fact that, although I possess a lot of knowledge about possible and probable names, I don't know anything *more* than you do

[09:28] X: ok, so you say ruling out what you see as likely not correct is ok?

[09:28] Eliezer: what I'm saying is that I possess a large amount of knowledge about possible names

[09:28] Eliezer: all of which influences what I would bet on

[09:28] Eliezer: if I had to take a real-world action, like, guessing someone's name with a gun to my head

[09:29] Eliezer: if I had to choose it would suddenly become very relevant that I knew Michael was one of the most statistically common names, but couldn't remember for which years it was the most common, and that I knew Michael was more likely to be a male name than a female name

[09:29] Eliezer: if an alien had a gun to its head, telling it "I don't know" at this point would not be helpful

[09:29] Eliezer: because there's a whole lot I know that it doesn't

[09:30] X: ok

[09:33] X: what about a question for which you really don't have any information?

[09:33] X: like something only an alien would know

[09:34] Eliezer: if I have no evidence I use an appropriate Ignorance Prior, which distributes probability evenly across all possibilities, and assigns only a very small amount to any individual possibility because there are so many

[09:35] Eliezer: if the person I'm talking to already knows to use an ignorance prior, I say "I don't know" because we already have the same probability distribution and I have nothing to add to that

[09:35] Eliezer: the ignorance prior tells me my betting odds

[09:35] Eliezer: it governs my choices

[09:35] X: and what if you don't know how to use an ignorance prior

[09:36] X: have never heard of it etc

[09:36] Eliezer: if I'm dealing with someone who doesn't know about ignorance priors, and who is dealing with the problem by making up this huge elaborate hypothesis with lots of moving parts and many places to go wrong, then the truth is that I automatically know s/he's wrong

[09:36] Eliezer: it may not be possible to explain this to them, short of training them from scratch in rationality

[09:36] Eliezer: but it is true

[09:36] Eliezer: and if the person trusts me for a rationalist, it may be both honest and helpful to tell them, "No, that's wrong"

[09:36] X: what if that person says, "i don't know what their name is", that ok?

[09:37] Eliezer: in real life you cannot choose "I don't know", it's not an option on your list of available actions

[09:37] Eliezer: in real life it's always, "I don't know, so I'm going to say Vkktor Blackdawn because I think it sounds cool"

[09:39] Eliezer: Vkktor Blackdawn is as (im)probable as anything else, but if you start assigning more probability to it than the ignorance prior calls for - because it sounds cool, because you don't have room in your mind for more than one possibility, or because you've started to construct an elaborate mental explanation of how the alien might end up named Vkktor Blackdawn

[09:39] Eliezer: then I know better

[09:40] Eliezer: and if you trust me, I may be able to honestly and usefully tell you so

[09:40] Eliezer: rather than saying "I don't know", which is always something to say, not to think

[09:40] Eliezer: this is important if someone asks you, "At what odds would you bet that the alien is named Vkktor Blackdawn?"

[09:41] Eliezer: or if you have to do anything else, based on your guesses and the weight you assign to them

[09:41] Eliezer: which is what probability is all about

[09:41] X: and if they say "I don't know, I don't know anything about probability"?

[09:41] Eliezer: then either they trust me blindly or I can't help them

[09:41] Eliezer: that's how it goes

[09:41] Eliezer: you can't always save people from themselves

[09:42] X: trust you blindly about what you are saying or about your guess as to what the alien's name is?

[09:42] Eliezer: trust me blindly when I tell them, "Don't bet at those odds."

"I don't know."

In the time of the Roman Empire, civic life was divided between the Blue and Green factions. The Blues and the Greens murdered each other in single combats, in ambushes, in group battles, in riots. Procopius said of the warring factions: “So there grows up in them against their fellow men a hostility which has no cause, and at no time does it cease or disappear, for it gives place neither to the ties of marriage nor of relationship nor of friendship, and the case is the same even though those who differ with respect to these colors be brothers or any other kin.”1 Edward Gibbon wrote: “The support of a faction became necessary to every candidate for civil or ecclesiastical honors.”2

Who were the Blues and the Greens? They were sports fans—the partisans of the blue and green chariot-racing teams.

Imagine a future society that flees into a vast underground network of caverns and seals the entrances. We shall not specify whether they flee disease, war, or radiation; we shall suppose the first Undergrounders manage to grow food, find water, recycle air, make light, and survive, and that their descendants thrive and eventually form cities. Of the world above, there are only legends written on scraps of paper; and one of these scraps of paper describes the sky, a vast open space of air above a great unbounded floor. The sky is cerulean in color, and contains strange floating objects like enormous tufts of white cotton. But the meaning of the word “cerulean” is controversial; some say that it refers to the color known as “blue,” and others that it refers to the color known as “green.”

In the early days of the underground society, the Blues and Greens contested with open violence; but today, truce prevails—a peace born of a growing sense of pointlessness. Cultural mores have changed; there is a large and prosperous middle class that has grown up with effective law enforcement and become unaccustomed to violence. The schools provide some sense of historical perspective; how long the battle between Blues and Greens continued, how many died, how little changed as a result. Minds have been laid open to the strange new philosophy that people are people, whether they be Blue or Green.

The conflict has not vanished. Society is still divided along Blue and Green lines, and there is a “Blue” and a “Green” position on almost every contemporary issue of political or cultural importance. The Blues advocate taxes on individual incomes, the Greens advocate taxes on merchant sales; the Blues advocate stricter marriage laws, while the Greens wish to make it easier to obtain divorces; the Blues take their support from the heart of city areas, while the more distant farmers and watersellers tend to be Green; the Blues believe that the Earth is a huge spherical rock at the center of the universe, the Greens that it is a huge flat rock circling some other object called a Sun. Not every Blue or every Green citizen takes the “Blue” or “Green” position on every issue, but it would be rare to find a city merchant who believed the sky was blue, and yet advocated an individual tax and freer marriage laws.

The Underground is still polarized; an uneasy peace. A few folk genuinely think that Blues and Greens should be friends, and it is now common for a Green to patronize a Blue shop, or for a Blue to visit a Green tavern. Yet from a truce originally born of exhaustion, there is a quietly growing spirit of tolerance, even friendship.

One day, the Underground is shaken by a minor earthquake. A sightseeing party of six is caught in the tremblor while looking at the ruins of ancient dwellings in the upper caverns. They feel the brief movement of the rock under their feet, and one of the tourists trips and scrapes her knee. The party decides to turn back, fearing further earthquakes. On their way back, one person catches a whiff of something strange in the air, a scent coming from a long-unused passageway. Ignoring the well-meant cautions of fellow travellers, the person borrows a powered lantern and walks into the passageway. The stone corridor wends upward . . . and upward . . . and finally terminates in a hole carved out of the world, a place where all stone ends. Distance, endless distance, stretches away into forever; a gathering space to hold a thousand cities. Unimaginably far above, too bright to look at directly, a searing spark casts light over all visible space, the naked filament of some huge light bulb. In the air, hanging unsupported, are great incomprehensible tufts of white cotton. And the vast glowing ceiling above . . . the color . . . is . . .

Now history branches, depending on which member of the sightseeing party decided to follow the corridor to the surface.

Aditya the Blue stood under the blue forever, and slowly smiled. It was not a pleasant smile. There was hatred, and wounded pride; it recalled every argument she’d ever had with a Green, every rivalry, every contested promotion. “You were right all along,” the sky whispered down at her, “and now you can prove it.” For a moment Aditya stood there, absorbing the message, glorying in it, and then she turned back to the stone corridor to tell the world. As Aditya walked, she curled her hand into a clenched fist. “The truce,” she said, “is over.”

 

Barron the Green stared uncomprehendingly at the chaos of colors for long seconds. Understanding, when it came, drove a pile-driver punch into the pit of his stomach. Tears started from his eyes. Barron thought of the Massacre of Cathay, where a Blue army had massacred every citizen of a Green town, including children; he thought of the ancient Blue general, Annas Rell, who had declared Greens “a pit of disease; a pestilence to be cleansed”; he thought of the glints of hatred he’d seen in Blue eyes and something inside him cracked. “How can you be on their side?” Barron screamed at the sky, and then he began to weep; because he knew, standing under the malevolent blue glare, that the universe had always been a place of evil.

 

Charles the Blue considered the blue ceiling, taken aback. As a professor in a mixed college, Charles had carefully emphasized that Blue and Green viewpoints were equally valid and deserving of tolerance: The sky was a metaphysical construct, and cerulean a color that could be seen in more than one way. Briefly, Charles wondered whether a Green, standing in this place, might not see a green ceiling above; or if perhaps the ceiling would be green at this time tomorrow; but he couldn’t stake the continued survival of civilization on that. This was merely a natural phenomenon of some kind, having nothing to do with moral philosophy or society . . . but one that might be readily misinterpreted, Charles feared. Charles sighed, and turned to go back into the corridor. Tomorrow he would come back alone and block off the passageway.

 

Daria, once Green, tried to breathe amid the ashes of her world. I will not flinch, Daria told herself, I will not look away. She had been Green all her life, and now she must be Blue. Her friends, her family, would turn from her. Speak the truth, even if your voice trembles, her father had told her; but her father was dead now, and her mother would never understand. Daria stared down the calm blue gaze of the sky, trying to accept it, and finally her breathing quietened. I was wrong, she said to herself mournfully; it’s not so complicated, after all. She would find new friends, and perhaps her family would forgive her . . . or, she wondered with a tinge of hope, rise to this same test, standing underneath this same sky? “The sky is blue,” Daria said experimentally, and nothing dire happened to her; but she couldn’t bring herself to smile. Daria the Blue exhaled sadly, and went back into the world, wondering what she would say.

 

Eddin, a Green, looked up at the blue sky and began to laugh cynically. The course of his world’s history came clear at last; even he couldn’t believe they’d been such fools. “Stupid,” Eddin said, “stupid, stupid, and all the time it was right here.” Hatred, murders, wars, and all along it was just a thing somewhere, that someone had written about like they’d write about any other thing. No poetry, no beauty, nothing that any sane person would ever care about, just one pointless thing that had been blown out of all proportion. Eddin leaned against the cave mouth wearily, trying to think of a way to prevent this information from blowing up the world, and wondering if they didn’t all deserve it.

 

Ferris gasped involuntarily, frozen by sheer wonder and delight. Ferris’s eyes darted hungrily about, fastening on each sight in turn before moving reluctantly to the next; the blue sky, the white clouds, the vast unknown outside, full of places and things (and people?) that no Undergrounder had ever seen. “Oh, so that’s what color it is,” Ferris said, and went exploring.

A Fable of Science and Politics

It is widely recognized that good science requires some kind of humility. What sort of humility is more controversial.

Consider the creationist who says: “But who can really know whether evolution is correct? It is just a theory. You should be more humble and open-minded.” Is this humility? The creationist practices a very selective underconfidence, refusing to integrate massive weights of evidence in favor of a conclusion they find uncomfortable. I would say that whether you call this “humility” or not, it is the wrong step in the dance.

What about the engineer who humbly designs fail-safe mechanisms into machinery, even though they’re damn sure the machinery won’t fail? This seems like a good kind of humility to me. Historically, it’s not unheard-of for an engineer to be damn sure a new machine won’t fail, and then it fails anyway.

What about the student who humbly double-checks the answers on their math test? Again I’d categorize that as good humility. The student who double-checks their answers wants to become stronger; they react to a possible inner flaw by doing what they can to repair the flaw.

What about a student who says, “Well, no matter how many times I check, I can’t ever be certain my test answers are correct,” and therefore doesn’t check even once? Even if this choice stems from an emotion similar to the emotion felt by the previous student, it is less wise.

You suggest studying harder, and the student replies: “No, it wouldn’t work for me; I’m not one of the smart kids like you; nay, one so lowly as myself can hope for no better lot.” This is social modesty, not humility. It has to do with regulating status in the tribe, rather than scientific process. If you ask someone to “be more humble,” by default they’ll associate the words to social modesty—which is an intuitive, everyday, ancestrally relevant concept. Scientific humility is a more recent and rarefied invention, and it is not inherently social. Scientific humility is something you would practice even if you were alone in a spacesuit, light years from Earth with no one watching. Or even if you received an absolute guarantee that no one would ever criticize you again, no matter what you said or thought of yourself. You’d still double-check your calculations if you were wise.

The student says: “But I’ve seen other students double-check their answers and then they still turned out to be wrong. Or what if, by the problem of induction, 2 + 2 = 5 this time around? No matter what I do, I won’t be sure of myself.” It sounds very profound, and very modest. But it is not coincidence that the student wants to hand in the test quickly, and go home and play video games.

The end of an era in physics does not always announce itself with thunder and trumpets; more often it begins with what seems like a small, small flaw . . . But because physicists have this arrogant idea that their models should work all the time, not just most of the time, they follow up on small flaws. Usually, the small flaw goes away under closer inspection. Rarely, the flaw widens to the point where it blows up the whole theory. Therefore it is written: “If you do not seek perfection you will halt before taking your first steps.”

But think of the social audacity of trying to be right all the time! I seriously suspect that if Science claimed that evolutionary theory is true most of the time but not all of the time—or if Science conceded that maybe on some days the Earth is flat, but who really knows—then scientists would have better social reputations. Science would be viewed as less confrontational, because we wouldn’t have to argue with people who say the Earth is flat—there would be room for compromise. When you argue a lot, people look upon you as confrontational. If you repeatedly refuse to compromise, it’s even worse. Consider it as a question of tribal status: scientists have certainly earned some extra status in exchange for such socially useful tools as medicine and cellphones. But this social status does not justify their insistence that only scientific ideas on evolution be taught in public schools. Priests also have high social status, after all. Scientists are getting above themselves—they won a little status, and now they think they’re chiefs of the whole tribe! They ought to be more humble, and compromise a little.

Many people seem to possess rather hazy views of “rationalist humility.” It is dangerous to have a prescriptive principle which you only vaguely comprehend; your mental picture may have so many degrees of freedom that it can adapt to justify almost any deed. Where people have vague mental models that can be used to argue anything, they usually end up believing whatever they started out wanting to believe. This is so convenient that people are often reluctant to give up vagueness. But the purpose of our ethics is to move us, not be moved by us.

“Humility” is a virtue that is often misunderstood. This doesn’t mean we should discard the concept of humility, but we should be careful using it. It may help to look at the actions recommended by a “humble” line of thinking, and ask: “Does acting this way make you stronger, or weaker?” If you think about the problem of induction as applied to a bridge that needs to stay up, it may sound reasonable to conclude that nothing is certain no matter what precautions are employed; but if you consider the real-world difference between adding a few extra cables, and shrugging, it seems clear enough what makes the stronger bridge.

The vast majority of appeals that I witness to “rationalist’s humility” are excuses to shrug. The one who buys a lottery ticket, saying, “But you can’t know that I’ll lose.” The one who disbelieves in evolution, saying, “But you can’t prove to me that it’s true.” The one who refuses to confront a difficult-looking problem, saying, “It’s probably too hard to solve.” The problem is motivated skepticism a.k.a. disconfirmation bias—more heavily scrutinizing assertions that we don’t want to believe.1 Humility, in its most commonly misunderstood form, is a fully general excuse not to believe something; since, after all, you can’t be sure. Beware of fully general excuses!

A further problem is that humility is all too easy to profess. Dennett, in Breaking the Spell: Religion as a Natural Phenomenon, points out that while many religious assertions are very hard to believe, it is easy for people to believe that they ought to believe them. Dennett terms this “belief in belief.” What would it mean to really assume, to really believe, that three is equal to one? It’s a lot easier to believe that you should, somehow, believe that three equals one, and to make this response at the appropriate points in church. Dennett suggests that much “religious belief” should be studied as “religious profession”—what people think they should believe and what they know they ought to say.

It is all too easy to meet every counterargument by saying, “Well, of course I could be wrong.” Then, having dutifully genuflected in the direction of Modesty, having made the required obeisance, you can go on about your way without changing a thing.

The temptation is always to claim the most points with the least effort. The temptation is to carefully integrate all incoming news in a way that lets us change our beliefs, and above all our actions, as little as possible. John Kenneth Galbraith said: “Faced with the choice of changing one’s mind and proving that there is no need to do so, almost everyone gets busy on the proof.”2 And the greater the inconvenience of changing one’s mind, the more effort people will expend on the proof.

But y’know, if you’re gonna do the same thing anyway, there’s no point in going to such incredible lengths to rationalize it. Often I have witnessed people encountering new information, apparently accepting it, and then carefully explaining why they are going to do exactly the same thing they planned to do previously, but with a different justification. The point of thinking is to shape our plans; if you’re going to keep the same plans anyway, why bother going to all that work to justify it? When you encounter new information, the hard part is to update, to react, rather than just letting the information disappear down a black hole. And humility, properly misunderstood, makes a wonderful black hole—all you have to do is admit you could be wrong. Therefore it is written: “To be humble is to take specific actions in anticipation of your own errors. To confess your fallibility and then do nothing about it is not humble; it is boasting of your modesty.”

The Proper Use of Humility

Robin Hanson explores the cognitive styles of "Foxes" and "Hedgehogs" in the context of predictive accuracy. He delves into a study that found Foxes, with their adaptable and flexible thinking, outperformed Hedgehogs, who focus on a single theoretical framework. Hanson also highlights the surprising findings that computer models outperformed human experts in predictions. This thought-provoking essay raises questions about the use of human experts in various fields and the potential for improving accuracy through alternative methods. Click to read more about the fascinating insights from this exploration of predictive success.

I want to follow up on my earlier post on Philip E. Tetlock’s book, Expert Political Judgment, and in particular his discovery of differential predictive accuracy between individuals with the cognitive styles corresponding to "Foxes" vs "Hedgehogs".  As several commenters guessed, his study found that Foxes (who have a flexible, adaptive, tentative cognitive style) significantly outperformed Hedgehogs (who are said to "know one thing and know it well" and to focus on a single, coherent theoretical framework in their analyses and predictions).I first want to emphasize that this is a wide-ranging book with a variety of points of view and directions of analysis.  I hope my focus on one aspect in these blog postings doesn’t give readers too narrow a view of Tetlock’s work.  Here is an excellent review of the book from The New Yorker that goes into more detail on the range of material covered.However, from the point of view of predictive accuracy and bias, Tetlock does organize much of his presentation around the Fox/Hedgehog distinction.  This is not because of some prejudice that this aspect of cognitive style is of supreme importance, but rather that it came out of the data.  More on this below the fold.The headline news from Tetlock’s book is that human experts, in general, did poorly in predictive accuracy.  They barely outperformed a model based on completely random guessing which is colloquially described as "chimps".  And humans were significantly bested by several formula-based models which made predictions based on extrapolating past results.  Here’s a table showing the success of various prediction methods, in terms of a rather complicated measure which I can’t explain here, but for which higher is better:-.045 – Expansive base rate model (recent past)-.025 – Restrictive base rate model (recent past)-.01  – Chimps0.00  – Human experts0.00  – Contemporary base rate model0.025 – Aggressive case-specific extrapolation model0.035 – Cautious case-specific extrapolation model0.07  – Autoregressive distributed lag modelsI won’t try to explain all the different models, except to note that the "chimps" guess at random, while the first two base rate models (which did poorly) make their guesses to match the probabilities of recent history.  The "contemporary base rate" model matches human performance but it "cheats" by using the base rate for the whole period including future events.The more elaborate models did considerably better, and the autoregression model in particular greatly outperformed humans.  In order to turn this measure into something more tangible, I will quote Tetlock, "whereas the best human forecasters were hard-pressed to predict more than 20 percent of the total variability in outcomes…, the generalized autoregressive distributed lag models explained on average 47 percent of the variance."So the loudest message from the data is that relatively straightforward computer models do much better than humans, which raises the question, why do we use human experts at all in these fields?  I don’t have an answer to that.  But clearly this points to the possibility of greatly improving the accuracy of our forecasts of political events, such as the current turmoil in the Middle East, by ignoring experts and just doing some curve fitting.  This area is ripe for further work.However Tetlock is a psychologist and chooses to focus instead on trying to understand details of why humans did as well or poorly as they did.  His first effort was to look for factors that correlate with prediction accuracy.  He focused on three categories: differences in background and accomplishments; different in content of belief systems; and differences in styles of reasoning.The first category produced no statistically significant correlations.  Factors such as education, years of experience, academic vs non-academic work, and access to classified information were not significant.  The strongest effect here was a mild negative correlation to fame: the better known an expert was, the poorer he did.Among factors relating to belief content, left-right ideology and idealist-realist distinctions were not significant.  There was a weak statistically significant measure Tetlock calls doomster-boomster relating to optimism about human potential.  Generally for all these categories the moderates did somewhat better than the extremists.It was in cognitive style that the strongest correlation was found, statistically significant at the 1% level.  And that is the Fox-Hedgehog distinction.  Let’s re-do the table above and put in Foxes and Hedgehogs:-.045 – Expansive base rate model (recent past)-.025 – Restrictive base rate model (recent past)-.025 – "Hedgehogs"-.01  – Chimps0.00  – Contemporary base rate model0.015 – "Foxes"0.025 – Aggressive case-specific extrapolation model0.035 – Cautious case-specific extrapolation model0.07  – Autoregressive distributed lag modelsThe best Foxes approach the accuracy of the case-specific extrapolation model, while the Hedgehogs do considerably worse than chimps.  The predictions of the best Foxes explain 20% of the variance in outcomes, while the worst Hedgehogs explain less than 7% of the variance.Tetlock found that some of the other cognitive factors listed above do moderate the overall dominance of Foxes over Hedgehogs.  One rather bizarre result relates to an experiment where Tetlock asked experts to also make predictions far from their field of expertise.  Among Foxes, as we might expect, experts do better than the "dilettantes" who are going outside their field.  But among Hedgehogs, the results are reversed!  Hedgehogs actually do worse in their own fields where they are supposed experts than when they are forced to make predictions in other areas.Another area of difference is the distinction between extremists and moderates on political philosophy.  Among Foxes, these categories didn’t matter much.  But among Hedgehogs, it made a huge difference, with Hedgehog extremists doing far worse than moderates.Again, I can’t do more than scratch the surface of the material which Tetlock presents in depth here.  But I will note one other result which was surprising and amusing.  There exist various methodologies to attempt to improve analysis and prediction by "de-biasing" individuals and getting them to be more open-minded in their approach.  Tetlock studied one such approach, scenario exercises, where participants are presented with a number of alternative scenarios before beginning their analysis, in the hope that this will get them to consider factors that they might otherwise overlook.In Tetlock’s study, this approach was a failure, but the effects were different for Foxes and Hedgehogs.  Hedgehogs tended to reject scenarios which did not fit their pet models, in effect snorting in derision at what seemed to them to be far-fetched and pointless speculation.  Their scores did not change in these exercises.  However, Foxes did respond to the scenarios, but in a negative way.  Their scores actually got worse as they seemed to get caught up in the complexities of all the different scenarios, becoming in effect too open-minded and giving too much consideration to inherently unlikely possibilities.Summing up this perhaps over-long posting, the Fox-Hedgehog distinction appears to be relevant and important in understanding how people analyze uncertain situations and make forecasts about possible outcomes.  Tetlock’s book is a great step forward in shedding light on the factors which influence predictive success.  Understanding ones own position on the Fox-Hedgehog scale can be a useful tool in calibrating our own predictive abilities and perhaps may point the way towards self-improvement.I took the test after creating it, and although it was difficult to be unbiased since I knew that Foxes were better, I tried to compensate for that.  My score ended up a -7, weakly Hedgehog.  Indeed I am attracted to the idea that there is some big insight, some magic tool (prediction markets? autoregressive distributed lag models?) which can cut through the messy complexity of human endeavor and provide clear insight into the fog of the future.  The lesson of Tetlock’s studies is that this temptation is more likely to lead to Hedgehogly narrow-mindedness than Foxy predictive success.

Foxes vs Hedgehogs: Predictive Success

Some time back I thought of an example which shed light for me on some of the fail-to-disagree results. Imagine that two players, A and B, are going to play a coin-guessing game. A coin is flipped out of sight of the two of them and they have to guess what it is. Each is privately given a hint about what the coin is, either heads or tails; and they are also told the hint "quality", a number from 0 to 1. A hint of quality 1 is perfect, and always matches the real coin value. A hint of quality 0 is useless, and is completely random and uncorrelated with the coin value. Further, each knows that the hint qualities are drawn from a uniform distribution from 0 to 1 – on the average, the hint quality is 0.5.The goal of the two players is to communicate and come up with the best guess as to the coin value. Now, if they can communicate freely, clearly their best strategy is to exchange their hint qualities and just follow the hint with the higher quality. However we will constrain them so they can’t do that. Instead all they can do is to describe their best guess at what the coin is, either heads or tails. And further, we will divide their communication into rounds, where in each round the players simultaneously announce their guesses to each other. Upon hearing the other player’s guess, each updates his own guess for the next round.Read on below the break for some sample games to see how the players can resolve their disagreement even with such stringent constraints.Here’s a straightforward example where we will suppose A gets a hint with quality 0.8 of Heads, and B gets a hint with quality 0.6 of Tails. Initially the two sides tell each other their best guess, which is the same as their hint:A:H  B:TNow they know they disagree. Their reasoning can be as follows:A: B’s hint quality is uniform in [0,1], averaging 0.5. My hint quality is higher than that at 0.8, so I will stay with Heads.B: A’s hint quality is uniform in [0,1], averaging 0.5. My hint quality is higher than that at 0.6, so I will stay with Tails.A:H  B:TSo they remain unchanged. Now they reason:A: B did not change, so his hint quality must be higher than 0.5. That is all I know, so it must be uniform in [0.5,1], averaging 0.75. My hint quality is higher than that at 0.8, so I will stay with Heads.B: A did not change, so his hint quality must be higher than 0.5, so it must be uniform in [0.5,1], averaging 0.75. My hint quality is lower than that at 0.6, so I will switch to Heads.A:H  B:HAnd they have come to agreement. If both A and B had had higher hint qualities, they might have persisted in their disagreement for more rounds, but each refusal to switch tells the other party that their hint quality must be even higher, and eventually one side will give way.It’s improbable that both sides will have high but opposite hint qualities. What happens in the more likely case where they have low but opposite hint qualities? Let’s suppose that A gets a hint of Heads with quality 0.1, and B gets a hint of Tails with quality 0.15.A:H  B:TA: B’s hint quality is uniform in [0,1], averaging 0.5, which is higher than my 0.1, so I will switch to Tails.B: A’s hint quality is uniform in [0,1], averaging 0.5, which is higher than my 0.15, so I will switch to Heads.A:T  B:HA: B switched, so his hint quality was lower than 0.5, making it uniform in [0,0.5] and averaging 0.25, which is higher than my 0.1, so I will stay with Tails (B’s original guess).B: A switched, so his hint quality was lower than 0.5, making it uniform in [0,0.5] and averaging 0.25, which is higher than my 0.1, so I will stay with Heads (A’s original guess).A:T  B:HA: B stayed the same, so his hint quality was lower than 0.25, making it uniform in [0,0.25] and averaging 0.125, which is higher than my 0.1, so I will stay with Tails.B: A stayed the same, so his hint quality was lower than 0.25, making it uniform in [0,0.25] and averaging 0.125, which is lower than my 0.15, so my original hint quality was higher, and I will switch back to my original Tails.A:T  B:TOnce again agreement is reached. Note that when both sides have a low hint quality, they initially switch to the other side’s original view, then they each stick with that new side. After enough rounds one of them decides that the other’s hint must have been so poor that his hint was better, and he switches back to reach agreement.An interesting case arises if the hint qualities are near 1/3 or 2/3. In that case we can get sequences like this (I will skip the reasoning, you can work it out if you like):A:H  B:TA:T  B:HA:H  B:TA:T  B:HA:H  B:HHere we can have both sides changing back and forth potentially several times, each side taking the other’s view, until they come to agreement.A few interesting points about this game. It’s a simple model that captures some of the flavor of the no-disagreement theorem. In the real world we have hints about reality in the form of our information; and there is something like a "hint quality" in terms of how good our information is. If we were Bayesians we could both report our hint qualities when we disagree, and go with the one that is higher. Even if we are limited merely to reporting our opinions as in this game, we should normally reach agreement pretty quickly.Another interesting aspect is that when you play the game, you can never anticipate your partner’s guess. On each round you have an idea of the range of possible hint qualities he might have, based on his play so far, and it always turns out that given that range, he is equally likely to guess Heads or Tails on the next round. This is related to Robin’s result that the course of opinions among Bayesians in resolving disagreement goes as a random walk.As I noted, in the real world it should be uncommon for two people to have high quality but opposing hints, because high quality hints are supposed to be accurate. Hence it should be rare for people to stubbornly disagree and stick to their original viewpoints. Much more common should be the case where people have low quality hints which disagree. In that case, as we saw, people should switch position at least once, and then (depending on how low the hint quality was) either stick to their reversed position or else possibly alternate some more. This should be a common course of disputation between Bayesians, but it is strikingly rare among humans.Another point this game illustrates is that the Aumannian notion of "common knowledge" may not be as easy to use as it seems. Note in this game that even after announcing their positions, players’ (current) views are not common knowledge. After each round, a player got new information that could have changed his view from when he stated it before. Once they reach agreement, then things seem to stabilize, but that may not be the case in general. I have constructed different games in which people can agree for two consecutive rounds and then disagree. It is an open question to me whether two people can agree for N rounds and then disagree, for arbitrary N.

The Coin Guessing Game

An interesting paradox arise if one tries to apply Aumann’s theorem in a world of widespread disagreement. The problem is especially apparent in considering the stronger principle which Eliezer calls the Modesty Argument, which claims not only that rational, mutually-respecting people should not agree to disagree, but that in general one should be agreeable and allow oneself to be easily persuaded in any argument.I’ll introduce it with one of my favorite jokes:This is a story about advice, and any story about advice becomes a story about a village rabbi. Two men came to see the rabbi of their village.The first one said, "Rabbi, I have a pear tree in my yard. My father planted it, I keep it watered and don’t let the chickens peck on its roots. One of its branches hangs over my neighbor’s wall. So what do I see yesterday but my neighbor standing there eating one of my pears. This is theft, and I want him to pay me damages."The rabbi nodded his head. "You’re right, you’re right."The other neighbor said, "Rabbi, you know that I have seven children–without my garden to feed them, how would I manage? But that tree casts a shadow where nothing will grow. So yesterday, when I go out to dig some potatoes, a pear from his tree falls and hits me right on the head. How am I hurting my neighbor if I eat it? And doesn’t he owe me something for his tree’s blocking sunlight?The rabbi thought and then said, "You’re right, you’re right."Meanwhile, the rabbi’s wife had been hearing all this. "How can you say, ‘You’re right’ to both of these men? Surely one of the men is right, and the other is wrong!"The rabbi looked unhappy. "You’re right, you’re right."Suppose that I am somewhat neutral on a given issue and I am going to argue with one of two partisans on the issue, who each have opposing views. Applying the Modesty Argument it might seem that I should be prepared to be persuaded to the view of whichever one of them I interact with. It is true, in talking to one of them, that we both know that other people exist with opposing views, but since this person has not been persuaded by this knowledge, he must have good reason to hold to his particular side. There is no reason a priori to suppose that my beliefs on this are any better founded than his, especially since we stipulated that it was a position on which I was relatively ignorant. So it is plausible, given that we reach agreement, and given his long-standing commitment to his position, that our resulting agreement will generally be favorable to his side of the argument.The problem is that I can equally well reason that if I were to interact instead with the person who takes the opposite view, I would be just as convinced to take that other side. This evokes a rather comical image of sequential interactions with the two partisans, in which I am first convinced of one side and then the other. It seems that someone following the Modesty Argument too literally will find himself in the position of the poor rabbi in the joke.The problem and paradox is that this behavior is irrational. You can’t be in a position where you have an expectation that you can gain some information that will move your opinion in a predictable direction. Just knowing this fact about the situation should be enough to move your opinion already. Robin has what I think is a formal proof of this in his paper, Disagreement is Unpredictable (not sure I understood it correctly though). For a rational person, every new piece of information is a surprise in the sense that he can’t predict which way it will change his opinion, on the average. When I post these messages here I have no idea whether I will be more or less convinced of my opinion as a result of the feedback I receive. That is how it always must be.Therefore I can’t be in a position where I know if I talk to person A I will be convinced that he is right, and if I talk to person B I will be convinced that A is wrong. A practitioner of the Modesty Argument who finds himself in this situation is doing something wrong.So there are two questions: what is wrong with the argument above that puts us in the shoes of the rabbi; and what in fact should a rational and modest person do in a world full of disagreement?First, I seem to have gone wrong when I concluded that if I argued with person A, I would come to agree with him. As I just pointed out, it is irrational to have an expectation that gaining information will predictably change one’s opinion. Instead, it must be the case that I am completely uncertain about how my opinion will change if I talk with A. Given that we will agree, it must be my expectation that A is as likely to change his mind as that he will convince me. Now, I argued above that this is unlikely given A’s long-standing position as a partisan in the dispute. Therefore I have to conclude that A does not meet the conditions for Aumann’s result; he is not rational, or honest, or at least he assumes that those he disagrees with lack these traits. In that case I cannot apply Aumann’s theorem and cannot act in accordance with the Modesty Argument in my discussions with A or B.It’s interesting to consider what would happen if A and B are in fact both rational and honest, but both distrust whether their counterpart shares these properties. This is a stable configuration, and it’s rational for A and B to disagree in this situation; but of course they lack the mutual respect which is generally associated with saying that they agree to disagree. Now suppose A and B both trust me as an "honest broker", and I trust them. Then I can’t disagree with A, and I can’t disagree with B, once I interact with them. What will happen is that when I interact with A, we two will agree; and when I interact with B, we two will agree. In this way, A and B will come to agreement by both agreeing with me (I might have to repeat the interaction a few times). This again illustrates how odd is the persistence of disagreement in the world, among people who claim to be rational and honest.But realistically, disagreement does persist, which leaves the question of how a rational person should behave. Unfortunately, I don’t think that Aumann’s theorem or the Modesty Argument offers much guidance. It’s not rational to adopt a stance where you are easily persuaded by everyone you talk to. There’s no particular reason, just because you are talking to a proponent of one side, to believe him, given that you know there are plenty of people around who believe the opposite. It’s not that you think you are better (smarter, more knowledgeable, more rational) than he is; it’s just that you assume there are many equally smart, knowledgeable and rational people on the other side. In the end, you have to make your decision on controversial issues using other grounds than the irrationality of disagreement.

Modesty in a Disagreeable World

You’ll often hear people, including people on this blog, talk about self-deception as a central factor in creating bias.  Work from cognitive neuroscience, however, shows that the "self-deception" description doesn’t really capture how deep the issue runs.  In this post, I’ll give an analogy that a number of us use these days to describe our understanding of the cognitive architecture of conscious speech.  The earliest I can remember using the analogy was in a paper I wrote in the mid/late 90s, and the model behind it drives some of my comments over at Chatty Apes.  Jon Haidt gives the analogy in an interview here.  Rob Kurzban has a superb new article on it here.  Below, I provide the analogy.Imagine some bureaucratic organization with a number of different departments.  A sort of caricature of George W. Bush’s presidential administration works.  There are a number of strong-willed people in key White House positions (including Bush, Dick Cheney, and Karl Rove) with overlapping but somewhat different agendas that lay out simple priorities (Cut taxes for rich people!  Get Saddam!  Help out our fat-cat cronies in defense and oil industries!  Keep us and people like us in power!).  There are a number of different topic-specific departments (Defense, Commerce, Education, etc.) with dedicated staffs that work on detailed operations.  Some departments or staffs are focused on coordinating action with other institutions like Congress or foreign governments.  And there are communications offices tasked specifically with coordinating information flow to the public in a way that will best allow the deep, unpublicized preferences of the Administration to be translated into action.Inside the Administration, there is a constant balancing act between what they really want to do and what is possible.  Some things they may want to do are limited by their ability to get people (Congress, foreign governments, the public) to go along with it.  Here, communications strategies become crucial.  Maybe they want to cut taxes specifically for rich people, but they understand that most people aren’t rich and will strongly disapprove of a bunch of rich-only tax cuts.  The solution may be to combine very large tax cuts for rich people with very small tax cuts for middle-class people in a terrifically confusing package, and then tell everyone, consistently and earnestly, that one’s proposal simply involves "across the board" tax cuts for everyone.  If most people don’t have the information necessary to learn the truth, and mostly they’re just relying on the Administration’s communications to figure out whether it’s a good idea for them, the Administration could achieve both goals of cutting rich people’s taxes and being well thought of by middle-class voters.So, the Administration decides to cut rich people’s taxes, and it comes up with a communication strategy that is maximally bent on effecting the policies and has little direct relationship to the real motivations behind the policies.  Optimally, the Administration will appoint press agents to communicate the message, and, even more optimally, will in fact not explain to the press agents what the real motives are behind the favored actions.  It will just tell those press agents the popular version with the popular motives.  The best liars believe the lie.If you’re in a position to ask an Administration press agent about the policy, that person will tell you, consistently and earnestly, that throughout the Administration the motives are the unobjectionable ones.  The President thinks the fair thing is to give all hard-working Americans a little across-the-board tax relief.Press secretaries often get it wrong through no fault of their own even with less vague things than motives.  Consider the Valerie Plame CIA leak scandal.  Early on, Scott McClellan, Bush’s press secretary at the time, positively insisted that Karl Rove had nothing to do with it.  "The President knows that Karl Rove wasn’t involved."  But then it turned out, not so surprisingly, that Rove and others were more than a little involved.  A likely story here is that Rove lied to McClellan and his staff, and that McClellan was sent out with bad information to be the most aggressive spokesman he could be.Now the very difficult point:  This analogy is meant to be about the human brain — it’s your brain I’m talking about.  Research from psychology and neuroscience shows that your brain has organizational characteristics similar to this caricature of the Bush Administration.   There are systems that are responsible for powerful, simple emotional reactions that serve to focus other systems to give fine-tuned attention to your brain’s priorities and preferred outcomes.  Most of the detailed systems don’t care much about why they have the jobs they are given; they just do the work of carrying them out in a highly distributed, bureaucratic way.  And you — the conscious, chatty you — are that dimwitted patsy, the misinformed press agent.   The conscious you is not the President and you’re certainly not Karl Rove.  You are the Scott McClellan for your bureaucratic brain.  You’re constantly being duped into believing public-friendly stories about yourself, because your entire job is to tell stories handed to you by ruthless, clever, unconscious communications systems.  You’re not the whole head; you’re just the talking head.You can’t even trust your own introspection, because your press agent simply doesn’t have direct access to the Oval Office.  The best you can do, even with your own behavior, is to try to piece together hypotheses about the hidden motives at work based on what the person actually does, situated in the context of things you know are generally true of why people might want to do those things."Self-deception" doesn’t fully capture the depth of what’s going on here.  When you talk to someone, you’re not talking to an integrated "self" that may or may not be deceived — you’re talking to a press agency that doesn’t really make many important decisions, that is only loosely connected to the rest of the bureaucracy, and that has the core mission not of truth-seeking, but bias that serves the interests of its genes.The thing I really worry about:  To the extent that our scientific accounts of people’s underlying motivations end up painting, as I think they will, an unflattering picture about human predictability and genetic selfishness, I think we’re likely to end up with a small group of scientists who acknowledge it, and a very large group of people (including most scientists) with press agencies that are strongly motivated to deny it and so drift farther away from paying attention to the better work in social science.  How’s that for optimism?

The cognitive architecture of bias

A few days ago I showed a plot of oil futures prices, and Robin made the point that it would be useful to see information about variance as well. For each futures contract there are a set of options which can be used to deduce information about how much uncertainty the market sees in future prices. For example, the commodities price yesterday for December 2010 oil was $67 per barrel. But for $2 you could buy a call option with a strike price of $100. This will have a value of P – $100 if at the end of 2010 the oil price P is higher than $100 (and will have no value if the oil price is lower than $100). Therefore if oil goes up to $105 you will more than double your money; if it hits $122 you will make ten times your investment. Clearly the market does not view these possibilities as very likely, or the option would not be as cheap as $2. Below the fold I will describe and illustrate a simple technique for deriving probability estimates for price targets merely by glancing at tables of option values. I find it useful and practical in terms of getting a quick overview of how likely the market judges various possibilities.To get more precise information from option prices, there are different methods you can use. One is to apply the Black-Scholes option valuation formula to deduce the market’s expectation for the volatility of the underlying commodity. You can then use this estimated volatility to model the commodity price as a random walk. In this way you can deduce a confidence interval for the market’s expectation of future prices at each time period.I want to describe a lesser known but simpler approach that lets you read off the market’s confidence intervals almost at a glance. It works like this. Look at a table of option trading prices for a given commodity contract, for example December 2010 crude oil. Consider out-of-the-money options, such as call options for $80, $90 and on up. We want to get a picture of the market’s estimate that oil will exceed each of those prices on that December.Let x be the option strike price and c(x) be the price of the corresponding call option. Now let us imagine that prices exist for every x value. This is not true in practice; oil option strike prices are typically separated by $1, sometimes only $0.50, in the center of the distribution, but get more widely spaced as we move out. But let us imagine an ideal contract where this is so.Now consider setting up a trading position where we go long (i.e. we buy) an option contract at strike price x, and simultaneously go short (i.e. we sell) an option contract at price x+dx for some small dx. This will cost us c(x)-c(x+dx) which is a positive amount since c(x) is a decreasing function. The value of this option at expiration is, if the oil price P is greater than x+dx, (P-x) – (P-x-dx) = dx. If P is less than x, the value is zero. And for small dx we can neglect the case where P is between x and dx.If the market’s estimate of the probability that the price will exceed x is p, then the expected value of the contract is therefore p*dx. And the cost ofthe contract must equal its expected value: c(x)-c(x+dx) = p*dx; divide by dx and we get (c(x)-c(x+dx))/dx = p. For small dx the left side approximates the negative of the derivative of the call option price with respect to the strike price, which we will call c’ (c-prime), producing the final result: -c'(x) = p.In other words, you can determine the market’s estimate of the chance of exceeding a given price simply by evaluating the derivative of the call option. And since option prices are typically separated by simple values like 1, 5 or 10, this can be often be done by merely subtracting two values, sometimes with a simple shift of the decimal point or a small division. The same process and reasoning applies to put options, and again it is just a matter of subtracting two entries in the table and dividing if necessary by the difference in strike prices.To show an example, here are some strike prices and values for December 2010 call options as of the close of trading December 26:70   7.4175   5.7380   4.3785   3.4090   2.73100 1.96110 1.49To estimate the probability of oil closing above 75 on that date, take the entries for 70 and 80 and subtract: 7.41-4.37 = 3.04. Then divide by 10, the distance between 70 and 80, to get 30.4% as the estimate of this probability. For the probability of oil closing above 80, 5.73-3.40 = 2.43, divided by 10 is 24.3%. For oil closing above 85 it is similarly 16.4%. For oil closing above 100, the example considered at the top of this post, use 90 and 110: 2.73-1.49 = 1.24. And divide by 20 this time, the difference between 90 and 110, to get 6.2%. Those are the odds which market prices imply that traders perceive for oil being above $100 per barrel at the end of 2010.Here’s a chart where I applied this process to the call options for this December 2010 contract, plotting the approximate derivative (Note that this term usually means something else in finance! This is really the derivative of a “derivative”) against the strike price (taken as the midpoint of the interval in the table). It effectively shows the market’s opinion of the probability of exceeding each price level at that date:Relating this back to the controversy over Peak Oil, again we see that markets do not appear to foresee a significant chance that we will be experiencing severely high oil prices in the 2010 time frame, such as would be associated with a shortage. Even the chance of hitting 90 or above is only about 10%, and that would not even be an all time high, adjusting for inflation.For completeness, here is the corresponding chart for put options. This indicates the probability of closing below the specified price at the end of 2010:A couple of caveats in applying this technique: first, it works best with closing prices, because during the day there is too much variation as trading goes on, and taking the derivative magnifies the noise. And second, the most actively traded long term contracts are for December and, to a lesser extent, June, and those will generally have the most complete set of option prices. I hope you will find this, as I have, to be a practical and useful tool in getting a picture of market probability estimates, which I assume will generally be greatly superior to what I might come up with on my own.[Update: the probability figures above are about 22% too low. Please see the comments for the corrected formula. Thanks to Perry Metzger for pointing out my error.]

Read Overcoming Bias

Choose Your Reading Path

Full Backlog

Best Hits

You'll start with:

How To Join

Hide Sociobiology Like Sex?

Quiz: Fox or Hedgehog?

About Overcoming Bias