In 1960, Peter Wason conducted a now-classic experiment that became known as the '2-4-6' task. (Wason 1960.) Subjects had to discover a rule, known to the experimenter but not to the subject - analogous to scientific research. Subjects wrote three numbers, such as '2-4-6' or '10-12-14', on cards, and the experimenter said whether the triplet fit the rule or did not fit the rule. Initially subjects were given the triplet 2-4-6, and told that this triplet fit the rule. Subjects could continue testing triplets until they felt sure they knew the experimenter's rule, at which point the subject announced the rule.
Although subjects typically expressed high confidence in their guesses, only 21% of Wason's subjects guessed the experimenter's rule, and replications of Wason's experiment usually report success rates of around 20%. Contrary to the advice of Karl Popper, subjects in Wason's task try to confirm their hypotheses rather than falsifying them. Thus, someone who forms the hypothesis "Numbers increasing by two" will test the triplets 8-10-12 or 20-22-24, hear that they fit, and confidently announce the rule. Someone who forms the hypothesis X-2X-3X will test the triplet 3-6-9, discover that it fits, and then announce that rule. In every case the actual rule is the same: the three numbers must be in ascending order. In some cases subjects devise, "test", and announce rules far more complicated than the actual answer.
Wason's 2-4-6 task is a "cold" form of confirmation bias; people seek confirming but not falsifying evidence. "Cold" means that the 2-4-6 task is an affectively neutral case of confirmation bias; the belief held is logical, not emotional. "Hot" refers to cases where the belief is emotionally charged, such as political argument. Unsurprisingly, "hot" confirmation biases are stronger - larger in effect and more resistant to change. Active, effortful confirmation biases are labeled motivated cognition (more ordinarily known as "rationalization"). As put by Brenner et. al. (2002) in "Remarks on Support Theory":
Clearly, in many circumstances, the desirability of believing a hypothesis may markedly influence its perceived support... Kunda (1990) discusses how people who are motivated to reach certain conclusions attempt to construct (in a biased fashion) a compelling case for their favored hypothesis that would convince an impartial audience. Gilovich (2000) suggests that conclusions a person does not want to believe are held to a higher standard than conclusions a person wants to believe. In the former case, the person asks if the evidence compels one to accept the conclusion, whereas in the latter case, the person asks instead if the evidence allows one to accept the conclusion.
When people subject disagreeable evidence to more scrutiny than agreeable evidence, this is known as motivated skepticism or disconfirmation bias. Disconfirmation bias is especially destructive for two reasons: First, two biased reasoners considering the same stream of evidence can shift their beliefs in opposite directions - both sides selectively accepting only favorable evidence. Gathering more evidence may not bring biased reasoners to agreement. Second, people who are more skilled skeptics - who know a larger litany of logical flaws - but apply that skill selectively, may change their minds more slowly than unskilled reasoners.
Taber and Lodge (2000) examined the prior attitudes and attitude changes of students when exposed to political literature for and against gun control and affirmative action. The study tested six hypotheses using two experiments:
1. Prior attitude effect. Subjects who feel strongly about an issue - even when encouraged to be objective - will evaluate supportive arguments more favorably than contrary arguments.
2. Disconfirmation bias. Subjects will spend more time and cognitive resources denigrating contrary arguments than supportive arguments.
3. Confirmation bias. Subjects free to choose their information sources will seek out supportive rather than contrary sources.
4. Attitude polarization. Exposing subjects to an apparently balanced set of pro and con arguments will exaggerate their initial polarization.
5. Attitude strength effect. Subjects voicing stronger attitudes will be more prone to the above biases.
6. Sophistication effect. Politically knowledgeable subjects, because they possess greater ammunition with which to counter-argue incongruent facts and arguments, will be more prone to the above biases.
Ironically, Taber and Lodge's experiments confirmed all six of the authors' prior hypotheses. Perhaps you will say: "The experiment only reflects the beliefs the authors started out with - it is just a case of confirmation bias." If so, then by making you a more sophisticated arguer - by teaching you another bias of which to accuse people - I have actually harmed you; I have made you slower to react to evidence. I have given you another opportunity to fail each time you face the challenge of changing your mind.
Heuristics and biases are widespread in human reasoning. Familiarity with heuristics and biases can enable us to detect a wide variety of logical flaws that might otherwise evade our inspection. But, as with any ability to detect flaws in reasoning, this inspection must be applied evenhandedly: both to our own ideas and the ideas of others; to ideas which discomfort us and to ideas which comfort us. Awareness of human fallibility is a dangerous knowledge, if you remind yourself of the fallibility of those who disagree with you. If I am selective about which arguments I inspect for errors, or even how hard I inspect for errors, then every new rule of rationality I learn, every new logical flaw I know how to detect, makes me that much stupider. Intelligence, to be useful, must be used for something other than defeating itself.
You cannot "rationalize" what is not rational to begin with - as if lying were called "truthization". There is no way to obtain more truth for a proposition by bribery, flattery, or the most passionate argument - you can make more people believe the proposition, but you cannot make it more true. To improve the truth of our beliefs we must change our beliefs. Not every change is an improvement, but every improvement is necessarily a change.
Our beliefs are more swiftly determined than we think. Griffin and Tversky (1992) discreetly approached 24 colleagues faced with a choice between two job offers, and asked them to estimate the probability that they would choose each job offer. The average confidence in the choice assigned the greater probability was a modest 66%. Yet only 1 of 24 respondents chose the option initially assigned the lower probability, yielding an overall accuracy of 96% (one of few reported instances of human under confidence).
The moral may be that once you can guess what your answer will be - once you can assign a greater probability to your answering one way than another - you have, in all probability, already decided. And if you were honest with yourself, you would often be able to guess your final answer within seconds of hearing the question. We change our minds less often than we think. How fleeting is that brief unnoticed moment when we can't yet guess what our answer will be, the tiny fragile instant when there's a chance for intelligence to act. In questions of choice, as in questions of fact.
Thor Shenkel said: "It ain't a true crisis of faith unless things could just as easily go either way."
Norman R. F. Maier said: "Do not propose solutions until the problem has been discussed as thoroughly as possible without suggesting any."
Robyn Dawes, commenting on Maier, said: "I have often used this edict with groups I have led - particularly when they face a very tough problem, which is when group members are most apt to propose solutions immediately."
In computer security, a "trusted system" is one that you are in fact trusting, not one that is in fact trustworthy. A "trusted system" is a system which, if it is untrustworthy, can cause a failure. When you read a paper which proposes that a potential global catastrophe is impossible, or has a specific annual probability, or can be managed using some specific strategy, then you trust the rationality of the authors. You trust the authors' ability to be driven from a comfortable conclusion to an uncomfortable one, even in the absence of overwhelming experimental evidence to prove a cherished hypothesis wrong. You trust that the authors didn't unconsciously look just a little bit harder for mistakes in equations that seemed to be leaning the wrong way, before you ever saw the final paper.
And if authority legislates that the mere suggestion of an existential risk is enough to shut down a project; or if it becomes a de facto truth of the political process that no possible calculation can overcome the burden of a suggestion once made; then no scientist will ever again make a suggestion, which is worse. I don't know how to solve this problem. But I think it would be well for estimators of existential risks to know something about heuristics and biases in general, and disconfirmation bias in particular.
Continue reading here: Anchoring adjustment and contamination
Was this article helpful?