Daniel Kahneman is an Israeli-American psychologist and Princeton professor who wrote the 2012 bestselling book Thinking, Fast and Slow and was awarded the 2002 Nobel Prize in Economics. Olivier Sibony teaches Strategy, Decision Making and Problem Solving at HEC Paris and wrote You’re About to Make a Terrible Mistake. Cass Sunstein is a former White House official, a professor at Harvard University, and the author of numerous books, most recently This Is Not Normal and Too Much Information.
Below, Olivier and Cass share 5 key insights from their new book, Noise: A Flaw in Human Judgment (available now from Amazon), which they co-wrote with Daniel. Listen to the audio version—read by Olivier and Cass themselves—in the Next Big Idea App.
1. Noise is a form of error, different from bias, but no less important.
Suppose that five shooters aim at the same target and all of them miss. Their shots are clustered to the bottom and to the left of the bullseye. This is a shared error. Obviously, it’s not good. Now, imagine another team of five shooters. Their shots are now all over the target, and there is no cluster this time. The center of gravity of the shots is close to the bullseye, yet they all miss it. Obviously, that’s not good either.
The metaphor of the targets is a good way to think about two kinds of error: bias and noise. In judgment, as in shooting, there are shared errors when most people err in the same direction. That’s bias. But there are also errors that are not shared—that’s noise. This is clearly a problem in forecasting. If ten sales analysts at a company try to forecast next year’s sales, and if the average of the forecasts is two points higher than what actually happens, we’ll say that the forecasters have an optimistic bias. But if the ten analysts do not all agree on the direction of the forecast, this is noise. Mathematically speaking, bias and noise contribute to overall error in exactly the same manner, but while it is clear that reducing bias will always reduce error, the value of reducing noise is not entirely intuitive.
“Wherever human beings make judgments, they are noisy. It’s a source of error as well as injustice.”
2. Wherever there is judgment, there is noise—and more of it than you think.
Of course, you wouldn’t expect people to agree perfectly. The surprise isn’t the existence of noise—it’s the size. In a study of judges deciding sentences in criminal cases, the average sentence was seven years. But the difference between sentences handed down by two randomly chosen judges was 3.5 years. That amount of noise could be considered a violation of equal treatment under the law.
Insurance executives expect a modest amount of noise in the process of setting premiums. They don’t expect their underwriters to agree entirely, but one study showed that the level of noise was five times as high as expected. That creates a problem of fairness for people buying insurance, as well as a profitability problem for the insurance companies. If their premiums are set too high, they lose business, and if they’re too low, they lose money in claims.
There’s noise in medicine, too. Doctors’ medical judgments can be impacted by factors like their mood, creating a lot of noise in the prescriptions and treatments they recommend.
Wherever human beings make judgments, they are noisy. That is a problem that we typically don’t attend to. And we really ought to, because it’s a source of error as well as injustice.
3. There are three sources of noise, and the largest is the least obvious.
Suppose that we ask two judges in the same courthouse—Judge A and Judge B—to look at ten different cases in which a defendant has been found guilty, and to set an appropriate sentence. Variability in their judgments of the same cases is unwanted in an ideal world—the sentence should not depend on the judge. This variability is noise. But where does it come from?
An answer probably already came to your mind—Judge A may be a tough judge, and Judge B may be a lenient, bleeding-heart judge. If we compute the average sentence that the two judges give to the ten defendants, there will indeed be a difference. The average level of judgments varies from one person to the next. And this is a first source of noise, which we call level noise.
There is another cause of noise that is quite intuitive, too: The judge is a human being. There are days when she is tired, in a bad mood, or in a hurry. And there are days when she is rested, attentive, and understanding. Her judgments will, of course, not be completely different. But they will also not be perfectly identical from one day to the next. Judgments of the same case by the same person will vary from one occasion to the next, and this is what we call occasion noise.
“When we expect consistency in judgments, uniqueness can become a liability.”
Then there is a third source of noise, and in many cases it is the largest. Imagine that you ask Judge A and Judge B to rank the ten cases from the one that deserves the most severe punishment to the least severe one. Their ranking is not going to be identical; Judge A may be tough in general, but there are offenses about which he is less stiff, and there are defendants that strike a chord with her. Judge B may be lenient in general, but perhaps she’s not as lenient as Judge A when the victim was especially vulnerable, or when the offender is a recidivist. In short, the judges have a different pattern of judgment. This, too, is a source of noise, which we call pattern noise.
What makes pattern noise particularly interesting and also hard to combat is that it reflects the values, preferences, and tastes of individuals. Judgments are different and therefore noisy because the people who made these judgments are different. We usually celebrate this individuality and this uniqueness of people, but when we expect consistency in judgments, uniqueness can become a liability.
4. Noise can be reduced by practicing decision hygiene.
If you wash your hands, you’re protecting yourself against germs, even if you don’t know what particular germs you’re protecting yourself against. This analogy applies to engaging in decision hygiene to reduce noise—even if you’re not clear on what the source of the noise is, you’re still protecting against it.
One way to reduce noise is to aggregate a number of judgments rather than to pick one. In medicine, it’s common to ask for a second opinion—that’s a noise-reducing strategy. Sometimes hospitals ask five or six doctors and take the majority or the average. That’s a simple form of decision hygiene, and it tends to reduce noise.
Here’s another way to reduce noise: Ask, “What are the particular ingredients of the judgment? Can I ask about four or five ingredients separately, and evaluate along those dimensions?” So in deciding whether to hire someone, don’t ask, “Do I think the person should be hired?” Do ask, “Is the person diligent? Does the person have relevant experience? Is the person rated highly by recommenders? Is the person intelligent in the way that intelligence matters for the job?” If you look at what we call mediating assessments, you can reduce noise. You can reduce bias that way, too.
“In any system that relies on professional judgments, there may be a large opportunity to improve judgments and decisions by reducing noise.”
5. There are many ways to reduce noise, but first you need to measure it.
There are many ways to reduce noise, but first you need to measure it, as not all noise-reduction techniques are equally applicable to all situations. For instance, one approach that is often used is to implement algorithms instead of human judgment. Algorithms are noise-free, unlike humans; if they are fed the same inputs, they should produce the same outputs. This is a large part of the reason why they perform as well as they do. Indeed, algorithms already perform better than human judges at many tasks, like reading some types of X-rays.
But does this mean we can replace humans with algorithms everywhere? Of course not. Sometimes it’s simply not practical. Sometimes it might be too expensive. Sometimes it would also violate important expectations that we have about the process of judgment—for instance, that we should be given a fair hearing by a qualified human. And sometimes we may fear that the algorithms reduce noise, but add bias because the data on which they have been trained may itself be biased.
Algorithms work to eliminate noise in some places, but they cannot be used everywhere. The same is true of all noise-reduction techniques. There is no silver bullet, no single technique that will work to reduce noise in all settings. But just because we find that one noise-reduction technique is inappropriate in a particular situation does not mean that we have to give up noise-reduction completely.
If algorithms are not welcome, for instance, we may want to use guidelines that reduce the discretionary power of individuals without eliminating it completely. Or perhaps we might want to aggregate several independent judgments. Or maybe we want to encourage decision-makers to use the mediating assessments protocol to structure their judgments.
In short, in each situation, we will need to determine which noise-reduction approaches to use. It may not even be possible or not worth it to eliminate noise completely. But the first thing to do is always to measure how much noise there is. If we don’t find out, we remain under the illusion that all good professionals would agree with us on our judgments. In other words, we remain oblivious to the problem of noise and the injustice that it creates. In any system that relies on professional judgments, there may be a large opportunity to improve judgments and decisions by reducing noise. And the first step should be to measure it.
To listen to the audio version read by Olivier Sibony and Cass Sunstein, download the Next Big Idea App today:
