Noise

Recently I wrote a blog about Informational Cascades, after having read about it in the book Noise by Daniel Kahneman et al. Actually, I should have written about the main theme of the book in the first place, of course: Noise. According to the authors noise is a much ignored and neglected problem in research and in many daily practices when decisions must be taken. Noise must be distinguished from Bias, so the authors. Unlike noise, they say, bias does get much attention in decision practices. The reason is that bias is relatively easy to establish and to avoid, if you invest some time in doing so, while noise is often difficult to establish, and once you know that it is there, it is difficult to get a grip on it.
When reading the book I got the impression that it is a rather recently discovered problem, though the authors do not say so. Anyway, I remember that already when I studied sociology around 1970, my methodology books paid attention to the problem of noise and bias. However, then they had other names. Then it was called the problem of reliability and validity of measuring instruments (since we are talking about sociology, you must think of 5-point or 7-point measuring scales, questionnaires and the like, but in fact the problem applies to any measurement). A measuring instrument is called valid if it measures what it is supposed to measure, and it is called reliable if it gives the same result each time when it is used to measure the same object.
This brings us to the heart of the noise-bias distinction, for a reliable measuring instrument is noise-free and if it is a valid instrument it is bias-free. Preferably it is both. Instead of further clarifying the valid-reliable distinction, I’ll continue discussing the bias-noise distinction, since in fact it is the same.
Kahneman et al. give a clear example of the distinction and once you understand it, you know what bias and noise are and the rest is “only” the problem how to avoid them. Four teams of four archers go to an archery range. Each person does one shot. Ideally, each archer should hit the bull’s eye. These are the results for each team (for practical reasons I copied the figure from the Wikipedia):

As you can see in the figure, all members of team-a hit the bull’s eye. The result was accurate. The result of team-b was also accurate in a sense, because all team members hit the same point but the shots are systematically off the target. Therefore, we call the result biased. The results of team-c are all around the bull’s eye, but they are scattered around the target. Therefore, we call the team result noisy. The result of team-d is both systematically off the target and it is scattered around the wrong target point. Therefore, we call this team result biased and noisy at the same time. I think that this illustration makes clear what the essence of bias and noise is, especially if we summarize the concepts this way: Bias is systematic deviation, while noise is random scatter. The difference between bias and noise becomes especially clear, if we look at the back side of the targets the archers were shooting at (see the figure below):

If you didn’t know where the bull’s eye of the target is, you couldn’t know whether team-a or team-b was closer to the bull’s eye, but it will be clear to you that both the results of team-c and team-d are noisy. As Kahneman et al. say: “A general property of noise is that you can recognize and measure it while knowing nothing about the target of the bias.” Or in terms of validity and reliability: We don’t know whether the results of teams a and b are valid, but we do know that the results of teams c and d are not reliable. Once you know what the problem is, the question is how to solve it, which often is far from easy. However, this must not be a reason to ignore the problem, for it’s really important as some examples by Kahneman et al. show:
- Faced with the same patient different doctors often make different judgements.
- Case managers in child protection agencies often make mistakes in judging whether a child must be placed in foster care since there is a risk of abuse. Some managers are more likely to send a child to foster care than others. However, children wrongly assigned to foster care have poorer life outcomes, like delinquency rates.
- Weather forecasts are known to be noisy.
- Interviewers of job candidates make widely different assessments of the same people. Moreover, performance ratings of the employees depend more on the person doing the assessment than on the performance being assessed.

Source
Daniel Kahneman, Oliver Sibony, Cass R. Sunstein, Noise. A Flaw in Human Judgment. London: William Collins, 2021; pp. 3-7.