A colleague recently recommended Noise: A Flaw in Human Judgment to me. It is about a topic I’ve studied for the past 25 years (statistical process variation). It is also co-authored by one of my favorite Professors of Behavioral Economics, Daniel Kahneman.
The book was released in 2021 and shares some good reasons to care about process variation but seems to be written for a time before Six Sigma Black Belts (like we have on the Lean East team) existed. The authors appear to be “academics” unaware that companies figured out years ago the benefits of controlling and minimizing process variation.
With the above in mind, I’m sharing some key points made in the book. However, please realize that most Lean Six Sigma Black Belts have known and practiced these ideas for many years!
Wherever There is Judgement, There is Noise
A main theme in the book is that decision-makers are human and therefore susceptible to bias.
Bias (several common ones are shared in this book) exists when most errors in a set of judgments are in the same direction. Examples include a judge who believes older people (like him or her) to be wiser and more experienced, executives who are too optimistic about sales year after year, or a gambler who keeps believing his luck will turn around soon.
Noise is the error that remains when bias is removed. System noise is observed in organizations that employ interchangeable professionals to make decisions. For example, physicians in an emergency room, underwriters in an insurance company, teachers in the classroom, or judges in a courtroom.
Errors of Judgement
Judgment error is due to both bias and noise. The authors expected to find both of these errors in their studies but were surprised at the extent of the problems caused by noise in many systems and processes. The authors separated this system noise into level noise and pattern noise.
Level noise is the variability of the average judgments made by different individuals. For example, some judges are harsher or more lenient than their peers.
A larger component of system noise is based on how judges rank a case (rather than whether they are harsh or lenient). Most of this pattern noise is the stable difference in personal rankings. For example, one judge might be lenient on minor traffic offenses while another whose sibling was killed in a car accident might be particularly harsh. My own son went to the DMV last year for his driving test and was passed by the evaluator with a reputation for being easy. The “hard” evaluator failed my son’s friend the same day.
Some pattern noise is a less stable difference in personal rankings. This occasion noise is often based on the mood of a judge. Years ago I worked on Sunday afternoons grading assignments as a teaching assistant. I joked at the time that how I graded often depended on how my favorite football team played in their game!
Issues with Noise
We often don’t account for the noise in our lives. We assume judgments by physicians, teachers, interviewers, and the like are consistent and dependable and consistent. Only a data-driven view of the world enables us to see the noise but many of us prefer to trust our own judgment.
Data that measures the statistical variation (due to both bias and noise) will help identify ways to reduce the process variation. Unfortunately, this variation data embarrasses both the individual judges and their organizations. Expert evaluators don’t want to have their lack of consistency published for all to see!
How to Reduce Noise (and Bias)
A variation reduction project should always begin with a “noise audit.” This provides a baseline measure of the issue and can show the opportunity available. Lean Six Sigma Black Belts often use a Gauge Repeatability and Reproducibility (GR&R) study to measure baseline noise. Feel free to contact us to learn more about these studies.
The greek letter sigma (σ) refers to the measure of process or system variation. A six-sigma process provides a quality output more than 99.99966% of the time. A Lean Six-Sigma Black Belt seeks to improve proesses to >6σ levels.
Kahneman and his co-authors share six principles to reduce system variation:
- The goal of judgment is accuracy, not individual expression. The authors recommend replacing individual judgment with rules or algorithms, whenever possible.
- Think statistically, and take the outside view of the case. Decision makers should compare the case at hand with past judgments as a reference for how to judge their case.
- Struture judgments into several independent tasks. One way to do this is by scoring from a checklist, such as the Apgar score.
- Resist premature intuitions. Blind studies or sequencing the information and only sharing information with professionals as they need it are two effective techniques.
- Obtain independent judgments from multiple judges, then consider aggregating these judgements. An example of this is an interiew panel that ranks candidates independently before comparing reviews.
- Favor relative judgments and relative scales. Relative judgments are less noisy because people tend to be better at making pairwise comparisons.
Contact Lean East for More Information
We hope you enjoyed this book summary of Noise: A Flaw in Human Judgment. We aren’t blogging much on this site anymore. If you are missing our lessons, please visit our new Personal Kaizen blog.
