5.1 False Positives and Negatives

How confident should we be about an apparent signal before we make decisions based upon it? What if we are wrong? These are questions faced by every medical patient, police officer, and president. The answer depends not only on the facts about the signal, but also on the perceived severity of the outcome in the event of an error—a signal falsely identified or inadvertently missed.

The Lesson in Context

This lesson gives students hands-on examples of false positives and negatives through a Jupyter notebook, in which they are asked to make moral judgments on the threshold for a positive test result based on the consequences of a false positive or negative.

Earlier Lessons

2.2 Systematic and Statistical Uncertainty

In an earlier lesson, students explored how statistical and systematic uncertainty can make data suggest false inferences, through either random variation within an insufficiently large sample (statistical uncertainty) or some systematic bias in the data (systematic uncertainty). Both of these sources of uncertainty can lead to either false positives or false negatives.

3.1 Probabilistic Reasoning

In this lesson, students will learn to use numeric credence levels to track and communicate their confidence levels about various claims and predictions. Credence levels can be a good way to make explicit the likelihood of false positives and false negatives in a given case, and one's tolerance for the two types of errors.

4.2 Finding Patterns in Random Noise

In the previous lesson, students explored how we often mistake noise for signal, because we are always looking for patterns through which to make sense of the world. This constant search for patterns can lead to false positives. On the other hand, sometimes there is so much noise that we miss real patterns, leading to false negatives.

Later Lessons

5.2 Scientific Optimism

"Scientific optimism" is a can-do attitude adopted by scientists by which they convince themselves that even difficult problems are solvable. With this "scientific optimism," scientists can successfully take on problems that take years or even decades to solve, with hundreds of steps and iterations involved in developing techniques, inventing technologies, collecting and analyzing data. Scientific optimism may also help scientists continue to detect false positives and false negatives in their previous work, iteratively improving our scientific understanding as we accumulate more evidence.

Takeaways

After this lesson, students should

Understand that there is always the inevitability of a trade-off—for a given test, one can reduce the risk of false positives by increasing the risk of false negatives, and vice versa. You make this trade-off when you determine what the threshold is (e.g. convict someone if you're 99% or 70% certain that they're guilty).
Understand that whether false positive or negative is preferable depends on a human values judgement on the consequences of either: (e.g. "innocent people shouldn't be in jail") and risks (e.g. dangerous people committing more crimes).
(Data science) Recognise that maximising the overall accuracy of a classification problem may sometimes be undesirable.
(Data science) Identify where a subtle classification problem may be hidden in a machine learning application.

False Positive/Type I Errors

A test yields a positive result, but in fact the condition is not present (e.g. your test says you have COVID, but you don't).

False Negative/Type II Errors

A test yields a negative result, but in fact the condition is present (e.g. your test says you don't have COVID, but you do).

Threshold

The cut-off quantity chosen for an indicator variable to decide when a given instance will be supposed positive vs. negative.

Unless the indicator variable is a perfect indicator of the underlying condition, there will be some probability of false positives and/or false negatives for any given threshold. Choosing a threshold is thus partly a matter of deciding one's relative tolerance for false positives and false negatives; a higher threshold will lead to more false negatives, a lower threshold to more false positives.

Incoming Asteroid

"The oncoming asteroid has only a 1% chance of hitting Earth. But if it does, life on Earth will be destroyed. It'll be expensive to stop the asteroid, but the risk is bad enough it's worth it."

Cancer Treatment

"I asked them to imagine that they faced a choice between two types of radiation therapy for early-stage breast cancer. The first treatment would leave them with a 15% chance of local recurrence and a 10% chance of moderate or severe breast fibrosis. The second treatment would leave them with only an 8% chance of local recurrence but a 30% chance of moderate or severe fibrosis. The radiation oncologists raised their hands in almost equal numbers for the two treatments. Some believed the higher risk of fibrosis was unacceptable, given the treatability of most local recurrences, whereas others believed the trauma of recurrence outweighed the discomfort of fibrosis. But sometimes physicians' values differ in important ways from those of many patients. When such value judgments are incorporated into professional treatment guidelines, without any explicit acknowledgment that a reasonable patient might choose an alternative course of treatment, they take potential choices away from patients."

NEJM Perspective on Treatment Guidelines and Patient Values

Article on how value judgments are embedded in professional treatment guidelines.

Exemplary Quotes

“We should do what we can about climate change. If we're wrong, and anthropogenic activity is not going to disrupt the climate too much in the end, then putting a lot of money into green energy is bad but not catastrophic. On the other hand, the risks of just letting it happen could be catastrophic. Even if there's only a 25% chance that humans can mitigate climate change significantly, that's worth the trouble.”

“It's true that sometimes seatbelts cause deaths, when people get stuck in them and can't get out. But they more often save lives, so it is prudent to wear your seatbelt whenever you drive.”

“Better that ten guilty persons escape than that one innocent suffer”

Sir William Blackstone (1765)

“An Iowa couple who welcomed their first baby in July had been preparing to say goodbye to him since about halfway through their pregnancy after doctors diagnosed the unborn child with a terminal brain malformation. However, hours after Drew and Ariann Corpstein's son was born, they learned that he had been misdiagnosed, and that his condition could be managed.”

Source

“In 2009-2011 Toyota recalled millions of vehicles because of a perceived issue of the floor mat catching the accelerator. While Toyota was unsure if it was driver error or the floor mat causing uncontrollable acceleration, they still issued the recall. While driver error was much more likely, the cost of blaming driver error and not issuing a recall for potentially faulty floor mats, customer deaths and inevitable lawsuits, was too high.”

Source

“Source”

There have been many people who still caught the virus even after a "negative" test result. These tests are just unreliable.

Most medical tests will inevitably produce false positive or false negative results at a certain rate. Although no conclusion can be totally accurate, it is possible to quantify the rate of such errors, so that the test result can be used to assess the probabilistic risks and odds of having a disease. See 3.1 Probabilistic Reasoning.

After this lesson, students should

Attitudes
1. Given some degree of uncertainty, appreciate that different kinds of errors come with different costs, such that in some cases it is worthwhile to presume the less likely alternative because the error you risk is less costly.
Concept Acquisition
1. False Positive/Type I Errors: A test yields a positive result, but in fact the condition is not present.
2. False Negative/Type II Errors: A test yields a negative result, but in fact the condition is present.
3. There is always the possibility of a trade-off—for a given test, one can reduce the risk of false positives by increasing the risk of false negatives, and vice versa.
4. Good decision-making under uncertainty involves having sufficient signal (an adequate test) and setting your threshold appropriately for the relative costs of false positives and false negatives.
Concept Application
1. Identify false positive (type I) and false negative (type II) errors in scientific and everyday situations.
2. Weigh the costs associated with false positives/negatives with the benefits associated with true positives/negatives when making a decision under uncertain conditions.
3. Explain how people could come to different decisions or policies as a result of different utilities/values associated with different types of errors, even if they agree about the relevant facts.

Additional Content

You must be logged in to see this content.