5.1 False Positives and Negatives

Useful Links

• Jupyter Notebook
• Three Column Overview of the Week
• Lesson Slides (2023 Master)
• Website Page

Readings and Assignments

• PlayPosit Video

Learning Goals

After this lesson, students should

1. Understand that there is always the inevitability of a trade-off—for a given test, one can reduce the risk of false positives by increasing the risk of false negatives, and vice versa. You make this trade-off when you determine what the threshold is (e.g. convict someone if you're 99% or 70% certain that they're guilty).
2. Understand that whether false positive or negative is preferable depends on a human values judgement on the consequences of either: (e.g. "innocent people shouldn't be in jail") and risks (e.g dangerous people committing more crimes).

Definitions

• False Positive/Type I Errors

  A test yields a positive result, but in fact the condition is not present (e.g. your test says you have COVID, but you don't).

• False Negative/Type II Errors

  A test yields a negative result, but in fact the condition is present (e.g. your test says you don't have COVID, but you do).

• Threshold

  The cut-off quantity chosen for an indicator variable to decide when a given instance will be supposed positive vs. negative.

Examples

• Incoming Asteroid: "The oncoming asteroid has only a 1% chance of hitting Earth. But if it does, life on Earth will be destroyed. It'll be expensive to stop the asteroid, but the risk is bad enough it's worth it."
• Cancer Treatment: "I asked them to imagine that they faced a choice between two types of radiation therapy for early-stage breast cancer. The first treatment would leave them with a 15% chance of local recurrence and a 10% chance of moderate or severe breast fibrosis. The second treatment would leave them with only an 8% chance of local recurrence but a 30% chance of moderate or severe fibrosis. The radiation oncologists raised their hands in almost equal numbers for the two treatments. Some believed the higher risk of fibrosis was unacceptable, given the treatability of most local recurrences, whereas others believed the trauma of recurrence outweighed the discomfort of fibrosis. But sometimes physicians' values differ in important ways from those of many patients. When such value judgments are incorporated into professional treatment guidelines, without any explicit acknowledgment that a reasonable patient might choose an alternative course of treatment, they take potential choices away from patients." source

Common Misconceptions

• There have been many people who still caught the virus even after a "negative" test result. These tests are just unreliable.

  Most medical tests will inevitably produce false positive or false negative results at a certain rate. Although no conclusion can be totally accurate, it is possible to quantify the rate of such errors, so that the test result can be used to assess the probabilistic risks and odds of having a disease. See 3.1 Probabilistic Reasoning.

Context

This lesson gives students hands-on examples of false positives and negatives through a Jupyter notebook, in which they are asked to make moral judgments on the threshold for a positive test result based on the consequences of a false positive or negative.

Before

2.2 Systematic and Statistical Uncertainty

In an earlier lesson, students explored how statistical and systematic uncertainty can make data suggest false inferences, through either random variation within an insufficiently large sample (statistical uncertainty) or some systematic bias in the data (systematic uncertainty). Both of these sources of uncertainty can lead to either false positives or false negatives.

3.1 Probabilistic Reasoning

In this lesson, students will learn to use numeric credence levels to track and communicate their confidence levels about various claims and predictions. Credence levels can be a good way to make explicit the likelihood of false positives and false negatives in a given case, and one's tolerance for the two types of errors.

4.2 Finding Patterns in Random Noise

In the previous lesson, students explored how we often mistake noise for signal, because we are always looking for patterns through which to make sense of the world. This constant search for patterns can lead to false positives. On the other hand, sometimes there is so much noise that we miss real patterns, leading to false negatives.

After

5.2 Scientific Optimism

"Scientific optimism" is a can-do attitude adopted by scientists by which they convince themselves that even difficult problems are solvable. With this "scientific optimism," scientists can successfully take on problems that take years or even decades to solve, with hundreds of steps and iterations involved in developing techniques, inventing technologies, collecting and analyzing data. Scientific optimism may also help scientists continue to detect false positives and false negatives in their previous work, iteratively improving our scientific understanding as we accumulate more evidence.

Recommended Outline

Before Class

• Review PlayPosit and discussion questions and ask faculty, Gabriel, or Emlen any questions you have.
• Go through the Jupyter notebook.

During Class

• (5 min) Come up with some fun way to assign the roles of spokesperson and notetaker (e.g. earliest birthday in the year, lives furthest from campus). Remind them of the responsibilities of these roles.
• (5 min) Quick review of false positives and false negatives.
• (40 min) Data science module.
• (30 min) Discussion questions.

Lesson Content

Warm-up Question

1. If you're more worried about false positives than false negatives (e.g. for judging a defendant as guilty, it's worse to sentence someone innocent than let someone guilty go free), that means you should:
   1. Lower your threshold for counting an instance as a positive
   2. Raise your threshold for counting an instance as a positive
   3. Ignore your threshold for counting an instance as a positive
   4. Increase your signal-to-noise ratio

Data Science Module

(40 minutes)

Have the students work through the data science notebook.

Data Science Module Questions

1. H. Pylori Questions
   1. You are the designer of the H. pylori test. Your job is to decide on a value for its threshold, which will be used by all clinical laboratories to make diagnoses. What threshold value do you choose?
2. Pregnancy Test Questions
   1. What should the sensitivity level of a home pregnancy test be in order to ensure (1) no false positives and (2) less than 15% false negatives on the 9th day of pregnancy?
   2. What should the sensitivity of a home pregnancy test be in order to ensure (1) no false positives and (2) less than 15% false negatives on the 11th day of pregnancy?
3. Smoke Detector Questions
   1. In the context of the smoke detector's alarm, what scenarios would be considered a true positive, a false positive, a true negative, or a false negative?
   2. For each of these scenarios, what is the consequence to the inhabitants of the house?
   3. With reference to these consequences, what criteria would you be considering when setting a threshold value for a smoke detector?
4. Test Score Questions
   1. Is there a correlation between ACT score and academic performance in the first year? Would you say that ACT score is a good admissions criterion?
   2. In the context of admission, what is considered a “positive diagnosis” or a “negative diagnosis?”
   3. With reference to the administration's demands, what scenarios would be considered a true positive, a false positive, a true negative, or a false negative?
   4. For each of these scenarios, what are its consequences?
   5. What ACT score would you set as the cut-off for admission? Why isn't it higher or lower?
   6. Considering individual students, why would such an admissions rule be problematic?
   7. With reference to the ACT Scores histogram and the false positive and false negative rates, would you say that ACT scores are a good admissions criterion?
5. Spam Filter Questions
   1. In the context of a spam/ham filter, what scenarios would be considered a true positive, a false positive, a true negative, or a false negative?
   2. For each of these scenarios, what is the consequence to the use of your email service?
   3. With reference to these consequences, what criteria would you be considering when setting a threshold value for a spam/ham filter?
   4. Choose a threshold value. With reference to the above criteria, why pick that one?
   5. Think about the ideas of true/false positive/negative. Who ultimately decides which email has been correctly classified as spam or not?
   6. Is the definition of “spam” something “real,” something decided by convention, some other type of model?
   7. Suppose an email provider touts “BEST SPAM FILTER IN TOWN. 99.9% ACCURACY!” Why should you be skeptical of their claim?

Discussion Questions

For each question, let students discuss in small groups, and then let them share their thoughts with the whole class.

1. (10 min) In 1983, the Soviet Union's nuclear early-warning radar reported five intercontinental ballistic missiles incoming from the United States. Rather than immediately sending this information up the chain-of-command, the officer on duty decided to wait for additional evidence before taking action. No further evidence came, and it ultimately turned out that the radar system had malfunctioned.
   What would be the costs (both immediate and long-term) of a false positive and false negative in this scenario?
2. (15 min) Suppose you have very bad migraines. Your doctors tell you migraines can be sparked by a lot of different things, which vary for different people: most kinds of food, coffee, stress, and/or bright sunlight. You want to find out what is causing your migraines, so you cut your diet to only potatoes and then slowly add one food back at a time, waiting a few days between each addition to see if you get a migraine.
   The whole thing is stressful, and you're so hungry and grumpy that you lose your sunglasses on Monday. On Thursday, you eat chocolate again for the first time. That afternoon, you get a relatively small migraine. You are very sad, because chocolate is your very favorite food.
   What are the different consequences of false positives and negatives in the two following scenarios:
   1. (7 min) Should you wait for a few days and try eating a small amount of chocolate again, or should you ban it from your cupboards? Why?
   2. (6 min) What if your migraine came with sudden, intense, lasting vertigo, and you work in skyscraper construction and can't take off work?
3. (7 min) Recount an instance in which you thought you made a right diagnosis of a situation, but then found out that you were wrong. (Because you're a human.)
   1. Explain whether it was an instance of a false positive or a false negative.
   2. What makes it a false positive/negative as compared to a false negative/positive?
   3. Which type of error do you think would have been better to make in this situation, and why?

Overflow

Discussion Questions

1. (10 min) In the Stanley Kubrick film Dr Strangelove, which takes place during the Cold War, a rogue American general sends the code to send planes to drop nuclear bombs on Russia because he wants to start a nuclear war. One of the pilots believes that he could only have received such instructions if Russia had struck first. Under these conditions (leaving aside the Doomsday Machine), what are the costs of a false positive (believing Russia struck first when it didn't) and what are the costs of a false negative (believing it is some kind of mistake and not dropping bombs, when in fact Russia did strike first)?