5.1 False Positives and Negatives

Considering the relative costs of each possible mistake helps us make better decisions under conditions of uncertainty, when we cannot eliminate the possibility of a mistake either way.

The Lesson in Context

This lesson gives students hands-on examples of false positives and negatives through a Jupyter notebook, in which they are asked to make moral judgments on the threshold for a positive test result based on the consequences of a false positive or negative.

2.2 Systematic and Statistical Uncertainty

• In an earlier lesson, students explored how statistical and systematic uncertainty can make data suggest false inferences, through either random variation within an insufficiently large sample (statistical uncertainty) or some systematic bias in the data (systematic uncertainty). Both of these sources of uncertainty can lead to either false positives or false negatives.

3.1 Probabilistic Reasoning

• In this lesson, students will learn to use numeric credence levels to track and communicate their confidence levels about various claims and predictions. Credence levels can be a good way to make explicit the likelihood of false positives and false negatives in a given case, and one's tolerance for the two types of errors.

4.2 Finding Patterns in Random Noise

• In the previous lesson, students explored how we often mistake noise for signal, because we are always looking for patterns through which to make sense of the world. This constant search for patterns can lead to false positives. On the other hand, sometimes there is so much noise that we miss real patterns, leading to false negatives.

5.2 Scientific Optimism

• "Scientific optimism" is a can-do attitude adopted by scientists by which they convince themselves that even difficult problems are solvable. With this "scientific optimism," scientists can successfully take on problems that take years or even decades to solve, with hundreds of steps and iterations involved in developing techniques, inventing technologies, collecting and analyzing data. Scientific optimism may also help scientists continue to detect false positives and false negatives in their previous work, iteratively improving our scientific understanding as we accumulate more evidence.

Takeaways

Learning GoalsDefinitionsExamplesCommon Misconceptions

After this lesson, students should

1. Understand that there is always the inevitability of a trade-off—for a given test, one can reduce the risk of false positives by increasing the risk of false negatives, and vice versa. You make this trade-off when you determine what the threshold is (e.g. convict someone if you're 99% or 70% certain that they're guilty).
2. Understand that whether false positive or negative is preferable depends on a human values judgement on the consequences of either: (e.g. "innocent people shouldn't be in jail") and risks (e.g dangerous people committing more crimes).

False Positive/Type I Errors

A test yields a positive result, but in fact the condition is not present (e.g. your test says you have COVID, but you don't).

False Negative/Type II Errors

A test yields a negative result, but in fact the condition is present (e.g. your test says you don't have COVID, but you do).

Threshold

The cut-off quantity chosen for an indicator variable to decide when a given instance will be supposed positive vs. negative.

Unless the indicator variable is a perfect indicator of the underlying condition, there will be some probability of false positives and/or false negatives for any given threshold. Choosing a threshold is thus partly a matter of deciding one's relative tolerance for false positives and false negatives; a higher threshold will lead to more false negatives, a lower threshold to more false positives.

Incoming Asteroid

"The oncoming asteroid has only a 1% chance of hitting Earth. But if it does, life on Earth will be destroyed. It'll be expensive to stop the asteroid, but the risk is bad enough it's worth it."

---

Cancer Treatment

"I asked them to imagine that they faced a choice between two types of radiation therapy for early-stage breast cancer. The first treatment would leave them with a 15% chance of local recurrence and a 10% chance of moderate or severe breast fibrosis. The second treatment would leave them with only an 8% chance of local recurrence but a 30% chance of moderate or severe fibrosis. The radiation oncologists raised their hands in almost equal numbers for the two treatments. Some believed the higher risk of fibrosis was unacceptable, given the treatability of most local recurrences, whereas others believed the trauma of recurrence outweighed the discomfort of fibrosis. But sometimes physicians' values differ in important ways from those of many patients. When such value judgments are incorporated into professional treatment guidelines, without any explicit acknowledgment that a reasonable patient might choose an alternative course of treatment, they take potential choices away from patients." (Source)

here have been many people who still caught the virus even after a "negative" test result. These tests are just unreliable.

Most medical tests will inevitably produce false positive or false negative results at a certain rate. Although no conclusion can be totally accurate, it is possible to quantify the rate of such errors, so that the test result can be used to assess the probabilistic risks and odds of having a disease. See 3.1 Probabilistic Reasoning.

Useful Resources

Lecture VideoDiscussion SlidesHandouts and Activities

Discussion Slides Template

The discussion slides for this lesson.

Data Science Notebook

Jupyter notebook on false positives and negatives.

Recommended Outline

Before Class

Go through the Jupyter notebook.

During Class

10 Minutes	Quickly review false positives and negatives.
40 Minutes	Have the students work through the data science module.
30 Minutes	Walk the students through the discussion questions.

Lesson Content

Warm-up Question

If you're more worried about false positives than false negatives (e.g. for judging a defendant as guilty, it's worse to sentence someone innocent than let someone guilty go free), that means you should:

1. Lower your threshold for counting an instance as a positive
2. Raise your threshold for counting an instance as a positive
3. Ignore your threshold for counting an instance as a positive
4. Increase your signal-to-noise ratio

Explanation

This will decrease false positives (although it will also increase false negatives).

Data Science Module

Have the students work through the data science notebook. This should take around 40 minutes.

The Jupyter notebook itself should be self explanatory and contains several discussion questions. We suggest pairing the students for this activity. If necessary, the instructor may demonstrate how the notebook works to the class.

Data Science Module Questions

These are the questions included in the data science notebook.

Part 1: H. Pylori

You are the designer of the H. pylori test. Your job is to decide on a value for its threshold, which will be used by all clinical laboratories to make diagnoses. What threshold value do you choose?

Any value between 2 and 21.

Part 2: Pregnancy Test

Question 2a

What should the sensitivity level of a home pregnancy test be in order to ensure (1) no false positives and (2) less than 15% false negatives on the 9th day of pregnancy?

Lots of possible values. For example, with a threshold of 0.18, you can ensure no false positives and a 7.65% false negative rate.

Question 2b

What should the sensitivity of a home pregnancy test be in order to ensure (1) no false positives and (2) less than 15% false negatives on the 11th day of pregnancy?

Lots of possible values. For example, with a threshold of 1.11, you can ensure no false positives and a 0.45% false negative rate.

Part 3: Smoke Detector

Question 3a

In the context of the smoke detector's alarm, what scenarios would be considered a true positive, a false positive, a true negative, or a false negative?

True Positive: Alarm goes off, there's a fire.
True Negative: Alarm doesn't go off, there isn't a fire.
False Positive: Alarm goes off, there isn't a fire.
False Negative: Alarm goes off, there is a fire.

Question 3b

For each of these scenarios, what is the consequence to the inhabitants of the house?

True Positive: Lives saved.
True Negative: Everything is calm and no one is bothered.
False Positive: Alarm is piercingly loud, disrupting a totally normal meal preparation. "Crying wolf" makes people less likely to believe the alarm when it goes off.
False Negative: House is burnt down, hurting the unknowing inhabitants.

Question 3c

With reference to these consequences, what criteria would you be considering when setting a threshold value for a smoke detector?

You might want to lower the threshold to minimize false negatives (very severe bad outcome), but not all the way to zero because false positives would be very annoying.

Question 3d

Choose a value that best satisfies the criteria you set forth above. Explain whether you are satisfied with your choice.

One possible value is 0.32, with 6.1% false positive and 5.09% false negative rates. This allows some false positives because it is more important to save lives, but it doesn't push the threshold so low as to make the alarm too much less believable in the long run.

Question 3e

You have no doubt experienced an annoying smoke detector that has gone off, but nothing has burnt down, perhaps even prompting a pointless trip by the fire brigade. Why do situations like this keep happening?

Alarm manufacturers made a conscious choice to allow some false positives, so that more true fires can be reported. This decision potentially saves more lives.

Part 4: SAT Scores vs. Admission

Question 4a

With reference to the above graph, is there a correlation between ACT score and academic performance in the first year? Would you say that ACT score is a good admissions criterion?

Question 4b

In the context of admission, what is considered a "positive diagnosis" or a "negative diagnosis?"

Question 4c

With reference to the administration's demands, what scenarios would be considered a true positive, a false positive, a true negative, or a false negative?

Question 4d

For each of these scenarios, what are its consequences?

Question 4e

What ACT score would you set as the cut-off for admission? Explain your reasoning for your chosen threshold, specifying why it isn't higher or lower.

1. Test Score Questions
   1. Is there a correlation between ACT score and academic performance in the first year? Would you say that ACT score is a good admissions criterion?
   2. In the context of admission, what is considered a "positive diagnosis" or a "negative diagnosis?"
   3. With reference to the administration's demands, what scenarios would be considered a true positive, a false positive, a true negative, or a false negative?
   4. For each of these scenarios, what are its consequences?
   5. What ACT score would you set as the cut-off for admission? Why isn't it higher or lower?
   6. Considering individual students, why would such an admissions rule be problematic?
   7. With reference to the ACT Scores histogram and the false positive and false negative rates, would you say that ACT scores are a good admissions criterion?
2. Spam Filter Questions
   1. In the context of a spam/ham filter, what scenarios would be considered a true positive, a false positive, a true negative, or a false negative?
   2. For each of these scenarios, what is the consequence to the use of your email service?
   3. With reference to these consequences, what criteria would you be considering when setting a threshold value for a spam/ham filter?
   4. Choose a threshold value. With reference to the above criteria, why pick that one?
   5. Think about the ideas of true/false positive/negative. Who ultimately decides which email has been correctly classified as spam or not?
   6. Is the definition of "spam" something "real," something decided by convention, some other type of model?
   7. Suppose an email provider touts "BEST SPAM FILTER IN TOWN. 99.9% ACCURACY!" Why should you be skeptical of their claim?

Class Discussion

Instructions

For each question, let students discuss in small groups, and then let them share their thoughts with the whole class.

10 Minutes	Discuss question 1.
12 Minutes	Discuss question 2. Spend about half the time on each part.
8 Minutes	Discuss question 3.

Discussion Questions

Question 1

In 1983, the Soviet Union's nuclear early-warning radar reported five intercontinental ballistic missiles incoming from the United States. Rather than immediately sending this information up the chain-of-command, the officer on duty decided to wait for additional evidence before taking action. No further evidence came, and it ultimately turned out that the radar system had malfunctioned.

What would be the costs (both immediate and long-term) of a false positive and false negative in this scenario?

Question 2

Suppose you have very bad migraines. Your doctors tell you migraines can be sparked by a lot of different things, which vary for different people: most kinds of food, coffee, stress, and/or bright sunlight. You want to find out what is causing your migraines, so you cut your diet to only potatoes and then slowly add one food back at a time, waiting a few days between each addition to see if you get a migraine.

The whole thing is stressful, and you're so hungry and grumpy that you lose your sunglasses on Monday. On Thursday, you eat chocolate again for the first time. That afternoon, you get a relatively small migraine. You are very sad, because chocolate is your very favorite food.

What are the different consequences of false positives and negatives in the two following scenarios.

1. Should you wait for a few days and try eating a small amount of chocolate again, or should you ban it from your cupboards? Why?
2. What if your migraine came with sudden, intense, lasting vertigo, and you work in skyscraper construction and can't take off work?

Answers

1. The cost of slight migraines from the piece of chocolate is small, yet the benefit of that delicious morsel of sweet and creamy chocolate is so great. So, it's worth giving it a second chance and seeing what happens.
2. The cost of migraines is now way higher as you could either lose your job or die. The benefit of the chocolate is the same. Maybe don't give the chocolate a second chance in this case.

Question 3

Recount an instance in which you thought you made a right diagnosis of a situation, but then found out that you were wrong. (Because you're a human.)

1. Explain whether it was an instance of a false positive or a false negative.
2. What makes it a false positive/negative as compared to a false negative/positive?
3. Which type of error do you think would have been better to make in this situation, and why?

---

⟨

Previous

4.2 Finding Patterns in Random Noise

⟩

Next

5.2 Scientific Optimism