5.1 False Positives and Negatives
More actions
Considering the relative costs of each possible mistake helps us make better decisions under conditions of uncertainty, when we cannot eliminate the possibility of a mistake either way.
The Lesson in Context
This lesson gives students hands-on examples of false positives and negatives through a Jupyter notebook, in which they are asked to make moral judgments on the threshold for a positive test result based on the consequences of a false positive or negative.
- In an earlier lesson, students explored how statistical and systematic uncertainty can make data suggest false inferences, through either random variation within an insufficiently large sample (statistical uncertainty) or some systematic bias in the data (systematic uncertainty). Both of these sources of uncertainty can lead to either false positives or false negatives.
- In this lesson, students will learn to use numeric credence levels to track and communicate their confidence levels about various claims and predictions. Credence levels can be a good way to make explicit the likelihood of false positives and false negatives in a given case, and one's tolerance for the two types of errors.
- In the previous lesson, students explored how we often mistake noise for signal, because we are always looking for patterns through which to make sense of the world. This constant search for patterns can lead to false positives. On the other hand, sometimes there is so much noise that we miss real patterns, leading to false negatives.
- "Scientific optimism" is a can-do attitude adopted by scientists by which they convince themselves that even difficult problems are solvable. With this "scientific optimism," scientists can successfully take on problems that take years or even decades to solve, with hundreds of steps and iterations involved in developing techniques, inventing technologies, collecting and analyzing data. Scientific optimism may also help scientists continue to detect false positives and false negatives in their previous work, iteratively improving our scientific understanding as we accumulate more evidence.
Takeaways
After this lesson, students should
- Understand that there is always the inevitability of a trade-off—for a given test, one can reduce the risk of false positives by increasing the risk of false negatives, and vice versa. You make this trade-off when you determine what the threshold is (e.g. convict someone if you're 99% or 70% certain that they're guilty).
- Understand that whether false positive or negative is preferable depends on a human values judgement on the consequences of either: (e.g. "innocent people shouldn't be in jail") and risks (e.g dangerous people committing more crimes).
False Positive/Type I Errors
False Negative/Type II Errors
Threshold
Unless the indicator variable is a perfect indicator of the underlying condition, there will be some probability of false positives and/or false negatives for any given threshold. Choosing a threshold is thus partly a matter of deciding one's relative tolerance for false positives and false negatives; a higher threshold will lead to more false negatives, a lower threshold to more false positives.
Incoming Asteroid
- "The oncoming asteroid has only a 1% chance of hitting Earth. But if it does, life on Earth will be destroyed. It'll be expensive to stop the asteroid, but the risk is bad enough it's worth it."
Cancer Treatment
- "I asked them to imagine that they faced a choice between two types of radiation therapy for early-stage breast cancer. The first treatment would leave them with a 15% chance of local recurrence and a 10% chance of moderate or severe breast fibrosis. The second treatment would leave them with only an 8% chance of local recurrence but a 30% chance of moderate or severe fibrosis. The radiation oncologists raised their hands in almost equal numbers for the two treatments. Some believed the higher risk of fibrosis was unacceptable, given the treatability of most local recurrences, whereas others believed the trauma of recurrence outweighed the discomfort of fibrosis. But sometimes physicians' values differ in important ways from those of many patients. When such value judgments are incorporated into professional treatment guidelines, without any explicit acknowledgment that a reasonable patient might choose an alternative course of treatment, they take potential choices away from patients." (Source)
here have been many people who still caught the virus even after a "negative" test result. These tests are just unreliable.
Useful Resources
Recommended Outline
Before Class
Go through the Jupyter notebook.
During Class
| 10 Minutes | Quickly review false positives and negatives. |
| 40 Minutes | Have the students work through the data science module. |
| 30 Minutes | Walk the students through the discussion questions. |
Lesson Content
Warm-up Question
If you're more worried about false positives than false negatives (e.g. for judging a defendant as guilty, it's worse to sentence someone innocent than let someone guilty go free), that means you should:
- Lower your threshold for counting an instance as a positive
- Raise your threshold for counting an instance as a positive
- Ignore your threshold for counting an instance as a positive
- Increase your signal-to-noise ratio
Explanation
This will decrease false positives (although it will also increase false negatives).
Data Science Module
Have the students work through the data science notebook. This should take around 40 minutes.
The Jupyter notebook itself should be self explanatory and contains several discussion questions. We suggest pairing the students for this activity. If necessary, the instructor may demonstrate how the notebook works to the class.
Data Science Module Questions
These are the questions included in the data science notebook.
Part 1: H. Pylori
You are the designer of the H. pylori test. Your job is to decide on a value for its threshold, which will be used by all clinical laboratories to make diagnoses. What threshold value do you choose?
Any value between 2 and 21.
Part 2: Pregnancy Test
Question 2a
What should the sensitivity level of a home pregnancy test be in order to ensure (1) no false positives and (2) less than 15% false negatives on the 9th day of pregnancy?
Lots of possible values. For example, with a threshold of 0.18, you can ensure no false positives and a 7.65% false negative rate.
Question 2b
What should the sensitivity of a home pregnancy test be in order to ensure (1) no false positives and (2) less than 15% false negatives on the 11th day of pregnancy?
Lots of possible values. For example, with a threshold of 1.11, you can ensure no false positives and a 0.45% false negative rate.
Part 3: Smoke Detector
Question 3a
In the context of the smoke detector's alarm, what scenarios would be considered a true positive, a false positive, a true negative, or a false negative?
True Positive: Alarm goes off, there's a fire.
True Negative: Alarm doesn't go off, there isn't a fire.
False Positive: Alarm goes off, there isn't a fire.
False Negative: Alarm goes off, there is a fire.
Question 3b
For each of these scenarios, what is the consequence to the inhabitants of the house?
Example Answers
True Positive: Lives saved.
True Negative: Everything is calm and no one is bothered.
False Positive: Alarm is piercingly loud, disrupting a totally normal meal preparation. "Crying wolf" makes people less likely to believe the alarm when it goes off.
False Negative: House is burnt down, hurting the unknowing inhabitants.
Question 3c
With reference to these consequences, what criteria would you be considering when setting a threshold value for a smoke detector?
You might want to lower the threshold to minimize false negatives (very severe bad outcome), but not all the way to zero because false positives would be very annoying.
Question 3d
Choose a value that best satisfies the criteria you set forth above. Explain whether you are satisfied with your choice.
One possible value is 0.32, with 6.1% false positive and 5.09% false negative rates. This allows some false positives because it is more important to save lives, but it doesn't push the threshold so low as to make the alarm too much less believable in the long run.
Question 3e
You have no doubt experienced an annoying smoke detector that has gone off, but nothing has burnt down, perhaps even prompting a pointless trip by the fire brigade. Why do situations like this keep happening?
Alarm manufacturers made a conscious choice to allow some false positives, so that more true fires can be reported. This decision potentially saves more lives.
Part 4: SAT Scores vs. Admission
Question 4a
With reference to the above graph, is there a correlation between ACT score and academic performance in the first year? Would you say that ACT score is a good admissions criterion?
Yes. The higher ACT score, the better the student tends to perform in college. ACT seems like a good indicator of academic success according to this graph.
Question 4b
In the context of admission, what is considered a "positive diagnosis" or a "negative diagnosis?"
Positive: Student performs well (for example, their GPA is greater than 3.0).
Negative: Student performs less well (for example, their GPA is less than 2.0).
Question 4c
With reference to the administration's demands, what scenarios would be considered a true positive, a false positive, a true negative, or a false negative?
True Positive: High ACT score. Good GPA.
True Negative: Low ACT score. Bad GPA.
False Positive: High ACT score. Bad GPA.
False Negative: Low ACT score. Good GPA.
Question 4d
For each of these scenarios, what are its consequences?
Example Answers
True Positive: Student admitted and succeeds in college.
True Negative: Under-qualified student rejected, college's reputation and standard maintained.
False Positive: Student admitted, but unable to graduate.
False Negative: Student rejected, but could've actually graduated with high honors.
Question 4e
What ACT score would you set as the cut-off for admission? Explain your reasoning for your chosen threshold, specifying why it isn't higher or lower.
Question 4f
Suppose all those who scored lower than your cut-off are outright rejected. Considering individual students, why would such an admission rule be problematic?
Question 4g
With reference to the above histogram and the false positives and false negatives rates, would you say that ACT score is a good admissions criterion?
- Test Score Questions
- Is there a correlation between ACT score and academic performance in the first year? Would you say that ACT score is a good admissions criterion?
- In the context of admission, what is considered a "positive diagnosis" or a "negative diagnosis?"
- With reference to the administration's demands, what scenarios would be considered a true positive, a false positive, a true negative, or a false negative?
- For each of these scenarios, what are its consequences?
- What ACT score would you set as the cut-off for admission? Why isn't it higher or lower?
- Considering individual students, why would such an admissions rule be problematic?
- With reference to the ACT Scores histogram and the false positive and false negative rates, would you say that ACT scores are a good admissions criterion?
- Spam Filter Questions
- In the context of a spam/ham filter, what scenarios would be considered a true positive, a false positive, a true negative, or a false negative?
- For each of these scenarios, what is the consequence to the use of your email service?
- With reference to these consequences, what criteria would you be considering when setting a threshold value for a spam/ham filter?
- Choose a threshold value. With reference to the above criteria, why pick that one?
- Think about the ideas of true/false positive/negative. Who ultimately decides which email has been correctly classified as spam or not?
- Is the definition of "spam" something "real," something decided by convention, some other type of model?
- Suppose an email provider touts "BEST SPAM FILTER IN TOWN. 99.9% ACCURACY!" Why should you be skeptical of their claim?
Class Discussion
Instructions
For each question, let students discuss in small groups, and then let them share their thoughts with the whole class.
| 10 Minutes | Discuss question 1. |
| 12 Minutes | Discuss question 2. Spend about half the time on each part. |
| 8 Minutes | Discuss question 3. |
Discussion Questions
Question 1
In 1983, the Soviet Union's nuclear early-warning radar reported five intercontinental ballistic missiles incoming from the United States. Rather than immediately sending this information up the chain-of-command, the officer on duty decided to wait for additional evidence before taking action. No further evidence came, and it ultimately turned out that the radar system had malfunctioned.
What would be the costs (both immediate and long-term) of a false positive and false negative in this scenario?
Question 2
Suppose you have very bad migraines. Your doctors tell you migraines can be sparked by a lot of different things, which vary for different people: most kinds of food, coffee, stress, and/or bright sunlight. You want to find out what is causing your migraines, so you cut your diet to only potatoes and then slowly add one food back at a time, waiting a few days between each addition to see if you get a migraine.
The whole thing is stressful, and you're so hungry and grumpy that you lose your sunglasses on Monday. On Thursday, you eat chocolate again for the first time. That afternoon, you get a relatively small migraine. You are very sad, because chocolate is your very favorite food.
What are the different consequences of false positives and negatives in the two following scenarios.
- Should you wait for a few days and try eating a small amount of chocolate again, or should you ban it from your cupboards? Why?
- What if your migraine came with sudden, intense, lasting vertigo, and you work in skyscraper construction and can't take off work?
Answers
- The cost of slight migraines from the piece of chocolate is small, yet the benefit of that delicious morsel of sweet and creamy chocolate is so great. So, it's worth giving it a second chance and seeing what happens.
- The cost of migraines is now way higher as you could either lose your job or die. The benefit of the chocolate is the same. Maybe don't give the chocolate a second chance in this case.
Question 3
Recount an instance in which you thought you made a right diagnosis of a situation, but then found out that you were wrong. (Because you're a human.)
- Explain whether it was an instance of a false positive or a false negative.
- What makes it a false positive/negative as compared to a false negative/positive?
- Which type of error do you think would have been better to make in this situation, and why?