5.1 False Positives and Negatives
More actions
Considering the relative costs of each possible mistake helps us make better decisions under conditions of uncertainty, when we cannot eliminate the possibility of a mistake either way.
The Lesson in Context
This lesson gives students hands-on examples of false positives and negatives through a Jupyter notebook, in which they are asked to make moral judgments on the threshold for a positive test result based on the consequences of a false positive or negative.
- In an earlier lesson, students explored how statistical and systematic uncertainty can make data suggest false inferences, through either random variation within an insufficiently large sample (statistical uncertainty) or some systematic bias in the data (systematic uncertainty). Both of these sources of uncertainty can lead to either false positives or false negatives.
- In this lesson, students will learn to use numeric credence levels to track and communicate their confidence levels about various claims and predictions. Credence levels can be a good way to make explicit the likelihood of false positives and false negatives in a given case, and one's tolerance for the two types of errors.
- In the previous lesson, students explored how we often mistake noise for signal, because we are always looking for patterns through which to make sense of the world. This constant search for patterns can lead to false positives. On the other hand, sometimes there is so much noise that we miss real patterns, leading to false negatives.
- "Scientific optimism" is a can-do attitude adopted by scientists by which they convince themselves that even difficult problems are solvable. With this "scientific optimism," scientists can successfully take on problems that take years or even decades to solve, with hundreds of steps and iterations involved in developing techniques, inventing technologies, collecting and analyzing data. Scientific optimism may also help scientists continue to detect false positives and false negatives in their previous work, iteratively improving our scientific understanding as we accumulate more evidence.
Takeaways
After this lesson, students should
- Understand that there is always the inevitability of a trade-off—for a given test, one can reduce the risk of false positives by increasing the risk of false negatives, and vice versa. You make this trade-off when you determine what the threshold is (e.g. convict someone if you're 99% or 70% certain that they're guilty).
- Understand that whether false positive or negative is preferable depends on a human values judgement on the consequences of either: (e.g. "innocent people shouldn't be in jail") and risks (e.g dangerous people committing more crimes).
False Positive/Type I Errors
False Negative/Type II Errors
Threshold
Unless the indicator variable is a perfect indicator of the underlying condition, there will be some probability of false positives and/or false negatives for any given threshold. Choosing a threshold is thus partly a matter of deciding one's relative tolerance for false positives and false negatives; a higher threshold will lead to more false negatives, a lower threshold to more false positives.
Incoming Asteroid
- "The oncoming asteroid has only a 1% chance of hitting Earth. But if it does, life on Earth will be destroyed. It'll be expensive to stop the asteroid, but the risk is bad enough it's worth it."
Cancer Treatment
- "I asked them to imagine that they faced a choice between two types of radiation therapy for early-stage breast cancer. The first treatment would leave them with a 15% chance of local recurrence and a 10% chance of moderate or severe breast fibrosis. The second treatment would leave them with only an 8% chance of local recurrence but a 30% chance of moderate or severe fibrosis. The radiation oncologists raised their hands in almost equal numbers for the two treatments. Some believed the higher risk of fibrosis was unacceptable, given the treatability of most local recurrences, whereas others believed the trauma of recurrence outweighed the discomfort of fibrosis. But sometimes physicians' values differ in important ways from those of many patients. When such value judgments are incorporated into professional treatment guidelines, without any explicit acknowledgment that a reasonable patient might choose an alternative course of treatment, they take potential choices away from patients." (Source)
There have been many people who still caught the virus even after a "negative" test result. These tests are just unreliable.
Useful Resources
Recommended Outline
Before Class
Go through the Jupyter notebook.
During Class
| 10 Minutes | Quickly review false positives and negatives. |
| 40 Minutes | Have the students work through the data science module. |
| 30 Minutes | Walk the students through the discussion questions. |
Lesson Content
Warm-up Question
If you're more worried about false positives than false negatives (e.g. for judging a defendant as guilty, it's worse to sentence someone innocent than let someone guilty go free), that means you should:
- Lower your threshold for counting an instance as a positive
- Raise your threshold for counting an instance as a positive
- Ignore your threshold for counting an instance as a positive
- Increase your signal-to-noise ratio
Explanation
This will decrease false positives (although it will also increase false negatives).
Data Science Module
Have the students work through the data science notebook. This should take around 40 minutes.
The Jupyter notebook itself should be self explanatory and contains several discussion questions. We suggest pairing the students for this activity. If necessary, the instructor may demonstrate how the notebook works to the class.
Part 1: H. Pylori
Helicobacter Pylori is a type of bacteria that can cause infections in the stomach. Symptoms of H. pylori include painful stomach ulcers. A common test for the infection is the urea breath test. After the patient ingests a tablet or drink containing urea enriched with the isotope carbon-13 (rather than the usual carbon-12), the test measures the proportion of carbon-13 in the carbon dioxide in the collected breath sample. If the amount of carbon-13 is in excess of the normal amount of carbon-13 in one's breath, then it suggests the presence of the bacteria in the stomach, because the bacteria have converted the ingested urea into carbon dioxide. The test result is how much excess carbon-13 is found in one's breath. For example, a test result of 5 means that there is 5 times as much carbon-13 as usual.
A study was done to find the distribution of the urea test results of subjects with or without H. pylori infection. Below is a histogram that shows the number of patients with a particular test result. Patients without the infection are plotted in blue; those who are infected are plotted in orange.
You are the designer of the H. pylori test. Your job is to decide on a value for its threshold, which will be used by all clinical laboratories to make diagnoses. What threshold value do you choose?
Any value between 2 and 21.
Part 2: Pregnancy Test
In the first days of pregnancy, the body produces a hormone called human chorionic gonadotropin (hCG). A home pregnancy kit works by detecting the presence of this hormone in the urine as an indication of pregnancy. The concentration of hCG is measured in units of milli-international units per millilitre (mIU/mL). hCG levels in a non-pregnant woman is typically less than 5 mIU/mL, and the number increases rapidly after the implantation of the embryo.
Question 2a
Home pregnancy test kits can only detect hCG if it is above a certain level. This is called the "sensitivity." If the test returns positive for all results above the sensitivity level and negative otherwise, what should the sensitivity level of a home pregnancy test be in order to ensure (1) no false positives and (2) less than 15% false negatives on the 9th day of pregnancy?
Lots of possible values. For example, with a threshold of 0.18, you can ensure no false positives and a 7.65% false negative rate.
Question 2b
What should the sensitivity of a home pregnancy test be in order to ensure (1) no false positives and (2) less than 15% false negatives on the 11th day of pregnancy?
Lots of possible values. For example, with a threshold of 1.11, you can ensure no false positives and a 0.45% false negative rate.
Part 3: Smoke Detector
Smoke detectors work by measuring how much the air is dispersing light. The more smoke there is, the less light will be able to pass through the air. The quantity measured by a smoke detector is called the "optical density", which has the units of "percent per metre" (%/m). The goal of a smoke detector is to warn inhabitants of a house of a possible dangerous fire. The following graph shows the number of times a certain value of optical density has been measured in various scenarios. The blue histogram counts those times when there is no dangerous fire (perhaps there is some greasy haze in the room from a stirfry), and the orange histogram counts those times when there is a dangerous fire (perhaps a stack of paper has been lit up by a cigarette).
Question 3a
In the context of the smoke detector's alarm, what scenarios would be considered a true positive, a false positive, a true negative, or a false negative?
True Positive: Alarm goes off, there's a fire.
True Negative: Alarm doesn't go off, there isn't a fire.
False Positive: Alarm goes off, there isn't a fire.
False Negative: Alarm goes off, there is a fire.
Question 3b
For each of these scenarios, what is the consequence to the inhabitants of the house?
Example Answers
True Positive: Lives saved.
True Negative: Everything is calm and no one is bothered.
False Positive: Alarm is piercingly loud, disrupting a totally normal meal preparation. "Crying wolf" makes people less likely to believe the alarm when it goes off.
False Negative: House is burnt down, hurting the unknowing inhabitants.
Question 3c
With reference to these consequences, what criteria would you be considering when setting a threshold value for a smoke detector?
You might want to lower the threshold to minimize false negatives (very severe bad outcome), but not all the way to zero because false positives would be very annoying.
Question 3d
Choose a value that best satisfies the criteria you set forth above. Explain whether you are satisfied with your choice.
One possible value is 0.32, with 6.1% false positive and 5.09% false negative rates. This allows some false positives because it is more important to save lives, but it doesn't push the threshold so low as to make the alarm too much less believable in the long run.
Question 3e
You have no doubt experienced an annoying smoke detector that has gone off, but nothing has burnt down, perhaps even prompting a pointless trip by the fire brigade. Why do situations like this keep happening?
Alarm manufacturers made a conscious choice to allow some false positives, so that more true fires can be reported. This decision potentially saves more lives.
Part 4: SAT Scores vs. Admission
The binary classification of positives and negatives as well as the trade-off between false positive rate and false negative rate have important social consequences, especially when the subjects of classification are individual humans.
One example of such a classification is the admission of undergraduates to universities on the basis of standardized test performance. A good score on tests like the ACT and SAT is known to correlate strongly with a high GPA in university. As the following graph shows, the higher the ACT score, the more likely it is that a student with that ACT score will achieve a high GPA.
Suppose you are a college admissions officer who is given the task of setting an ACT score cut-off, so that only students who score above this cut-off will be considered for admission at all. The college administration tells you that whatever value you choose, they would like at least 50% of the admitted students to eventually have a GPA of over 3.0 in their first year, in order to maintain the university's historically positive reputation.
Relating ACT Composite Score to Different Levels of First-Year College GPA
More information from the ACT about the relation between their test and students' college GPAs.
Question 4a
With reference to the above graph, is there a correlation between ACT score and academic performance in the first year? Would you say that ACT score is a good admissions criterion?
Yes. The higher ACT score, the better the student tends to perform in college. ACT seems like a good indicator of academic success according to this graph.
Question 4b
In the context of admission, what is considered a "positive diagnosis" or a "negative diagnosis?"
Positive: Student performs well (for example, their GPA is greater than 3.0).
Negative: Student performs less well (for example, their GPA is less than 2.0).
Question 4c
With reference to the administration's demands, what scenarios would be considered a true positive, a false positive, a true negative, or a false negative?
True Positive: High ACT score. Good GPA.
True Negative: Low ACT score. Bad GPA.
False Positive: High ACT score. Bad GPA.
False Negative: Low ACT score. Good GPA.
Question 4d
For each of these scenarios, what are its consequences?
Example Answers
True Positive: Student admitted and succeeds in college.
True Negative: Under-qualified student rejected, college's reputation and standard maintained.
False Positive: Student admitted, but unable to graduate.
False Negative: Student rejected, but could've actually graduated with high honors.
Question 4e
What ACT score would you set as the cut-off for admission? Explain your reasoning for your chosen threshold, specifying why it isn't higher or lower.
You have it at around 23. This way, more than half the students with a projected GPA greater than or equal to 3.0 are admitted and more than half of the students with a projected GPA less than 3.0 are rejected. Any higher and we run the risk of losing many qualified students. Any lower and we run the risk of lowering the academic reputation of the college.
Question 4f
Suppose all those who scored lower than your cut-off are outright rejected. Considering individual students, why would such an admission rule be problematic?
There is substantial overlap in ACT scores between students who have a high GPA and those with an average or low GPA. If you're an individual student within that overlap, you may be wrongly rejected.
Question 4g
With reference to the above histogram and the false positives and false negatives rates, would you say that ACT score is a good admissions criterion?
This is a value decision with regards about whether or not high GPA is what we even should be aiming for when admitting students. But if we proceed with that goal then ACT scores alone are a fairly poor predictor of whether individual students' ultimate GPA. It has enough uncertainty that it is likely not a good admissions criterion.
Part 5: Spam Filter
We are no strangers to spam—unsolicited emails. Many email providers implement a filter that is able to recognize spam emails just from its text. We call such a problem "spam vs ham classification" (ham is not spam). The filter system usually assigns a "spam score" to each email. The higher the score, the more likely that email is spam.
On a basic level, if the email contains phrases like collect your lotto prize, I am a Nigerian prince, or singles in your area, some points may be added to its spam score. Likewise, an abundance of typos like helo, doller, bizness, or wrold may also contribute to the spam score. More sophisticated filters may also look at other factors such as grammatical errors, capitalization, hyperlinks, and formatting.
In the following histogram, we've recorded the spam/ham classification of 10,000 emails, as well as their spam scores.
Question 5a
Suppose you are the programmer of this spam/ham filter. You must set a threshold value for the spam score, such that emails with a score higher than the threshold will be tossed into the spam folder automatically.
In the context of a spam/ham filter, what scenarios would be considered a true positive, a false positive, a true negative, or a false negative?
True Positive: Spam is tossed into the spam folder.
True Negative: Useful email delivered to the user's inbox.
False Positive: Useful email is tossed into the spam folder.
False Negative: Spam is delivered to the user's inbox.
Question 5b
For each of these scenarios, what is the consequence to the user of your email service?
Example Answers
True Positive: Happy user.
True Negative: Happy user.
False Positive: Important work email lost. User fails to respond and get's fired. Their life crumbles around them.
False Negative: User is annoyed and could potentially fall victim to some scam.
Question 5c
With reference to these consequences, what criteria would you be considering when setting a threshold value for a spam/ham filter?
Maximize the amount of false positives while minimizing the amount of false negatives.
Question 5d
Drag the slider in the graph above to change the threshold value, while paying attention to the false positive and false negative rates. Choose a threshold value. Explain your choice with reference to the criteria mentioned above.
One possible value is 85. This lowers the number of false positives (1.31%) and slightly raises the number of false negatives (12.54%). This tradeoff is based on the assumption that it's better to show spam to the user than to throw away their important work emails.
Question 5e
Think about the ideas of true/false positive/negative. Who ultimately decides which email has been correctly classified as spam or not?
The user ultimately decides whether an email is spam or ham to them. Maybe someone likes to see ads!
Question 5f
Is the definition of "spam" a realist, conventionalist, or operationalist one?
For the filter, it's more of a conventionalist one. Everyone generally agrees that "singles in your area" is a sign of spam or a scam. However, it's also operationalist. Every user has their own judgment of what would be spam to them.
Question 5g
Suppose en email provider touts "BEST SPAM FILTER IN TOWN. 99.9% ACCURACY!" Why should you be skeptical of their claim? Explain.
The word "accuracy" is misleading because it doesn't reveal anything about the false positive and false negative rates. A spam filter could filter out 100% of the spam and also toss away most of its users' useful emails. The filter could also claim 100% "accuracy" in identifying spam but ignore false positives.
Class Discussion
Instructions
For each question, let students discuss in small groups, and then let them share their thoughts with the whole class.
| 10 Minutes | Discuss question 1. |
| 12 Minutes | Discuss question 2. Spend about half the time on each part. |
| 8 Minutes | Discuss question 3. |
Discussion Questions
Nuclear Annihilation
In 1983, the Soviet Union's nuclear early-warning radar reported five intercontinental ballistic missiles incoming from the United States. Rather than immediately sending this information up the chain-of-command, the officer on duty decided to wait for additional evidence before taking action. No further evidence came, and it ultimately turned out that the radar system had malfunctioned.
What would be the costs (both immediate and long-term) of a false positive and false negative in this scenario?
A Terrible, Horrible, No Good, Very Bad Week
Suppose you have very bad migraines. Your doctors tell you migraines can be sparked by a lot of different things, which vary for different people: most kinds of food, coffee, stress, and/or bright sunlight. You want to find out what is causing your migraines, so you cut your diet to only potatoes and then slowly add one food back at a time, waiting a few days between each addition to see if you get a migraine.
The whole thing is stressful, and you're so hungry and grumpy that you lose your sunglasses on Monday. On Thursday, you eat chocolate again for the first time. That afternoon, you get a relatively small migraine. You are very sad, because chocolate is your very favorite food.
What are the different consequences of false positives and negatives in the two following scenarios.
- Should you wait for a few days and try eating a small amount of chocolate again, or should you ban it from your cupboards? Why?
- What if your migraine came with sudden, intense, lasting vertigo, and you work in skyscraper construction and can't take off work?
Answers
- The cost of slight migraines from the piece of chocolate is small, yet the benefit of that delicious morsel of sweet and creamy chocolate is so great. So, it's worth giving it a second chance and seeing what happens.
- The cost of migraines is now way higher as you could either lose your job or die. The benefit of the chocolate is the same. Maybe don't give the chocolate a second chance in this case.
Final Question
Recount an instance in which you thought you made a right diagnosis of a situation, but then found out that you were wrong. (Because you're a loser human.)
- Explain whether it was an instance of a false positive or a false negative.
- What makes it a false positive/negative as compared to a false negative/positive?
- Which type of error do you think would have been better to make in this situation, and why?