2.2 Systematic and Statistical Uncertainty

Learning Goals

[Link to PlayPosit]

Animation narrated by Saul:

More details

After this lesson, students should

1. Realize that our contact with reality is often mediated by measurement and quantification. We need to be aware that every measurement comes with some degree of uncertainty (deviation from the “true” value in reality).
2. Identify sources of measurement uncertainty/error that introduce statistical uncertainty/error, that introduce systematic uncertainty/error, and that introduce both.
3. Understand how to use repeated measures to reduce statistical uncertainty.
4. Recognize the difficulty of removing systematic uncertainty, and that the process of science involves creativity in identifying sources of systematic uncertainty and inventing strategies to reduce or eliminate them.

Definitions

• Statistical Uncertainty

  Differences between reality and our measurement on the basis of random imprecisions.

Systematic Uncertainty: Differences between reality and our measurement that skew our results in one direction.

• Accuracy

  How close the measured value is to the true value.

• Precision

  How similar are all the measured values of the same thing (consistency).

Examples

• "It won't do us any good to average lots of test subjects' heights together if our tape measure got shrunk in the wash!" (Systematic uncertainty)
• "Sure, those polls all claim to be accurate within three percentage points, but they just mean that their statistical accuracy is that good. The people they are talking to might not be representative of the whole population. For example, older people can be more likely to pick up the phone and talk to pollsters, so there might be a systematic bias in that direction."
• "Indeed, [the 2016] election has demonstrated, quite emphatically, that none of the polling models out there have adequately controlled for [systematic uncertainties]. Unless you understand and quantify your systematic errors -- and you can't do that if you don't understand how your polling might be biased—election forecasts will suffer from the GIGO problem: garbage in, garbage out." Source
• “The stiffness of many springs depends on their temperature. If you measure the stiffness of a spring many times, by compressing and decompressing it, the internal friction inside the spring may cause it to warm. You may see this by a systematic trend in your data set; for example, each data point in a data set will be smaller than the previous one.” Source
• “Biologists often test cancer drugs on cell lines. Cell lines are cell cultures (groups of living cells grown under controlled conditions, generally outside their natural environment) with a uniform genetic makeup. Conclusions about all cells of the cell type made from measurements or experiments performed on cell lines suffer from systematic error — cells in the body do not have a completely uniform genetic makeup and exist in conditions vastly different from a cell culture.” This is a more complex life science example of systematic error.
• "If we estimate the effect of a drug on weight by randomly assigning people to take the drug vs. not take it and then measure their weight after a year, we could subtract the average weight loss of drug-takers vs. non-drug-takers to get the effect size of the drug on weight loss. But the people know if they're taking a drug for weight loss, so there could be a placebo effect creating a systematic bias. So the better way to do the experiment is to give the control group sugar pills. Then we can be more confident that any weight loss is due to the drug, and not a systematic bias created by the placebo effect."
• If you asked just a handful of random people on the street how much they slept the night before, the average of their answers could be quite different from the true average of the whole population due to random differences between people (statistical uncertainty). This can be improved by asking more people (say, hundreds or thousands). However, if you asked hundreds of random people on a college campus the same question, all of their answers could be skewed in one direction due to collective sleep deprivation (systematic uncertainty), which would not be improved by asking more college students.
• Suppose you are an astronomer measuring the brightness of a star. The star twinkles due to random atmospheric fluctuations (statistical uncertainty), but the presence of the atmosphere itself, together with clouds, always reduces the brightness of the star (systematic uncertainty).

Common Misconceptions

• The words "uncertainty" and "error" mean that our instruments or measurement methods are somehow broken, deficient, or not to be trusted.

  These words describe the inevitable and perfectly acceptable gap between measurement and reality.

• A single measurement can only have either systematic or statistical uncertainty.

  Every measurement comes with systematic and statistical uncertainty, often with multiple sources to different degrees.

Context

This lesson teaches students that the inevitable imperfections of instrumental measurements of the real world can be quantified and studied in their own rights. They are categorised into statistical uncertainty and systematic uncertainty. We will teach them to be aware of the sources of uncertainty in each measurement and some elementary ways to mitigate them. This is illustrated by the human histogram activity, in which students can see statistical distributions and physically experience the effects of systematic uncertainty. We will also discuss how systematic uncertainties affected results of political polling in the 2016 US presidential election.

Before

1.2 Shared Reality

It is inevitable that our experience or measurement of the external reality is imperfect. This lesson’s concepts help to quantify these imperfections.

2.1 Senses and Instrumentation

No instrument is perfect. Systematic and statistical uncertainties help quantify these imperfections and allow us to compare two different instruments or methods of measurement.

After

3.1 Causation and Correlation

• Randomised assignment is one way to remove the systematic uncertainty by making sure that the intervention and control groups are not correlated with some other variable related to the method of assignment itself, e.g. a male vs. female group in a drug trial.
• Placebo effect is a systematic uncertainty in the measurement of the effectiveness of a treatment. Therefore, we must “subtract” the effect of the placebo treatment from the effect of the real treatment.

Recommended Outline

Before Class

• Prepare a seating chart.
• Review PlayPosit and discussion questions and ask faculty, Gabriel, or Emlen any questions you have.
• Prepare sheets of paper with ranges of heights for the human histogram activity (see below).
• Remind students to read the FiveThirtyEight article.
• (Optional) Prepare a presentation.

During Class

• (5 min) Come up with some fun way to assign the roles of spokesperson and notetaker (e.g. earliest birthday in the year, lives furthest from campus). Remind them of the responsibilities of these roles. This is also time for people to arrive.
• (2 min) Clicker question
• (7 min) Concept review questions
• (20 min) FiveThirtyEight reading discussion
• (33 min) Human Histogram
• This leaves 13 minutes of wiggle room for any activities that go long. If you have time at the end you can collect and answer any people’s lingering questions.

After Class

• Put the results from the Human Histogram activity into this document.
• Collect answers from notetakers for the forum / plenary.

Lesson Content

Discussion Questions

1. Question
   1. Option 1
   2. Options 2

Activity 1: Name

[Brief description of and motivation for the activity]

Discussion Questions

1. Question 1
   1. Subquestion a

1. 1. Subquestion b