2.2 Systematic and Statistical Uncertainty

Any measurement by an instrument comes with its inevitable imperfections. How do we quantify and communicate the extent to which the reading on an instrument can be trusted? We classify reasons why the reading may differ from the true value into two broad categories—systematic uncertainty and statistical uncertainty. We must learn to deal with uncertainty in our knowledge.

The Lesson in Context

We physically illustrate the difference between statistical and systematic uncertainty with the human histogram activity, in which every student gets to participate as a data point. We will also discuss how systematic uncertainties affected results of political polling in the 2016 US presidential election. This is the first in a series of lessons that familiarize students with the important concept of epistemic uncertainty.

Earlier Lessons

1.2 Shared Reality and Modeling

• It is inevitable that our experience or measurement of the external reality is imperfect. This lesson's concepts help to quantify these imperfections.

2.1 Senses and Instrumentation

• No instrument is perfect. Systematic and statistical uncertainties help quantify these imperfections and allow us to compare two different instruments or methods of measurement.

---

Later Lessons

3.1 Probabilistic Reasoning

• Instrumental uncertainty can be expressed as error bars and confidence intervals. These translate to a probabilistic understanding of where the true value lies.

6.1 Correlation and Causation

• Randomized assignment is one way to remove the systematic uncertainty by making sure that the intervention and control groups are not correlated with some other variable related to the method of assignment itself, e.g. a male vs. female group in a drug trial.
• Placebo effect is a systematic uncertainty in the measurement of the effectiveness of a treatment. Therefore, we must "subtract" the effect of the placebo treatment from the effect of the real treatment.

Takeaways

Learning GoalsDefinitionsExamplesCommon MisconceptionsExpanded Learning Goals

After this lesson, students should

1. Realize that our contact with reality is often mediated by measurement and quantification. We need to be aware that every measurement comes with some degree of uncertainty (deviation from the "true" value in reality).
2. Identify sources of measurement uncertainty/error that introduce statistical uncertainty/error, that introduce systematic uncertainty/error, and that introduce both.
3. Understand how to use repeated measures to reduce statistical uncertainty.
4. Recognize the difficulty of removing systematic uncertainty, and that the process of science involves creativity in identifying sources of systematic uncertainty and inventing strategies to reduce or eliminate them.

Students will likely keep asking "but how can I tell between statistical and systematic uncertainties", and the answer would be to offer as many diverse examples as possible. Also if you can reduce the uncertainty simply by collecting more data, it's statistical.

Statistical Uncertainty

Differences between reality and measurements on the basis of random imprecisions or "noise."

Systematic Uncertainty

Differences between reality and measurements that skew results in one direction.

Accuracy

How close the measured value is to the true value.

Precision

How similar are all the measured values of the same thing (consistency).

A measurement can be very precise but wrong/inaccurate (low statistical uncertainty but high systematic uncertainty), or it could have a large variance between subsequent measurements but average accurately to the true value (low systematic uncertainty but high statistical uncertainty).

Proxy

An observable measurement used to approximate a quantity that is not directly observable or measurable. E.g. people's ratings of agreement with the statement "I am happy" on a scale from 1 to 7, is a measure (proxy) of their happiness, or using zip code as a proxy for socio-economic status.

Because a proxy is not a direct measure of the quantity of interest, it is a place that systematic bias can creep in. For example, using self-report ratings of happiness as a measure of happiness may be subject to cultural differences and/or comparison effects.

Triangulation

A method of dealing with systematic errors inherent in a type of measurement by attempting to measure the same phenomenon in multiple different ways or through lots of different proxies.

Proxy Example

When measuring crime rate, it is only possible to measure the rate of reported crime, making it a proxy of the true crime rate. Even when measuring temperature, it is only possible to observe the reading on an instrument, also a form of proxy.

Simple Systematic Uncertainty Example

"It won't do us any good to average lots of test subjects' heights together if our tape measure got shrunk in the wash!"

Simple Statistical Uncertainty Example

"Sure, those polls all claim to be accurate within three percentage points, but they just mean that their statistical accuracy is that good. The people they are talking to might not be representative of the whole population. For example, older people can be more likely to pick up the phone and talk to pollsters, so there might be a systematic bias in that direction."

Polling Models

"Indeed, [the 2016] election has demonstrated, quite emphatically, that none of the polling models out there have adequately controlled for [systematic uncertainties]. Unless you understand and quantify your systematic errors—and you can't do that if you don't understand how your polling might be biased—election forecasts will suffer from the GIGO problem: garbage in, garbage out."

The Science of Error: How Polling Botched the 2016 Election

Forbes article on systematic error in 2016 election polling.

Stiffness of Springs

"The stiffness of many springs depends on their temperature. If you measure the stiffness of a spring many times, by compressing and decompressing it, the internal friction inside the spring may cause it to warm. You may see this by a systematic trend in your data set; for example, each data point in a data set will be smaller than the previous one."

Sources of Systematic Error

Notes covering the spring-stiffness example.

Systematic Error in Life Sciences

"Biologists often test cancer drugs on cell lines. Cell lines are cell cultures (groups of living cells grown under controlled conditions, generally outside their natural environment) with a uniform genetic makeup. Conclusions about all cells of the cell type made from measurements or experiments performed on cell lines suffer from systematic error — cells in the body do not have a completely uniform genetic makeup and exist in conditions vastly different from a cell culture." This is a more complex life science example of systematic error.

Reducing Systematic Error

"If we estimate the effect of a drug on weight by randomly assigning people to take the drug vs. not take it and then measure their weight after a year, we could subtract the average weight loss of drug-takers vs. non-drug-takers to get the effect size of the drug on weight loss. But the people know if they're taking a drug for weight loss, so there could be a placebo effect creating a systematic bias. So the better way to do the experiment is to give the control group sugar pills. Then we can be more confident that any weight loss is due to the drug, and not a systematic bias created by the placebo effect."

Sleep Questionnaire

If you asked just a handful of random people on the street how much they slept the night before, the average of their answers could be quite different from the true average of the whole population due to random differences between people (statistical uncertainty). This can be improved by asking more people (say, hundreds or thousands). However, if you asked hundreds of random people on a college campus the same question, all of their answers could be skewed in one direction due to collective sleep deprivation (systematic uncertainty), which would not be improved by asking more college students.

Brightness of a Star

Suppose you are an astronomer measuring the brightness of a star. The star twinkles due to random atmospheric fluctuations (statistical uncertainty), but the presence of the atmosphere itself, together with clouds, always reduces the brightness of the star (systematic uncertainty).

Exemplary Quotes

“It won't do us any good to average lots and lots of test subjects' heights together if our tape measure got shrunk in the wash!”

“Sure, those polls all claim to be accurate within three percentage points, but they just mean that their statistical accuracy is that good. They have no idea if the people who they are reaching in their polling might be a highly skewed segment of the public so that they have, for example, a systematic bias towards the opinions of an older than average population.”

“Indeed, this election has demonstrated, quite emphatically, that none of the polling models out there have adequately controlled for them. Unless you understand and quantify your systematic errors—and you can't do that if you don't understand how your polling might be biased—election forecasts will suffer from the GIGO problem: garbage in, garbage out.”

Source

“Biologists often test cancer drugs on cell lines. Cell lines are cell cultures (groups of living cells grown under controlled conditions, generally outside their natural environment) with a uniform genetic makeup. Conclusions about all cells of the cell type made from measurements or experiments performed on cell lines suffer from systematic error—cells in the body do not have a completely uniform genetic makeup and exist in conditions vastly different from a cell culture.”

The words "uncertainty" and "error" mean that our instruments or measurement methods are somehow broken, deficient, or not to be trusted.

These words describe the inevitable and perfectly acceptable gap between measurement and reality.

A single measurement can only have either systematic or statistical uncertainty.

Every measurement can come with systematic and statistical uncertainty, often with multiple sources to different degrees.

After this lesson, students should

1. Attitudes
   1. Be aware that every measurement comes with some degree of uncertainty.
2. Concept Acquisition
   1. Statistical Uncertainty/Error: Differences between reality and our measurement on the basis of random imprecisions.
      1. All measurements have a certain amount of variance, which are just differences between multiple measurements due to error and/or genuine variation in the sample. These differences will not all go in the same direction.
      2. Statistical uncertainty can be reduced by averaging a larger amount of data.
   2. Systematic Uncertainty/Error: Differences between reality and our measurement that skew our results in one direction.
      1. Such measurements will show a consistent bias—that is, a consistent deviation from reality in one direction.
      2. Systematic uncertainty cannot be reduced by averaging a larger amount of data.
3. Concept Application
   1. Identify sources of measurement uncertainty/error that introduce statistical uncertainty/error, that introduce systematic uncertainty/error, and that introduce both.
   2. Suggest approaches to reducing or constraining measurement uncertainty (both statistical and systematic).

Additional Content

You must be logged in to see this content.

---

⟨

All lesson plans

⟨

Previous

2.1 Senses and Instrumentation

⟩

Next

3.1 Probabilistic Reasoning