3.1 Probabilistic Reasoning

Learning Goals

[Link to PlayPosit]

[Link to instructional video]

More details

After this lesson, students should

Recognize that every proposition comes with a degree of uncertainty.
Learn to value and defend scientific expressions of uncertainty.
Understand that because every proposition comes with a degree of uncertainty:
1. Partial and probabilistic information still has value.
2. Back-up plans are important since no information is absolutely certain.
3. One’s attitude should be more directed towards accurately assigning confidence levels rather than being overinvested in being “right.”
4. Scientific culture primarily uses a language of probabilities, not certain facts.
5. Even correctly-done science will obtain incorrect results some of the time.

Definitions

Credence
Level of confidence that a claim is true, from 0 to 1.
Confidence
Essentially a synonym for credence, as in “level of confidence,” instead of colloquial meaning, “state of having a lot of confidence.”
Accuracy
How frequently one is correct; proximity to a true value.
Calibration
How closely confidence and accuracy correspond; that is, how accurate a person or system is at estimating the probability that they are correct.

Examples

Weather forecasts.
Using polls to predict elections.
You might decide your credence level that your crush will say yes if you ask them to the dance, and use that to decide whether to ask or not.
Credence levels predicting the probabilities of natural disasters within specific time frames (earthquakes, floods, wildfires etc.), and using these to make decisions about disaster preparation.
Use credence levels about getting into various colleges to decide what to use as a safety school.
Using credence levels about passing tests to decide whether to study more.
Saul's story of a physicist who cancelled a lecture five minutes in because the presenter wasn't sure how his error bars were calculated.

Common Misconceptions

My aunt still caught COVID even after taking the vaccine, therefore it’s not true that the vaccine prevents COVID.
When phrased in terms of risks, the effectiveness of a vaccine is understood as the reduced probability that one would contract the disease after taking the vaccine, rather than total protection. Partial, probabilistic, improvements still have tangible causal consequences.

Context

After introducing the concept of scientific uncertainty in previous lessons, we now teach the students that this uncertainty permeates all discussions of facts. Every factual claim should inherently carry a level of confidence as a percentage. It allows scientists to be open to the possibility that they may be wrong, while still being able to meaningfully discuss and compare the validity of factual statements. We aim to teach students that this way of thinking is important in daily life, often in the context of risk assessment, as well as in common discourse about social issues.

Before

1.1 Introduction

Facts vs. values: Since credence levels can only be assigned to factual statements, it is important to first distinguish between statements of fact and statements of value.

3.1 Systematic and Statistical Uncertainty

Measurements in the real world are imperfect, and measurement uncertainties/errors can be studied and quantified. This translates to a confidence interval for every measurement result, i.e. “We are x% confident that the true value lies within this interval.”

3.2 Signal and Noise

In that lesson, students explore how the signal they are looking for in data can be difficult to find amid the noise (random variation, random error, imperfect measurements, etc.). Because data is a mix of signal and noise, inferences from data tend to have some degree of uncertainty, which may be usefully quantified using credence levels or probabilities.

4.1 Finding Patterns in Random Noise

Since spurious patterns are expected to arise from random noise alone, any claim of actual pattern must carry with it a level of confidence that it is not due to random noise.
p-value: The probability that the observed pattern is due to random noise. In other words, one minus the p-value gives the level of confidence that the observed pattern is not due to random noise.

4.2 False Positives and Negatives

Since every binary test has a certain rate of false positives and false negatives, the result of such a test should only be understood as a recommendation of odds or risks, rather than a conclusive determination. Successive test results help one adjust their belief as well as their confidence level in that belief, e.g. whether one is suffering from a disease.

After

6.2 Calibration of Credence Levels

This lesson will follow up on the current one by teaching students how to calculate the calibration, or quality, of their credence, noticing and quantifying both underconfidence and overconfidence.

Recommended Outline

Before Class

[Any essential logistical things that need to be done for this class]
Prepare a seating chart.
Review PlayPosit and discussion questions and ask faculty, Gabriel, or Emlen any questions you have.
Print out the handouts.
(Optional) Prepare a presentation.

During Class

(5 min) Come up with some fun way to assign the roles of spokesperson and notetaker (e.g. earliest birthday in the year, lives furthest from campus). Remind them of the responsibilities of these roles.
(15 min) Have the students fill out the credence level warmup survey.
(47 min) Conduct both probabilistic debates.
(8 min) Have the students make their future credence levels predictions.
(5 min) Collect questions for plenary.

After Class

Tell students to keep their handouts, so that the future predictions may be revealed in the next lesson, 6.2 Calibration of Credence Levels.
Collect answers from notetakers for the forum / plenary.

Lesson Content

Clicker Question

Question

First option
Second option
Third option
Fourth option

Activity 1: Name

[Brief description of and motivation for the activity]

Instructions

Discussion Questions

Question 1
Subquestion a

Subquestion b

Collect Questions for Plenary

(5 min) Collect remaining questions from the students for faculty in plenary (can be questions for clarification, extension, discussion, etc.), and add [ here].