3.2 Calibration of Credence Levels

Learning Goals

[Link to PlayPosit]

[Link to instructional video]

More details

After this lesson, students should

1. Be wary of high levels of confidence.
2. Understand the different ways in which scientists in different fields discuss credence levels (e.g. 95% confidence interval, error bars, 3σ).
3. Appreciate that one can improve on the calibration of their credence levels, and one should strive to reach an accurate calibration.

Definitions

• Confidence interval

  A range of values within which the true value of interest lies with a specified probability (typically 95%, meaning there is a 5% chance that the true value falls outside of this range).

• Error bars

  Smaller bars on a graph that show the range of likely true values around the observed value, typically a 95% confidence interval, or the observed value ± the standard error or standard deviation (equivalent to 68% confidence interval).

Examples

• In the figure below, lawyers have a wide range of predictions (20%-100%) for how likely they are to win any given case. However, the actual results show a much narrower band (40%-70%). Really, the outcome is more of a toss-up. (Source)

  

• In this figure, calibration curves are shown for experienced nurses as well as students training to become nurses. In both cases, they appear to be overconfident when outcomes are more likely and underconfident when outcomes are less likely. The nurses did not seem to become better calibrated with time. (Source)

  

• It is possible (and expected) for singular election forecasts to turn out to be wrong some of the time. It is fine as long as the predictions within that credence range are overall well calibrated. It is also impossible to calibrate the credence of a singular prediction, it is only possible for an aggregate of predictions. (Source)

Common Misconceptions

• He seems super confident, and she said she was only 85% sure, so we should trust him over her.

  It is more important to have the self-awareness of how often they are wrong, than to always insist they are right. If possible, use the outcomes of their past predictions to evaluate the calibration of their confidence before placing trust in them.

• Am I well calibrated in this particular prediction?

  The question doesn’t really make sense. You can’t have a calibration for a single prediction. A calibration level is only meaningfully defined over a collection of predictions for which you can see how many came ultimately true.

Context

This lesson continues the previous lesson, 6.1 Probabilistic Reasoning, and addresses the importance of the calibration of credence levels. We illustrate with some real-life examples that people, even experts, often exhibit overconfidence. By resolving the Future Predictions activity from the previous lesson, we show the students how to improve their own calibration. Finally, with the Actively Open-minded Thinking and Growth Mindset surveys, we introduce psychological attitudes that can help motivate one to improve their calibration.

Before

1.2 Shared Reality]

Raft vs. pyramid: Having concrete methods to assess and calibrate credence levels allows scientists to comfortably discuss their imperfect results without being overly invested in being right. This also leaves open the possibility that scientific claims can be revised or overturned in light of new evidence. This is a strength of the scientific method, rather than a weakness.

4.2 False Positives and Negatives

With accurately calibrated credence levels, decisions can be made on whether to act upon a prediction by considering the seriousness of its consequences. For example, a city should invest heavily in storm protection even when there is a 10% confidence that a strong hurricane will hit the city.

5.1 Scientific Optimism

Actively Open-minded Thinking and Growth Mindset are ways to encourage self-improvement and iterative progress, attitudes necessary to achieve scientific progress.

6.1 Probabilistic Thinking

The current lesson refines the previous one by introducing concrete ways to assess one’s credence calibration and to improve upon them.

After

8.1, 8.2 Heuristics and Biases

Representativeness (and base-rate neglect) and availability heuristic may be a source of over- or under-confidence in estimating the likelihood of future events.

11.1 Singular and General Causation

Weakly related: Since the calibration of credence levels is a statistical quantity, it is impossible and meaningless to discuss the calibration of a singular prediction. It is only meaningful to discuss the calibration of an aggregate of many predictions in a certain credence range.

Recommended Outline

Before Class

• [Any essential logistical things that need to be done for this class]
• Prepare a seating chart.
• Review PlayPosit and discussion questions and ask faculty, Gabriel, or Emlen any questions you have.
• (Optional) Prepare a presentation.

During Class

• (5 min) Come up with some fun way to assign the roles of spokesperson and notetaker (e.g. earliest birthday in the year, lives furthest from campus). Remind them of the responsibilities of these roles.
• ([#] min) [Description of some module from #Lesson Content]
• (5 min) Collect questions for plenary.

After Class

• [Any essential logistical things that need to be done as followup for this class]
• Collect answers from notetakers for the forum / plenary.

Lesson Content

Clicker Question

Question

1. First option
2. Second option
3. Third option
4. Fourth option

Activity 1: Name

[Brief description of and motivation for the activity]

Discussion Questions

1. Question 1
   1. Subquestion a

   2. Subquestion b

Collect Questions for Plenary

(5 min) Collect remaining questions from the students for faculty in plenary (can be questions for clarification, extension, discussion, etc.), and add [ here].