Topics and lesson plans

All the lesson plans used in the full fourteen week version of the course that's run at UC Berkeley. These include lecture videos as well detailed content that can be used in smaller discussion sections.

1.1 Introduction and When Is Science Relevant	When is science relevant? The many uses of a scientific approach.
1.2 Shared Reality and Modeling	Science is grounded in belief in a common, shared reality with some degree of regularity.
2.1 Senses and Instrumentation	Science uses both our direct senses and a variety of instruments to extend our ability to observe phenomena. We trust our instruments for the same reasons we trust our senses; interactive exploration and comparison.
2.2 Systematic and Statistical Uncertainty	This topic explores the sources of error and uncertainty in data.
3.1 Probabilistic Reasoning	Using meta-judgments of the likelihood that your best judgment is right— how confident you are—enables decisions that take uncertainty into account.
3.2 Calibration of Credence Levels	It is important to check the calibration of credence levels; that is, how good one's judgments are about how likely each of one's claims is to be right.
4.1 Signal and Noise	The challenges of finding the information we want amidst messy data.
4.2 Finding Patterns in Random Noise	We often find mistake noise for signal; how do we minimize these mistakes, given that they are not always easy to tell apart?
5.1 False Positives and Negatives	Considering the relative costs of each possible mistake helps us make better decisions under conditions of uncertainty, when we cannot eliminate the possibility of a mistake either way.
5.2 Scientific Optimism	Without scientific optimism, the idea that science is necessarily iterative and if we as scientists keep looking we will eventually gain insights, scientists would have discovered far less than they have.
6.1 Correlation and Causation	An introduction to the scientific approach to determining causal relationships.
6.2 Hill's Criteria	Building on Correlation and Causation, we examine how to collect evidence for causality in more difficult cases.
7.1 Causation, Blame, and Policy	Distinguishing singular causation (A caused B) from general causation (X tends to cause Y).
7.2 Emergent Phenomena	Many phenomena in science are emergent, i.e., visible only at higher levels of organization. This tends to occur when large numbers of elements interact, e.g. as in individuals on social media.
8.1 Orders of Understanding	Because each event and/or phenomenon has many causal factors, it is often important to distinguish which factors affect it the most and which factors play a smaller role.
8.2 Fermi Problems	Estimating quantities based on what we know.
9.1 Heuristics	Some of the heuristics biases that make our probability judgments go awry.
9.2 Biases	Some of the psychological biases that make our probability judgments go awry.
10.1 Confirmation Bias	Our tendency to preserve our existing or preferred beliefs, even against the evidence.
10.2 Blinding	Blind analysis, the practice of deciding how we will analyze data before finding out if the analysis we have chosen supports our hypothesis, counteracts confirmation bias.
11.1 Pathological Science	How to catch bad science.
11.2 When Is Science Suspect	The capacity for science to be misused to reinforce existing power structures.
12.1 Wisdom of Crowds and Herd Thinking	Explore ways that groups fall short of their optimal reasoning ability. There are better and worse ways to aggregate a group's knowledge.
12.2 Grill the Guest	Confront a working scientist about their work using course concepts.
13.1 Denver Bullet Study	The Denver Bullet Study offers one approach to integrating facts and values in a controversial real-world problem, drawing facts from a set of experts, gauging the values of different stakeholders, and bringing these together for a final decision.
13.2 Deliberative Polling	Another approach to getting groups of people to come together to make decisions, in a process where the integration of facts and values is scaffolded.
14.1 Scenario Planning	A third approach to integrating facts and values under conditions of uncertainty about what the future will be like.
14.2 Wrap Up	An overview and conclusion of the course.

List of Lesson Plans

• 1.1 Introduction and When Is Science Relevant
• 1.2 Shared Reality and Modeling
• 2.1 Senses and Instrumentation
• 2.2 Systematic and Statistical Uncertainty
• 3.1 Probabilistic Reasoning
• 3.2 Calibration of Credence Levels
• 4.1 Signal and Noise
• 4.2 Finding Patterns in Random Noise
• 5.1 False Positives and Negatives
• 5.2 Scientific Optimism
• 6.1 Correlation and Causation
• 6.2 Hill's Criteria
• 7.1 Causation, Blame, and Policy
• 7.2 Emergent Phenomena
• 8.1 Orders of Understanding
• 8.2 Fermi Problems
• 9.1 Heuristics
• 9.2 Biases
• 10.1 Confirmation Bias
• 10.2 Blinding
• 11.1 Pathological Science
• 11.2 When Is Science Suspect
• 12.1 Wisdom of Crowds and Herd Thinking
• 12.2 Grill the Guest
• 13.1 Denver Bullet Study
• 13.2 Deliberative Polling
• 14.1 Scenario Planning
• 14.2 Wrap Up

Week 1: The Purpose and Limitations of Science

• 1.1 Introduction and When Is Science Relevant
• 1.2 Shared Reality and Modeling

Week 2: What we can Measure

• 2.1 Senses and Instrumentation
• 2.2 Systematic and Statistical Uncertainty

Week 3: Statements of Uncertainty

• 3.1 Probabilistic Reasoning
• 3.2 Calibration of Credence Levels

Week 4: Dealing with Noise

• 4.1 Signal and Noise
• 4.2 Finding Patterns in Random Noise

Week 5: Trade-offs and Progress

• 5.1 False Positives and Negatives
• 5.2 Scientific Optimism

Week 6: Establishing Causation

• 6.1 Correlation and Causation
• 6.2 Hill's Criteria

Week 7: Types and Consequences of Causation

• 7.1 Causation, Blame, and Policy
• 7.2 Emergent Phenomena

Week 8: Describing and Simplifying Reality

• 8.1 Orders of Understanding
• 8.2 Fermi Problems

Week 9: Heuristics and Biases

• 9.1 Heuristics
• 9.2 Biases

Week 10: Catching and Countering Biases

• 10.1 Confirmation Bias
• 10.2 Blinding

Week 11: Science Gone Wrong

• 11.1 Pathological Science
• 11.2 When Is Science Suspect

Week 12: Group Evaluations

• 12.1 Wisdom of Crowds and Herd Thinking
• 12.2 Grill the Guest

Week 13: Better Group Decision Making

• 13.1 Denver Bullet Study
• 13.2 Deliberative Polling

Week 14: The Outlook and Potential of Science

• 14.1 Scenario Planning
• 14.2 Wrap Up

• 1.1 Introduction and When Is Science Relevant
  • Democracy vs. epistocracy
  • Facts vs. values
• 1.2 Shared Reality and Modeling
  • Shared reality
  • Raft vs. pyramid
  • Evaluation of models
  • Science vs. decree
  • Scientific realism vs. anti-realism
  • Operationalism, conventionalism, and realism
• 2.1 Senses and Instrumentation
  • Validation of instruments through interactive exploration, triangulation with other instruments, and comparison with direct senses
• 2.2 Systematic and Statistical Uncertainty
  • Measurement proxies as sources of systematic and statistical uncertainty
• 3.1 Probabilistic Reasoning & 3.2 Calibration of Credence Levels
  • The value of partial and probabilistic information
  • Words of estimative probability (probably, likely, definitely, etc.)
  • [math]\displaystyle{ p }[/math]-values and statistical significance
  • Error bars and confidence intervals
  • Strategies to improve calibration of credence levels (feedback, AOT, growth mindset, etc.)
• 4.1 Signal and Noise & 4.2 Finding Patterns in Random Noise
  • Signal-to-noise ratio
  • [math]\displaystyle{ p }[/math]-hacking
  • Look elsewhere effect
  • Gambler's fallacy
  • Hot-hand fallacy
  • File drawer effect
  • HARKing (hypothesizing after results are known)
  • Effect size (as distinct from statistical significance)
• 5.1 False Positives and Negatives
  • Thresholds between positive and negative detections
  • Trade offs between false positives and negatives
• 5.2 Scientific Optimism
  • Iterative progress
• 6.1 Correlation and Causation
  • Causation as correlation under intervention
  • Randomized controlled trials
  • Different directions of causation
  • Spurious correlations
• 6.2 Hill's Criteria (Causation in the Messy Real World)
  • Natural experiments
  • Hill's criteria for causation
  • Causal networks
• 7.1 Causation, Blame, and Policy
  • Singular and general causation
  • Acts of omission vs. commission (and the omission bias)
  • Status quo bias
• 7.2 Emergent Phenomena
  • Global effects that arise through the interaction of small pieces (rather than general causation)
  • Explanation at different scales
  • Scientific reductionism
• 8.1 Orders of Understanding
  • Orders of magnitude
  • Multiple causes of comparable importance
  • Orders of importance of causes
  • Refinement of models using higher order descriptions
  • Scale of impact of policies
• 8.2 Fermi Problems
  • Fermi problems
• 9.1 Heuristics
  • Base rate neglect
  • Representativeness heuristic and conjunction fallacy
  • Availability heuristic
  • Bounded rationality
• 9.2 Biases
  • Fundamental attribution error
  • Conformity
  • Obedience
  • Temporal discounting
• 10.1 Confirmation Bias
  • Selective exposure
  • Biased assimilation
  • Strategies for reducing confirmation bias (AOT, etc.)
• 10.2 Blinding
  • Techniques for blind analysis
  • Preregistration
  • Registered replication
  • Adversarial collaboration
  • Peer review
• 11.1 Pathological Science
  • The spectrum of poor research
  • Langmuir's pathological science indicators
• 11.2 When Is Science Suspect
  • The validity and reliability of social science metrics
  • External validity
  • The difficulty of creating and applying social science metrics in cross-cultural contexts
  • How researchers' biases and backgrounds shape their research agendas
• 12.1 Wisdom of Crowds and Herd Thinking
  • The Wisdom of Crowds effect removing independent biases via error cancellation
  • How humans can engage in herd thinking to cluster around biased answers to group questions
• 13.1 Denver Bullet Study
  • The Denver Bullet Study process of integrating factual analysis from experts with value ratings from different stakeholders
• 13.2 Deliberative Polling
  • The deliberative polling process of having the general public make decisions as though they were experts
• 14.1 Scenario Planning
  • The Scenario Planning process of predicting and responding to possible futures
  • What makes good drivers to use in Scenario Planning

Anatomy of a Lesson Plan

Work in Progress

This section is currently being written or undergoing major revision.

Each lesson plan has the following components:

1. Useful Links
2. Readings and Assignments
3. Learning Goals
4. Context
5. Recommended Outline
6. Lesson Content
7. Overflow (this section may sometimes be missing)

Useful Links

This section contains links to the prerecorded lecture videos (called "PlayPosits") that the students watch prior to discussion sections (called "labscussions"), the slides to use during labscussion, a "three column" overview of the week, and a link to Boxthe "website page". There may also be other links to handouts, videos, activities, papers, or other things relevant to the lesson.

Three Column Overview of the Week

This is a bullet point summary of everything in the PlayPosit videos, labscussion sections, and plenary session for the week. It's mainly intended for faculty to get a sense of what's being taught, but it may be helpful to others as well.

Lesson Slides

See the section on the anatomy of labscussion slides for more info.

These are the slides for this lesson. For most lessons you can use these slides directly as is. For a handful of lessons there will be different versions of the slides for each section. In those cases, separate links for each section will be made available. If you make any changes to the lesson slides, please duplicate the slides and leave them in the same Google Drive folder with your section number(s) in parentheses at the end of the name. This format is important for record keeping purposes.

A student-viewable version of the lesson slides will be linked in the syllabus after each lesson as many of the activities rely on the students being unaware of certain elements of them.

Website Page

You should generally ignoreBox the website page link. It's kept for historical reasons.

This website contains out of date content and may not necessarily align with the versions of the lessons presented on this wiki. The wiki is the most up to date version of the content. However, the Information Box on the website may still be a useful reference if you want a slightly different perspective on the material.

Readings and Assignments

This section has links to relevant readings and videos that can be assigned to students. In most cases, this also includes a supplemental lecture video presented by one or more professors that have previously taught the course. In the UC Berkeley version of the course, the students watch these videos prior to section.

Learning Goals

This is what we intend the students to take away from the lesson. The learning goals begin with the abstract ideas we want to walk away with followed by any specific definitions we would like the students to learn. These learning goals will be made available to the students and are included in the slides for every lesson. This is the content that is in scope for the quizzes and final exam.

This section also has "examples" and "common misconceptions" the the students often have or end up with. Not all the examples come up within the lessons normally. Nevertheless, you are welcome to use them. You are welcome to add any other examples you stumble upon or misconceptions you find your students falling victim to.

Context

This section contains general information about how this lesson fits into the broader picture of the course as well as which other lessons tie into it.

Recommended Outline

This includes any steps that the instructors or students need to take prior to labscussion section as well as any wrap-up that needs to be done after. The "during class" part of this section has recommended minute-by-minute timelines so that you get through the content. Feel free to adjust this timeline to fit the needs of your section.

Lesson Content

This is the actual meat of the lesson. It has detailed explanations for each activity in the order that they're done in labscussion.

Anything in a blue box is a useful note or tip for the instructor.

Anything in an orange box is a warning or note of caution for the instructor.

Anything in a green box or written in text like this is the answer or a sample answer to a question.

Overflow

This section contains activities and discussions that are not currently being used but that we want to maintain a record of regardless. Many of them were used in previous iterations of the course.

Anatomy of Labscussion Slides

The slides for each labscussion are color coded. The "gold-blue" slides represent the "introduction" slides and typically have content you can quickly brush through.

The remaining slides cycle through "red", "blue", and "green" slides. These always progress in the same order. A change in color corresponds is a visual cue that shows the labscussion is moving on to a new activity. All the slides for the same activity are of the same color. The colors keep cycling if there are more than three activities in a lesson. The section's agenda is also shown whenever there's a transition to a new activity.

• 

• 

• 

Introductions

Every slide deck begins with a title card that introduces the lesson. Followed by the lesson's agenda, learning goals, and definitions. Note that the lessons are (typically) not designed to have much time for review at the start. So, you don't have to spend time going over the learning goals and definitions in each section. They're mainly included so that you can flip back to them during class if it comes up and so that the students can reference them easily after class.

---

⟨

Previous

Assembling your team

⟩

Next

1.1 Introduction and When Is Science Relevant