6.2 Hill's Criteria

Building on Correlation and Causation, we examine how to collect evidence for causality in more difficult cases.

The Lesson in Context

This is a discussion-based lesson that familiarizes students with the concept of Hill's criteria, which are used when an ideal experiment (e.g. RCT) could not be done due to resource or ethical considerations. The criteria themselves are not difficult, but students typically have trouble associating their names with their meanings, and they would benefit from a diverse range of illustrative examples.

2.2 Systematic and Statistical Uncertainty

• When coming up with alternative explanations to an apparent correlation between two variables, it helps to consider factors that contribute to systematic and statistical uncertainties.

3.1 Probabilistic Reasoning

• A credence level is always associated with any scientific claim of causation. As each Hill's criterion adds to a case for causation, so our credence level for a causal relation increases. However, unlike in the case of RCTs, it may be difficult to quantify.

6.1 Correlation and Causation

• RCTs are introduced as ideal experiments for establishing causal relationships. These are often not possible due to resource or ethical considerations, and we must resort to Hill's criteria to examine the plausibility of causation. Causation is defined as correlation under intervention. When manual intervention is not possible, Hill's criteria can still build a strong case for causation.

11.2 When Is Science Suspect

• Students will explore how science has sometimes been used to justify the oppression of certain human groups. For example, current differences in achievement between human subgroups have been used to infer fundamental differences in biological or cognitive capacity, when in fact they may be sufficiently explained by differences in opportunity. These socially problematic causal inferences stem in part from the impossibility of an RCT that intervenes on genetics while keeping social opportunities equal between subgroups. It is important to recognise potential social implications when evaluating non-RCT evidence for causation, such as when drawing conclusions from observational studies alone.

Takeaways

Learning GoalsDefinitionsExamplesCommon Misconceptions

After this lesson, students should

1. Identify cases in which "ideal" RCT experiments are not possible, due to ethical or practical constraints.
2. For a given scenario in which a causal hypothesis/claim is being made, identify plausible alternative hypotheses that could be consistent with the data.
3. Identify additional sources of evidence that could be used to help mitigate flawed experiments, including prior plausibility, dose-response relationships, specificity, temporal ordering, and consistency across contexts.
4. Recognize when causal evidence in the absence of an RCT can be fairly compelling, especially if there are many different types of evidence combined.

Hill's criteria

• Prior plausibility

Can a plausible mechanism be constructed, or is there some other basis for interpreting the current evidence in terms of one causal structure over another, such as data from other studies?

• Temporality/Temporal Sequence

Did the hypothesized cause precede the effect?

• Specificity

Specific predictions for specific consequences that have come true are less likely to be caused by other factors.

• Dose-response Curve

Do the quantities of the hypothesized cause correlate with the quantity, severity, or frequency of the hypothesized effect across ranges?

• Consistency Across Contexts

Does the correlation appear across diverse contexts?

This list may differ from the one on Wikipedia or elsewhere. It may be worth mentioning to students that these are the criteria we have chosen to focus on in this course.

It is not necessary for all of Hill's criteria to be satisfied to infer causation. Each criterion adds to the case for causation. Some criteria are not applicable in certain situations (e.g. dose-response curve in whether light switches cause the light to turn on and off).

• Scrotal cancer in chimney sweeps, example by John.

• The causal connection between leaded gasoline across the world and violent crimes in many countries. (See video above, 17:57.)
  • Prior Plausibility: High levels of lead are known to cause cognitive damage. It is conceivable that extended exposure to lower levels of lead could have similar effects.
  • Temporality: In the graph shown in the video, the levels of violent crime correlate with the use of leaded gasoline, but delayed by about 20 years.
  • Specificity: Within the same demographic group, delinquents are 4 times more likely to have elevated bone lead concentrations than non-delinquents.
  • Dose-response Curve: See temporality.
  • Consistency Across Contexts: The delayed rise in violent crime after increased use of leaded gasoline is observed in many industrialized countries.

Since we can't run a randomized controlled trial on whether CO2 emissions cause global warming, we can't ever know whether it does.

A non-RCT study can still be serve as potentially weaker, but sometimes just as strong, evidence for causation.

Students are quick to notice small sample size, slower to notice problems with experimental design.

Students struggle to generate non-RCT types of evidence for causality, although they are better at recognizing it.

Identifying natural experiments is also difficult, probably because many students find the principles of RCTs slippery.

Useful Resources

Lecture VideoDiscussion SlidesHandouts and ActivitiesReadings and Assignments

Discussion Slides Template

The discussion slides for this lesson.

Lifetime Earnings and the Vietnam Era Draft Lottery

The paper the students analyze in the first part of the paper analysis activity.

Sunset time and the economic effects of social jetlag: evidence from US time zone borders

The paper the students analyze in the second part of the paper analysis activity.

The Environment and Disease: Association or Causation?

Recommended Outline

Before Class

Review PlayPosit and discussion questions and ask faculty, Gabriel, or Emlen any questions you have.

During Class

5 Minutes	Come up with some fun way to assign the roles of spokesperson and notetaker (e.g. earliest birthday in the year, lives furthest from campus). Remind them of the responsibilities of these roles.
15 Minutes	First set of discussion questions: Concept review.
25 Minutes	#Paper Analysis (Part 2): #Vietnam War Draft Lottery and #Sunset Time and Social Jetlag
15 Minutes	Further discussions: How to apply Hill's criteria to the social jetlag study.
15 Minutes	Go over the COVID-19 example.
5 Minutes	Answer any lingering questions. These activities have a tendency to go long, so some extra wiggle room is left at the end.

Lesson Content

Concept Review

1. (5 min) Review: What are the three essential elements of a Randomized Controlled Trial?

(10 min) For both of the following hypotheses, what RCT would you use to test it? Why would it be challenging to perform?

1. For a child, does spending more than 22 hours in their home per day over their childhood increase their risk of developing myopia?
2. On the level of individual towns, does implementing universal basic income reduce the rate of violent crimes in 10 years? In 150 years?

Paper Analysis (Part 2)

(25 min)

This is a continuation of last lesson's paper analysis, in which we looked at a classic example of a proper RCT. Here, we will look at two more papers in which an RCT is difficult to conduct, but a strong case can still be made about causation based on experimental results.

For each of the two papers, students should try to answer the following.

1. What causal relationship is this paper trying to study? What is the hypothesis?
2. What, if it exists at all, is the experimental intervention? (i.e., What is the independent variable that is being manipulated)?
3. What is the dependent variable that is being measured (i.e., the variable the researchers anticipate may be affected by the experimental intervention)?
4. How is the independent variable manipulated? Are there control and intervention groups? Are those randomly assigned? Is the control condition a good one (only the independent variable is different, with all else kept equal)? Is this an RCT?
5. What is the result of the experiment?
6. Can the causal relationship in question 1 be concluded from the experimental results? If not, what, if anything, can be concluded? How confident are you in this conclusion?
7. Can you think of an alternative explanation for the data?

Vietnam War Draft Lottery

Example of natural experiment, where the intervention and assignment are not performed by the experimenter but by environmental factors: Lifetime Earnings and the Vietnam Era Draft Lottery

Upshot: This is a natural experiment. All the aspects of an RCT are still satisfied, but the randomized selection and intervention process is not performed by the experimenter but by an external agent/natural process.

Sunset Time and Social Jetlag

Not technically an RCT but could be argued to have comparable strength: Sunset Time and the Economic Effects of Social Jetlag

Discussion Questions

In your small group, explain Hill's criteria using aspects of the Sunset Time and Social Jetlag study as examples.

Covid-19

(15 min)

There has been a lot of controversy regarding mask wearing policies to combat the spread of COVID-19. Suppose it were very difficult to conduct a randomized controlled trial to study the effect of community mask wearing on COVID spread. (Such RCTs have in fact been conducted; see this paper.) However, we'd still like to find out whether there is such a causal connection using Hill's criteria.

Work in small groups. For each of the Hill's criteria, what evidence, if observed, would support that criterion?

1. Plausible mechanism
2. Temporal sequence
3. Consistency across contexts
4. Dose-response curve
5. Specificity

---

⟨

Previous

6.1 Correlation and Causation

⟩

Next

7.1 Causation, Blame, and Policy