6.2 Hill's Criteria

Learning Goals

[Link to PlayPosit]

[Link to instructional video]

More details

After this lesson, students should

1. Identify cases in which "ideal" experiments are not possible, due to ethical or practical constraints.
2. For a given scenario in which a causal hypothesis/claim is being made, identify plausible alternative hypotheses that could be consistent with the data.
3. Identify additional sources of evidence that could be used to help mitigate flawed experiments, including prior plausibility, dose-response relationships, specificity, temporal ordering, and multiple consistency across contexts.
4. Recognize when causal evidence in the absence of an RCT can be fairly compelling, especially if there are many different types of evidence combined.

Definitions

Hill's criteria:

• Prior plausibility

  Can a plausible mechanism be constructed, or is there some other basis for interpreting the current evidence in terms of one causal structure over another, such as data from other studies?

• Temporality/temporal sequence

  Did the hypothesized cause precede the effect?

• Specificity

  Specific predictions for specific consequences that have come true are less likely to be caused by other factors.

• Dose-response curve

  Do the quantities of the hypothesized cause correlate with the quantity, severity, or frequency of the hypothesized effect across ranges?

• Consistency across contexts

  Does the correlation appear across diverse contexts?

Common Misconceptions

• Since we can't run a randomized controlled trial on whether CO2 emissions cause global warming, we can't ever know whether it does.

  A non-RCT study can still be serve as potentially weaker, but sometimes just as strong, evidence for causation.

• Students are quick to notice small sample size, slower to notice problems with experimental design.
• Students struggle to generate non-RCT types of evidence for causality, although they are better at recognizing it.
• Identifying natural experiments is also difficult, probably because many students find the principles of RCTs slippery.

Context

This is a discussion-based lesson that familiarises students with the concept of Hill's criteria, which are used when an ideal experiment (e.g. RCT) could not be done due to resource or ethical considerations. The criteria themselves are not difficult, but students typically have trouble remembering their names, and they would benefit from a diverse range of examples.

Before

2.2 Systematic and Statistical Uncertainty

When coming up with alternative explanations to an apparent correlation between two variables, it helps to consider factors that contribute to systematic and statistical uncertainties.

4.2 Probabilistic Reasoning

In the upcoming lesson on Probabilistic Reasoning, students will explore how to use quantified credence (i.e., confidence) levels to track and communicate their certainty about claims and predictions. For example, a .50 credence level indicates a 50-50 chance the claim is false or true, whereas a 1.00 credence level indicates complete certainty. Because it is difficult to impossible to conclusively infer causation without an RCT, Hill's criteria are typically used to infer causation with varying degrees of confidence, such that each additional piece of evidence (especially of new types) increases the confidence that the causal claim is true.

5.2 Correlation and Causation

• RCTs are introduced as ideal experiments. These are often not possible due to resource or ethical considerations, and we must resort to Hill's criteria to examine the plausibility of a causation.
• Causation is defined as correlation under manipulation. When manual manipulation is not possible, Hill's criteria can still help establish causation.

After

9.2 When Is Science Suspect

In this upcoming lesson, students will explore how science has sometimes been used to justify the oppression of certain human groups, e.g. by inferring from current differences in achievement fundamental differences in capacity, when in fact these differences in achievement can be fully explained by differences in opportunity. Because it is impossible to randomly assign individuals to human groups such as "African-Americans" or "women," all of this science is observational, and subject to the limitations of observational research. In evaluating non-RCT evidence, it is important to remember these limitations of observational as opposed to experimental research, especially when data may be used to exacerbate existing injustices.

Recommended Outline

Before Class

• [Any essential logistical things that need to be done for this class]
• Prepare a seating chart.
• Review PlayPosit and discussion questions and ask faculty, Gabriel, or Emlen any questions you have.
• (Optional) Prepare a presentation.

During Class

• (5 min) Come up with some fun way to assign the roles of spokesperson and notetaker (e.g. earliest birthday in the year, lives furthest from campus). Remind them of the responsibilities of these roles.
• (21 min) First set of discussion questions: Concept review & time zone effect on sleep.
• (16 min) Second set of discussion questions: MSG and headaches.
• (13 min) Third set of discussion questions: Covid-19.
• (20 min) Fourth set of discussion questions: Broader questions about evidence for causality.
• (5 min) Collect questions for plenary.

Lesson Content

Clicker Question

1. Question
   1. Option 1
   2. Options 2

Concept Review

1. (8 min) Review: What are the three essential elements of a Randomized Controlled Trial?
2. (3 min) With reference to the essential aspects of RCTs, come up with a situation in which you could not do it.
3. (10 min) Recall the sleep study performed across time zone borders in the US (Sunset Time and the Economic Effects of Social Jetlag), which we looked at in 2.2 Causation and Correlation. In your small group, explain Hill's criteria using language that would help a friend or family member understand the concept and use aspects of the sleep study as examples.

Chinese Restaurant Syndrome

Starting from the late 1960s, some people in the USA noticed that they had a headache after eating at Chinese restaurants. They attributed their symptoms to the consumption of monosodium glutamate (MSG), a common food additive in East Asian cuisines. "Chinese Restaurant Syndrome" was coined as a term referring to the negative health effects of MSG.

1. (4 min) Describe a randomised controlled trial (RCT) that can test the hypothesis that "consumption of MSG causes headaches."
2. (10 min) Suppose that for some reason a RCT could not be done, but after some research, the following facts are known.
   • MSG also appears in Western cuisine, naturally in tomatoes and artificially added in many processed foods, and Americans who reported Chinese Restaurant Syndrome did not experience such symptoms after eating a Western meal containing MSG.
   • People in East Asian countries, who consistently consume meals containing high amounts of MSG, do not report such symptoms.
   • No biological pathway is known that connects MSG consumption and headaches.
   1. (2 min) Why might a RCT not be able to be conducted?
   2. (6 min) Each of the three points above can be used as evidence against a causal relationship between MSG consumption and headaches. For each point, discuss which of Hill's criteria for causality is not satisfied as a result of that fact.

Based on Hill's Criteria we can say that there is not a general causation between MSG and the symptoms claimed by "Chinese Restaurant Syndrome." See the quote at this link:

"These adverse event reports helped trigger FDA to ask the independent scientific group Federation of American Societies for Experimental Biology (FASEB) to examine the safety of MSG in the 1990s. FASEB's report concluded that MSG is safe. The FASEB report identified some short-term, transient, and generally mild symptoms, such as headache, numbness, flushing, tingling, palpitations, and drowsiness that may occur in some sensitive individuals who consume 3 grams or more of MSG without food. However, a typical serving of a food with added MSG contains less than 0.5 grams of MSG. Consuming more than 3 grams of MSG without food at one time is unlikely."

Covid-19

In the early days of Covid the virus was primarily found in Wuhan, China and Italy. At the same time there was a surge of pneumonia cases in those same areas. There were no studies yet on it, yet we knew with fairly high confidence that the virus was causing sickness. Explain how, even before rigorous studies had been done, each of Hill's criteria was applied as evidence that Covid was causing sickness.

1. (3 min) Plausible mechanism
2. (3 min) Temporal sequence
3. (3 min) Consistency across contexts
4. (2 min) Dose-response curve
5. (1 min) Specificity

Causal Link Brainstorming

1. (5 min) Brainstorm three causal links that you are pretty confident are real without RCT evidence.
2. (4 min) Why are you confident about these causal links?
3. (5 min) Can you come up with a situation where someone might come to the wrong conclusions about these causal links?
4. (5 min) How might using just one of Hill's criteria lead someone to a wrong conclusion?

Collect Questions for Plenary

(5 min) Collect remaining questions from the students for faculty in plenary (can be questions for clarification, extension, discussion, etc.), and add [ here].