6.1 Correlation and Causation

Useful Links

• Lesson Slides
• Website Page

Learning Goals

After this lesson, students should

1. Be able to explain why correlation is insufficient to demonstrate causation because there are multiple causal structures that lead to correlation:
   1. A causes B (direct causation);
   2. B causes A (reverse causation);
   3. A and B are both caused by C
   4. A causes B and B causes A (bidirectional or cyclic causation);
   5. There is no connection between A and B, and the correlation is a coincidence;
   6. The effect of A on B depends on C.
2. Be able to explain and justify the essential features of a Randomized Controlled Trial (RCT): An attempt to identify causal relations by randomly assigning subjects into two groups and then performing an experimental intervention on the subjects in one of the groups.
   1. Be able to recognize and explain the function of a control condition.
   2. Be able to recognize and explain the function of randomized assignment.
3. Recognize the epistemic power of a well-designed RCT as evidence for causation, if the experimental condition turns out to be significantly different from the control condition.

Definitions

• Randomized Controlled Trial (RCT)

  An attempt to identify causal relations by randomly assigning subjects into two groups and then performing an experimental intervention on the subjects in one of the groups. It consists of three essential components:

  • Experimental intervention

    The act of the experimenter changing a variable under study (the "independent variable") on a subset of the sample, to see if it influences a second variable (the "dependent variable").

  • Control group/condition

    A subset of the study sample, often half, treated the same as the rest except that the experimental intervention is withheld. This yields a baseline against which the part of the sample subjected to the experimental intervention (the "experimental condition") can be compared.

  • Random assignment

    Individual samples are randomly assigned to the intervention or control group. This ensures that the variable under study is the only difference between the two groups and reduces systematic differences in any other variable between them due to the way the samples have been assigned to them.

• Correlation

  The correlation between two numerical variables X and Y is a measure of how much they increase/decrease with each other.

  • Positive correlation

    Y increases as X increases.

  • Negative/Inverse correlation

    Y decreases as X increases.

• Causation

  X causes Y if and only if X and Y are correlated under interventions on X.

Common Misconceptions

• You cannot infer anything from an RCT about anything that wasn't in the study.

  What's important is that the sample you're studying is, in key ways, representative of the group you want to make conclusions about. What these "key ways" are varies a lot depending on what's being studied. But, the more similar the group you're making conclusions about is to the one sampled in the RCT, the more likely it is to be true. For example, a conclusion about students at one university may be very likely to hold for students at a similar school. It could also hold for university students in general. Depending on what you're studying, it might hold for people of that age group in general. And if you want to generalize the conclusion to an even broader population (the country or world as a whole) then you need to think very carefully about whether your sample represents any confounding variables in this larger group.

• An RCT doesn't tell you very much if it's only a small fraction of the total population that you want to study.

  There is some sense in which this is true. You need to have a sufficiently large sample to "average out" all the differences between the group you're studying. But, once this criteria is met, the sheer size of the sample is no longer a concern. The main issue then is what group your sample is representative of.

• The control and intervention groups need to be roughly the same size.

  This isn't true so long as both groups are sufficiently large to capture the differences within the sample.

• If an RCT shows strong evidence that X causes Y, then X must cause Y in every single case.

  The RCT demonstrates that a relationship tends to exist on the scale of an overall population. It does not mean that individual cases are necessarily subject to that exact condition.

Context

We introduce one definition of causation—as (statistically significant) correlation under intervention—and the Randomised Controlled Trial (RCT), which is a method of isolating and studying a causal relationship between two variables, even when the world is full of complex causal structures and random variations. The current topic also foreshadows Scientific Optimism, the scientific attitude of optimism that it is possible to make progress even on difficult problems, even if the progress is slow and iterative. RCTs are a scientific method which enormously advanced our collective capacity to advance scientific knowledge of causal relations between variables. In the scientific optimism topic, we will further explore how new scientific and mathematical methods and technologies continue to expand our ability to collect and interpret data.

Before

1.2 Shared Reality and Modeling

• Causation is a part of the shared reality and thus can be studied by empirical observation and experimentation.

2.2 Systematic and Statistical Uncertainty

• The measurement of correlation and causation is subject to both statistical and systematic uncertainty, and RCTs are designed to mitigate these uncertainties.
• Statistical uncertainty: Apparent correlation between two variables might occur simply due to randomness. An RCT should be performed on a sufficiently large representative sample.
• Systematic uncertainty: Samples are randomly assigned to either the intervention or the control group, in order to remove (by averaging out) any potential systematic differences between the two groups due to the way they are assigned.

4.1 Signal and Noise

• We can conclude a causal relationship from an RCT if there is a statistically significant difference in the dependent variable between the intervention and control groups. However, a small difference between them is inevitable due to random variations (statistical uncertainty). :* We are trying to detect a significant difference (signal), which may sometimes be difficult to distinguish from a difference that arises by random chance (noise).
• A strong signal would be a difference that is much larger than what could be expected from random chance alone.

4.2 Finding Patterns in Random Noise

• Statistical concepts such as p-value help us quantify the statistical significance of the result of an RCT—it is the probability that the observed correlation may be produced by random chance alone.
• No RCT can claim 100% confidence, but it must give a p-value, which quantifies even the tiniest possibility that the result may be a random fluke.

After

6.2 Hill's Criteria

Despite the power of RCTs in studying causal relationships, there are yet many cases in which an experimental intervention or control condition is not feasible due to resource or ethics concerns. It is still possible to extract valuable causal information from non-RCT studies using Hill's criteria.

7.1 Causation, Blame, and Policy

RCTs probe general causation about a population or a collection of phenomena, but it does not make claims about the precise causal pathway or whether a causal relationship occurs in any singular individual in this population.

Recommended Outline

Before Class

• Review Jupyter Notebook activity in time to ask any questions you may have.
• Prepare a seating chart.
• Review PlayPosit and discussion questions and ask faculty, Gabriel, or Emlen any questions you have.
• (Optional) Prepare a presentation.

During Class

• (5 min) Come up with some fun way to assign the roles of spokesperson and notetaker (e.g. earliest birthday in the year, lives furthest from campus). Remind them of the responsibilities of these roles.
• (5 min) Clicker question and review concepts from PlayPosit.
• (30 min) Work through Truffula Jupyter Notebook in pairs.
• (35 min) Work through the paper analysis activity in small groups.

Lesson Content

Clicker Question

1. Suppose there's an epidemic of Lyn's disease and a new drug is proposed as a treatment. 10,000 patients with the disease are given the drug, and 8,700 of them recover. Does this result:
   1. Give no info about the presence or absence of a causal link
   2. Establish that the treatment makes no difference.
   3. Tentatively confirm the efficacy of the treatment, though more evidence may be needed.
   4. Demonstrate conclusively the existence of an effect.

Jupyter Notebook

Datahub link

SSS Demo:

Instructions

1. (20 min) Jupyter notebook.
2. (10 min) Post-notebook questions.

Discussion Questions

1. "If the experimenter is merely observing and measuring, without actively performing an intervention themself, then it is not an RCT." Is this a correct statement?

2. Psychological studies often recruit subjects from university undergraduates. This means that their sampling is not random. How is random sampling different from randomized assignment? How does the sampling method affect the validity or conclusion of a study?

1. In the Fireside Chat with Professor O'Reilly, what did his research point to as the reason why shoelaces come untied? What steps did he take to establish causation instead of correlation?
2. Consider the following passage from the Scientific American article "How Diversity Makes Us Smarter", [1]

   "For this reason, diversity appears to lead to higher-quality scientific research. In 2014 Richard Freeman, an economics professor at Harvard University and director of the Science and Engineering Workforce Project at the National Bureau of Economic Research, along with Wei Huang, a Harvard economics Ph.D. candidate, examined the ethnic identity of the authors of 1.5 million scientific papers written between 1985 and 2008 using Thomson Reuters's Web of Science, a comprehensive database of published research. They found that papers written by diverse groups receive more citations and have higher impact factors than papers written by people from the same ethnic group. Moreover, they found that stronger papers were associated with a greater number of author addresses; geographical diversity, and a larger number of references, is a reflection of more intellectual diversity."

   The authors claim that intellectual diversity is a driving factor of higher quality scientific research. Considering what you have learned about causation and correlation, what are some other possible factors that could contribute to this observed effect?
3. Cognitive flexibility is broadly defined as "the ability to use different thinking strategies and mental frameworks". While this is applicable to many SSS concepts, in what way does cognitive flexibility assist in assessing statements about causation and correlation?

Paper Analysis (Part 1)

Students will read the abstract or skim through the text of three different scientific papers (one today and two in 6.2 Hill's Criteria) and comment on the extent to which they follow the structure of an RCT. For students who are unfamiliar with scientific papers, this will be an opportunity to demystify them.

Instructions

• (5 min) Explain activity.
• (20 min) Have the class review the paper in small groups.
• (10 min) Have the class share thoughts on the paper.

Discussion Questions

The students should try to answer the following.

1. What causal relationship is this paper trying to study? What is the hypothesis?
2. What, if it exists at all, is the experimental intervention? (i.e., What is the independent variable that is being manipulated)?
3. What is the dependent variable that is being measured (i.e., the variable the researchers anticipate may be affected by the experimental intervention)?
4. How is the independent variable manipulated? Are there control and intervention groups? Are those randomly assigned? Is the control condition a good one (only the independent variable is different, with all else kept equal)? Is this an RCT?
5. What is the result of the experiment?
6. Can the causal relationship in question 1 be concluded from the experimental results? If not, what, if anything, can be concluded? How confident are you in this conclusion?
7. Can you think of an alternative explanation for the data?

Paper

• Classic example of RCT: The Preventive Effect of the Nordic Hamstring Exercise on Hamstring Injuries in Amateur Soccer Players