4.2 Finding Patterns in Random Noise

Learning Goals

[Link to PlayPosit]

More details

After this lesson, students should

1. Understand that people tend to see any regularity as a pattern (i.e., see more signal than there is), even when “patterns” occur by chance (i.e. are pure noise).
2. Resist the opposing temptations of both the Gambler’s Fallacy (the expectation that a run will soon break and quickly balance out) and the Hot-Hand Fallacy (the expectation that a run will continue).
3. Recognize the Look Elsewhere Effect: If there is a low probability of obtaining a false positive in any given instance, the more times you try (the more questions you ask, measures you take, or studies you run without statistical correction), the more you increase the probability of getting a false positive. This occurs when one:
   1. Asks too many questions of the same data set, reporting only statistically significant results.
   2. Asks the same question of multiple data sets, reporting only statistically significant results.
   3. Runs a test or similar tests too many times, reporting only statistically significant results.
   4. This also occurs in everyday life, e.g. when one looks at a whole lot of phenomena and only takes note of the most surprising-looking patterns, not properly taking into account the larger number of unsurprising patterns/lack of pattern.

Definitions

• Look Elsewhere Effect

  If there is a low probability of obtaining a false positive in any given instance, the more times you try (the more questions you ask, measures you take, or studies you run without statistical correction), the more you increase the probability of getting some false positive. The effect occurs when one:

  • Asks too many questions of the same data set, reporting only statistically significant results.
  • Asks the same question of multiple data sets, reporting only statistically significant results.
  • Runs a test or similar tests too many times, reporting only statistically significant results.
  • This also occurs in everyday life, e.g. when one looks at a whole lot of phenomena and only takes note of the most surprising-looking patterns, not properly taking into account the larger number of unsurprising patterns/lack of pattern.

• p-value

  The statistic used most often as a measure of statistical significance. The probability of getting a result as extreme or more if in fact the hypothesis is false, simply through random noise. The typical cut-off for a statistically significant p-value is p<.05.

• p-hacking

  A subset of the Look Elsewhere Effect that occurs when people conduct multiple statistical tests and only report those with p-values over .05 (the traditional threshold for publication and statistical significance, which indicates a tolerance of 5% false positives). Common techniques that constitute p-hacking include:

  • Running different statistical analyses on the same dataset and only reporting the statistically significant ones.
  • Analyzing multiple DVs and only reporting the statistically significant ones.
  • Gradually increasing the sample size until the p-value falls below .05.

• Gambler's Fallacy

  Expecting that streaks (e.g. Tails Tails Tails Tails) will be broken, such that future results will quickly “average out” earlier ones, even when all trials are independent.

• Hot-hand Fallacy

  Expecting that streaks (e.g. winning hands in Poker) will continue, even when all trials are independent.

• Statistical Significant

  How unlikely a given set of results would be if the null hypothesis were true (i.e. if the hypothesized effect did not actually exist).

Examples

• Mr. Goxx: A hamster that actively manages a cryptocurrency portfolio by running in his “intention wheel” to determine what he’s buying/selling and then goes through either a “BUY” or “SELL” decision-tunnel to decide what he’s doing with it. As of october 2021, his portfolio was up nearly 30% from june when he started trading crypto. His decisions are streamed live on Twitch.
• Cold reading in a crowd: A medium shouts out to a crowd somewhat common names, such as William and Butler, and common ailments such as, “heart problem” or “passed in their sleep”. It is very likely that a couple people in the crowd can match a majority of these features to someone in their life, making the claims seem impressively accurate (seemingly small p-value), but this ignores all the other members of the audience who could not match any of the claims. This is an illustration of p-hacking.
• p-hacking through incrementing sample size: “We conducted the study on 1000 participants, and our p-value is just slightly above 0.05. Let’s recruit another 20 participants to see if our p-value can dip under 0.05.” If they keep increasing their sample size by 20 each time, the p-value will fluctuate just from chance alone, possibly dipping below 0.05 even if there isn’t a real signal. This is equivalent to only selecting a subsample of the data that would confirm a hypothesis, omitting that an even larger sample would have rejected the same hypothesis.

Common Misconceptions

• Look Elsewhere Effect means that more data leads to more misinterpretations, or "too much data is bad".

  In fact, more data reduces statistical uncertainty and thus strengthens inferences. What can lead to more misinterpretations through the Look Elsewhere Effect is when people ask too many statistical questions of the dataset without correcting for multiple comparisons.

• More separate analyses someone runs, the better their analysis will be.

  In fact, the more analyses someone runs, the more likely they are to hit upon a false positive, unless they correct for multiple comparisons (which can be done statistically).

• Wow, that was such a striking coincidence! It must have some hidden significance.

  Even very striking coincidences are bound to happen if you look across enough events. We forget how many events we have looked through, and remember only the coincidence. (See 10.1 Confirmation Bias.)

Context

This lesson continues 3.2 Signal and Noise by elaborating on ways in which random noise can emulate signals (produce apparent patterns) in many different contexts. We introduce the idea of p-values to quantify the statistical significance of patterns and describe various tempting statistical fallacies we tend to make as laypersons or scientists, such as gambler’s fallacy and p-hacking. We play a game in which students try to produce a random string of coin tosses by thought, which reveals that a truly random string in fact contains more apparent patterns than one intuitively expects. Two other activities also illustrate how spurious patterns are in fact expected to arise from random noise.

Before

3.2 Signal and Noise

• A signal is a particular pattern of the data that we are seeking, while noise is something that introduces uncertainty or error into the measurement of that data, in a way that sometimes produces spurious signals.
• p-values are one way to quantify how statistically significant a measured signal is compared to the noise. It is defined as the probability that the measured signal is produced entirely by random noise alone even when the underlying cause of a signal is absent.

After

8.2 Heuristics and Biases II

Cognitive heuristics and biases such as confirmation bias may mislead us into seeing a pattern in random data where there is none.

9.2 Pathological Science

p-hacking is one source of pathological science, where authors fail to disclose the measurements in which the supposed signal is not observed, thereby falsely inflating the statistical significance of the reported signal.

Recommended Outline

Before Class

• Prepare a seating chart.
• Review PlayPosit and discussion questions and ask faculty, Gabriel, or Emlen any questions you have.
• Familiarize yourself with the Google Doc [link …] for the Fool the Professor game.
• Get enough pennies for the whole class for the Fool the Professor game.
• Print handouts for Stock Prediction and Snowy Pictures activities.
• Prepare slides for Snowy Pictures activity.
• Familiarize yourself with the Jupyter notebook.
• (Optional) Prepare a presentation.

During Class

• (5 min) Come up with some fun way to assign the roles of spokesperson and notetaker (e.g. earliest birthday in the year, lives furthest from campus). Remind them of the responsibilities of these roles.
• (20 min) Discussion questions.
• (2 min) Clicker question.
• (15 min) Fool the Professor activity.
• (8 min) Stock predictions activity.
• (4 min) Snowy Pictures activity.
• (26 min) Use the remaining time to start work on the p-hacking notebook.

After Class

• One GSI should compile histograms for the Fool the Professor game.
  1. Separately for each of the two strings of coin tosses, tally the number of times a consecutive run of N heads or tails occurs. (This may be done with a script or the search-and-replace function in a text editor.)
  2. Mask which string is the true random one by labelling them A and B.
  3. Make the following two histograms for the tallies of consecutive runs, stacking the tallies for heads and tails.

Lesson Content

Clicker Question

1. Question
   1. Option 1
   2. Options 2

Activity 1: Name

[Brief description of and motivation for the activity]

Discussion Questions

1. Question 1
   1. Subquestion a

1. 1. Subquestion b

Collect Questions for Plenary

(5 min) Collect remaining questions from the students for faculty in plenary (can be questions for clarification, extension, discussion, etc.), and add [ here].