Jump to content
Toggle search
Toggle menu
75
257
179
8.5K
Sense & Sensibility & Science
Toggle preferences menu
Toggle personal menu
Not logged in
Your IP address will be publicly visible if you make any edits.

10.2 Blinding: Difference between revisions

From Sense & Sensibility & Science
More actions
← Older editNewer edit →
// Edit via Wikitext Extension for VSCode
// Edit via Wikitext Extension for VSCode
Line 5: Line 5:
{{Navbox}}
{{Navbox}}


== Useful Links ==
== The Lesson in Context ==


* [[:File:Kabob Kalimba Worksheet.pdf|Kabob Kalimba Worksheet]]
<!-- Always begin section with a description of this lesson in relation to the course as a whole. -->
* [https://docs.google.com/spreadsheets/d/1e2joE4obZttn6rKqeQUfT0GikA80iK7-fXgBM1wfDTo/edit?usp=sharing Kabob Kalimba Analysis Spreadsheet Template]
This lesson offers solutions to potential pitfalls in scientific studies raised in [[10.1 Confirmation Bias]] and [[4.2 Finding Patterns in Random Noise]]. We use a stick measurement activity to illustrate the effects of confirmation bias and to motivate techniques to reduce its effect, especially blind analysis. These techniques are not universally employed in all fields of science today, and students pursuing a scientific career are encouraged to introduce these techniques in their own work.
* [https://docs.google.com/forms/d/1qQjyLoZiJC5n6ZkpW3iwJqWyexF8jBTWf6zi4MPKj34/edit?usp=drivesdk Kabob Kalimba Response Form Template]
* [https://docs.google.com/document/d/1LeAiHBYuHHcwGbpfF4sXwykTYqiSLe2vnrCRchNZgyo/edit?usp=sharing Three Column Overview of the Week]
* [https://docs.google.com/presentation/d/1JY2QW7yJ0VhlVrLgOLWhzgnWnPhAIxs1KZ1_QZlCPkU/edit?usp=drivesdk Lesson Slides (2023 Master)]
* [https://sensesensibilityscience.berkeley.edu/topic/20 Website Page]
* [https://youtu.be/2JyOok92sCQ Tube Measurement Video]
 
=== Readings and Assignments ===


* [[:File:Hide Results to Seek the Truth - MacCoun, Perlmutter.pdf|Hide Results to Seek the Truth]]
<!-- Expandable section relating this lesson to earlier lessons. -->
{{Expand|Relation to Earlier Lessons|
{{ContextLesson|4.2 Finding Patterns in Random Noise}}
{{ContextRelation|Blind analysis is one way to prevent some forms of <math>p</math>-hacking. For example, choices to be made about a study, such as the exact statement of the hypothesis, definitions of terms, and statistical techniques, can be preregistered or made "blinded" to the data. The effects on the final result due to these choices may be hidden from the researcher during analysis. These techniques prevent motivated reasoning during analysis decisions.}}
{{ContextLesson|10.1 Confirmation Bias}}
{{ContextRelation|Blind analysis helps prevent scientists from the temptation to make choices that make the result more likely to confirm the researcher's own prediction or to match currently accepted knowledge. Otherwise, non-confirming, surprising results may be incorrectly missed.}}
}}
<!-- Expandable section relating this lesson to later lessons. -->
{{Expand|Relation to Later Lessons|
{{ContextLesson|13.1 Denver Bullet Study}}
{{ContextRelation|In group decision making, when factual evaluation and values evaluation are made by two different groups of people without knowledge of the other, it prevents evaluation motivated by the need to confirm a personal belief.}}
}}


==== Lecture Video ====
== Takeaways ==


<youtube>https://youtu.be/tovt651VVZ4</youtube>
<tabber>


== Learning Goals ==
|-|Learning Goals=


After this lesson, students should
After this lesson, students should
<!-- Learning goals are written as a numbered list. -->
# Recognize what types of blinding are useful for solving what types of errors.
# Be able to explain why blind analysis might be needed, by explaining the errors that can arise in its absence.
# Be able to explain why blind analysis might be needed, by explaining the errors that can arise in its absence.
# Recognize when blind analysis is being used and explain what function it serves. Identify situations and decisions in which blind analysis would be useful.
# Recognize when blind analysis is being used and explain what function it serves. Identify situations and decisions in which blind analysis would be useful.
Line 33: Line 39:
# Propose how to use blind analysis for simple studies.
# Propose how to use blind analysis for simple studies.


=== Definitions ===
|-|Definitions=


* '''Blind Analysis'''
<!-- Definitions must be written with the Definition and Subdefinition templates. The first Definition should have the "first=yes" flag at the end. -->
*: Making all decisions regarding data analysis before the results of interest are unveiled, such that expectations about the results do not bias the analysis. Usually co-occurs with a commitment to publicize the results however they turn out.
{{Definition|Blind Analysis|Making all decisions regarding data analysis before the results of interest are unveiled, such that expectations about the results do not bias the analysis. Usually co-occurs with a commitment to publicize the results however they turn out.|first=yes}}
* '''Preregistration'''
{{Definition|Double Blind|Studies where both the participants and the administrators of the experiment are blinded as to whether they're in the control or intervention group.}}
*: A research group publicly commits to a specific set of methods and analyses before they conduct their research.
{{Definition|Preregistration|A research group publicly commits to a specific set of methods and analyses before they conduct their research.}}
* '''Registered Replication'''
{{Definition|Registered Replication|One or more research groups commit to a specific set of methods and procedures to replicate earlier work to see if they get the same results (typically with the input of the original research team). Results are publicized regardless of outcome.}}
*: One or more research groups commit to a specific set of methods and procedures to replicate earlier work to see if they get the same results (typically with the input of the original research team). Results are publicized regardless of outcome.
{{Definition|Registered Reports|When studies are peer reviewed and journals commit to publishing before the research is undertaken. This reduces publication biases where journals prioritize interesting or statistically significant findings over null results.}}
* '''Adversarial Collaboration'''
{{Definition|Adversarial Collaboration|Scientists with opposing views agree to all the details of how data should be gathered and analyzed before any of the results are known.}}
*: Scientists with opposing views agree to all the details of how data should be gathered and analyzed before any of the results are known.
{{Definition|Peer Review|New results are evaluated by other experts in the same field to determine whether they are valid. This only reduces confirmation bias insofar as reviewers don't share the same biases.}}
* '''Peer Review'''
<br />
*: New results are evaluated by other experts in the same field to determine whether they are valid. This only reduces confirmation bias insofar as reviewers don't share the same biases.


=== Examples ===
|-|Examples=


* Fermilab's muon ''g''&minus;2 experiment performed highly precise measurements of the magnetic dipole moment of muons to test the theoretical predictions of the currently accepted model of elementary particles. Blinding is done by injecting a secret code into all of the data that would undergo analysis, so that the scientists involved would not make specific choices in the analysis in a way that makes the final value agree with the theoretical prediction. The secret code was kept in a physical locker, the opening of which was highly publicized in the announcement event. Once the data was "unscrambled", the result shows that there is indeed a sizeable deviation of the measured value from the theoretical prediction. [https://www.youtube.com/watch?v=HtdVH1Wp7fs (Short video about this process)]
<!-- Example formatting is still experimental. -->
* One way in which ''p''-hacking could occur is to choose or alter the analysis method after one has seen the results of that method to be undesirable. As an example, suppose a psychologist performs an experiment with 100 participants, sees that the results are at a statistical significance of ''p'' = 0.06, just shy of the ''p'' < 0.05 threshold for publication. They then decide to recruit another 100 participants to "improve their results", finally leading to ''p'' = 0.04, good enough for publication. This is a form of p-hacking, as p-values can dip below 0.05 as one slowly increases the sample size simply by random chance. To guard against this phenomenon, the sample size of a study is a required item in the preregistration process. See a demo of this type of p-hacking [https://rstudio-pubs-static.s3.amazonaws.com/318451_8dbb1fba8952424fb722196f98587429.html on this page].
'''Muon <math>g</math>&minus;2 Experiment'''
: This experiment performed highly precise measurements of the magnetic dipole moment of muons to test the theoretical predictions of the currently accepted model of elementary particles. Blinding is done by injecting a secret code into all of the data that would undergo analysis, so that the scientists involved would not make specific choices in the analysis in a way that makes the final value agree with the theoretical prediction. The secret code was kept in a physical locker, the opening of which was highly publicized in the announcement event. Once the data was "unscrambled", the result shows that there is indeed a sizeable deviation of the measured value from the theoretical prediction.
{{LinkCard
|url=https://www.youtube.com/watch?v=HtdVH1Wp7fs
|title=Why is the Muin <math>g</math>&minus;2 Experiment Shifting Time?
|description=A short video about this process.}}
{{Line}}
'''[Example Name]'''
: [Example Description]


=== Common Misconceptions ===
|-|Common Misconceptions=


* Students may confuse blind analysis with a double blind experiment. The latter is used primarily in treatment testing in conjunction with a placebo, such that the patient is prevented from knowing whether they received the real treatment or placebo, and the doctor is also prevented from knowing this fact in order not to inadvertently reveal this fact to the patient through subtle signs. The former type of blinding applies to the analysis process ''once the data has been collected''. In the case of treatment testing, blind analysis may be employed whether or not double blinding is.
<!-- Misconceptions must be written with the Misconception template. The first Misconception should have the "first=yes" flag at the end. -->
{{Misconception|[Description of misconception.]|[Resolution.]|first=yes}}
{{Misconception|[Description of misconception.]|[Resolution.]}}


== Context ==
</tabber>


This lesson offers solutions to potential pitfalls in scientific studies raised in [[10.1 Confirmation Bias]] and [[4.2 Finding Patterns in Random Noise]]. We use a stick measurement activity to illustrate the effects of confirmation bias and to motivate techniques to reduce its effect, especially blind analysis. These techniques are not universally employed in all fields of science today, and students pursuing a scientific career are encouraged to introduce these techniques in their own work.
== Useful Resources ==


=== Before ===
<tabber>


: '''[[4.2 Finding Patterns in Random Noise]]'''
|-|Lecture Video=
:: Blind analysis is one way to prevent some forms of p-hacking. For example, choices to be made about a study, such as the exact statement of the hypothesis, definitions of terms, and statistical techniques, can be preregistered or made "blinded" to the data. The effects on the final result due to these choices may be hidden from the researcher during analysis. These techniques prevent motivated reasoning during analysis decisions.
: '''[[10.1 Confirmation Bias]]'''
:: Blind analysis helps prevent scientists from the temptation to make choices that make the result more likely to confirm the researcher's own prediction or to match currently accepted knowledge. Otherwise, non-confirming, surprising results may be incorrectly missed.


=== After ===
<br /><center><youtube>[id]</youtube></center><br />


: '''[[13.1 Denver Bullet Study]]'''
|-|Discussion Slides=
:: In group decision making, when factual evaluation and values evaluation are made by two different groups of people without knowledge of the other, it prevents evaluation motivated by the need to confirm a personal belief.


== Recommended Outline ==
{{LinkCard
|url=[address]
|title=Discussion Slides Template
|description=The discussion slides for this lesson.
}}
<br />


=== Before Class ===
|-|Handouts and Activities=


* Prepare a seating chart.
{{LinkCard
* Do the preparation for the [[#Kabob Kalimba|kabob kalimba activity]].
|url=[address]
* Review PlayPosit and discussion questions and ask faculty, Gabriel, or Emlen any questions you have.
|title=[Title]
* (Optional) Prepare a presentation.
|description=[Description.]}}
{{LinkCard
|url=[address]
|title=[Title]
|description=[Description.]}}
<br />


=== During Class ===
|-|Readings and Assignments=


* (5 min) Come up with some fun way to assign the roles of spokesperson and notetaker (e.g. earliest birthday in the year, lives furthest from campus). Remind them of the responsibilities of these roles.
{{LinkCard
* (1 min) Inform the students of the papers to read for next week's [[11.1 Pathological Science|pathological science]] lesson.
|url=[address]
* (74 min) Spend the rest of the day on the [[#Tube Measurement|tube measurement]] activity.
|title=[Title]
|description=[Description.]}}
{{LinkCard
|url=[address]
|title=[Title]
|description=[Description.]}}
<br />


=== After Class ===
</tabber>


* Collect answers from notetakers for the forum / plenary.
== Recommended Outline ==


== Lesson Content ==
=== Before Class ===


=== Kabob Kalimba ===
[Any steps that need to be taken before class.]


This activity gives the students a chance to try and make some scientific measurement while falling into or avoiding several of the pitfalls that blind analysis could help with. In groups of three, the students will be cantilevering wooden skewers off the edge of a table and measuring the lengths at which they produce notes of different frequencies. The students' ultimate task is to find the ratio of the lengths of skewer stick (measured from the edge of the table to the end of the stick) for two notes that are an octave apart. The actual value is √2≃1.41. But, there is substantial priming to suggest to the students that the value they should expect is 1.5. If the students think of applying the methods of blind analysis, they should be able to avoid falling into this trap.
=== During Class ===


==== Preparation ====
{| class="wikitable" style="margin-left: 0px; margin-right: auto;"
|[n] Minutes
|[Activity.]
|-
|[n] Minutes
|[Activity.]
|}


# Print out at least ten copies of the [[:File:Kabob Kalimba Worksheet.pdf|Kabob Kalimba worksheet]].
== Lesson Content ==
# Acquire at least 100 wooden kabob sticks.
# Make sure you have some method to number the groups the students work in.
# Have at least one sheet of paper for students to write down their final answers on.
 
==== Instructions ====
 
# (5 min) Introduce and setup the activity.
## Go through the slides briefly introducing the activity. {{Caution|You may have to demonstrate the activity as well.}}
## Split the class into groups of three.
## Assign each group a number.
## Give every group a copy of the [[:File:Kabob Kalimba Worksheet.pdf|worksheet]] and three.
## Remind the students to try and whisper for this activity so as to not mess up other people's data.
# (30 min) Let the students work on the activity and try to get their measurements.
# (5 min) Debrief the students on the gimmick of the activity.
## Have every group write their final ratio answer down on your piece of paper along with their group number.
## Reveal the final answer and the "trick" of the activity. {{Answer|The "trick" is that there's lots of priming for the answer to be 1.5 (or 3/2) but the real value is about 1.41. The priming showed up in lots of ways. Most obviously, the chart on the spreadsheet is centered on 1.5 and worksheet suggests finding the ratio between the 2nd and 3rd notes.}}
## Display and go through the answers the students came up with and see if anyone used or could have benefitted from blind analysis.
# (40 min) Go through the discussion questions with the whole class.
 
==== Discussion Questions ====


(20 minutes)
=== Activity 1 ===


Answer these questions as a whole class.
[Brief description of and motivation for the activity]
# (1 min) What do you think the correct answer is? (Hint: It is not 2.)
{{BoxCaution|[Common misconception or thing to look out for.]}}
# (3 min) Which senses or measurement instruments did you use?
{{BoxWarning|[Thing you really need to look out for!]}}
# (4 min) What is the signal you are trying to measure, and what are the sources of noise?
{{BoxTip|title=[Title]|[Useful tip, guideline, or other background.]}}
# (4 min) What are the sources of systematic and statistical uncertainty? How did you address them?
# (4 min) What are some choices you had to make in your measurement and analysis? What are some places into which confirmation bias could have crept?
# (4 min) Did you use blind analysis? If so, how did you do it?
 
(8 min) Then have the groups discuss the following in small groups:
# Is there anything you would do differently to guard against bias in the ''measurement'' process? {{Answer|There's lots of ways to do this. But, they're all based around division of labor. The person listening for the octaves should probably not be the person using a ruler and making the measurements. You also could have someone independently make rules with different new units that convert inches or centimeters in ways that are unknown to the measurer.}}
# Is there anything you would do differently to guard against bias in the ''analysis'' process? {{Answer|This also depends on division of labor. What's most important is that the person doing the analysis has no sense of that the actual ratio values are when they're deciding what data points to include or not. The person measuring may also add a secret number to their measurements that's not revealed to the analyzer until after the analysis work is complete.}}
 
(8 min) Have the students come back and share over what they came up with as a class.
 
<!-- == Overflow ==
 
<div class="toccolours mw-collapsible mw-collapsed" style="overflow:auto;">
<div style="font-weight:bold;line-height:1.6;">Extra content that's not currently part of the official lesson plan.</div>
<div class="mw-collapsible-content">
 
=== Tube Measurement ===
 
<youtube>https://youtu.be/2JyOok92sCQ</youtube>
 
For an in-person version of this activity, students would blow some plastic tubes to produce pitches. A short (fixed-length) tube, a long (adjustable-length) tube, and some measurement tools are provided. They are asked to find the ratio of lengths of the long tube to the short tube when the pitches they produce are one octave apart. Since the tuning and measurement are both difficult, some variation is inevitable in the final result. When asked to report a final ratio, the students may be inclined to ignore certain outlying points in order to confirm the expectation that the ratio is equal to 2, even when the true ratio is closer to 2.1. (Refer to old handouts before 2019 for this version.) This lets students experience first-hand the pull of confirmation bias and motivates certain blind analysis techniques.
 
In the Zoom version of this activity, the tuning and measurement are done by the course staff and video recorded. Half the groups are given "unblinded" worksheets and the other half "blinded" worksheets. The unblinded worksheets contain the actual measured ratios and try to prime students to confirm the ratio of 2. The blinded worksheets contain the same measured ratios, but with a secret number ''X'' added to all the values, so that students have to make choices about which data points to include without seeing whether the result will be close to 2.


==== Instructions ====
==== Instructions ====


# Send the worksheets to students. Half the groups should receive the blinded worksheet and the other half the unblinded worksheets. Do not refer to them as blinded/unblinded or tell them what the difference between them is.
{| class="wikitable" style="margin-left: 0px; margin-right: auto;"
# Send the video link above to students. They may refer to parts of the video during their data analysis, but they do not need to watch the whole video.
|[n] Minutes
# (X min) Let students work on the task and offer assistance wherever needed.
|[Activity.]
# After X min, remind each group to agree on a single ratio.
|-
# Ask each group to tell you their final ratio privately, without revealing to other groups. For the blinded groups, you need to tell them that the secret number ''X'' = 1.37.{{Caution|You may alternatively use this [https://forms.gle/gMh9deSihge3Kc7T7 Google Form], filled by one person per group.}} {{Todo|Update Google Form link.}}
|[n] Minutes
# Close all breakout rooms. Reveal the difference between the blinded/unblinded worksheets and share the results of the whole class. We expect that the blinded groups would report something close to 1.41, while the unblinded groups would report something close to 1.5 due to confirmation bias.{{Caution|It's okay if this result doesn't occur. You could say that SSS students have often defied common human psychology.}}
|[Activity.]
# Ask students in the unblinded groups to describe what they were thinking when they were doing the analysis. Then ask the blinded groups.
|}


==== Discussion Questions ====
==== Discussion Questions ====


# What was the purpose of adding a secret number ''X'' to all the measured ratios for the blinded groups? {{Answer|It shifts all the numbers away from the expected value of 2, in order to prevent confirmation bias.}}
[Question 1]
# Are there other places where confirmation bias may creep in? Can you come up with other methods (during measurement or analysis) to further reduce confirmation bias?
{{BoxCaution|[Possible misconception that may need to be corrected and clarified.]|small=right}}
 
{{BoxAnswer|[Intended answer to the above question.]}}
</div></div> -->
{{Line}}
 
[Question 2]
{{NavCard|prev=10.1 Confirmation Bias|next=11.1 Pathological Science}}
{{BoxCaution|[Possible misconception that may need to be corrected and clarified.]|small=right}}
[[Category:Lesson plans]]
{{BoxAnswer|[Intended answer to the above question.]}}{{NavCard|prev=[Previous Lesson]|next=[Next Lesson]}}
<includeonly>[[Category:Lesson plans]]</includeonly>

Revision as of 08:52, 14 August 2023

Blind analysis, the practice of deciding how we will analyze data before finding out if the analysis we have chosen supports our hypothesis, counteracts confirmation bias.



The Lesson in Context

This lesson offers solutions to potential pitfalls in scientific studies raised in 10.1 Confirmation Bias and 4.2 Finding Patterns in Random Noise. We use a stick measurement activity to illustrate the effects of confirmation bias and to motivate techniques to reduce its effect, especially blind analysis. These techniques are not universally employed in all fields of science today, and students pursuing a scientific career are encouraged to introduce these techniques in their own work.

4.2 Finding Patterns in Random Noise
  • Blind analysis is one way to prevent some forms of [math]\displaystyle{ p }[/math]-hacking. For example, choices to be made about a study, such as the exact statement of the hypothesis, definitions of terms, and statistical techniques, can be preregistered or made "blinded" to the data. The effects on the final result due to these choices may be hidden from the researcher during analysis. These techniques prevent motivated reasoning during analysis decisions.
10.1 Confirmation Bias
  • Blind analysis helps prevent scientists from the temptation to make choices that make the result more likely to confirm the researcher's own prediction or to match currently accepted knowledge. Otherwise, non-confirming, surprising results may be incorrectly missed.
13.1 Denver Bullet Study
  • In group decision making, when factual evaluation and values evaluation are made by two different groups of people without knowledge of the other, it prevents evaluation motivated by the need to confirm a personal belief.


Takeaways

After this lesson, students should

  1. Recognize what types of blinding are useful for solving what types of errors.
  2. Be able to explain why blind analysis might be needed, by explaining the errors that can arise in its absence.
  3. Recognize when blind analysis is being used and explain what function it serves. Identify situations and decisions in which blind analysis would be useful.
  4. Be able to evaluate techniques (e.g., registered replication, adversarial collaboration, peer review)
    1. for ability to address confirmation bias, and
    2. in comparison to blind analysis.
  5. Propose how to use blind analysis for simple studies.

Blind Analysis

Making all decisions regarding data analysis before the results of interest are unveiled, such that expectations about the results do not bias the analysis. Usually co-occurs with a commitment to publicize the results however they turn out.

Double Blind

Studies where both the participants and the administrators of the experiment are blinded as to whether they're in the control or intervention group.

Preregistration

A research group publicly commits to a specific set of methods and analyses before they conduct their research.

Registered Replication

One or more research groups commit to a specific set of methods and procedures to replicate earlier work to see if they get the same results (typically with the input of the original research team). Results are publicized regardless of outcome.

Registered Reports

When studies are peer reviewed and journals commit to publishing before the research is undertaken. This reduces publication biases where journals prioritize interesting or statistically significant findings over null results.

Adversarial Collaboration

Scientists with opposing views agree to all the details of how data should be gathered and analyzed before any of the results are known.

Peer Review

New results are evaluated by other experts in the same field to determine whether they are valid. This only reduces confirmation bias insofar as reviewers don't share the same biases.


Muon [math]\displaystyle{ g }[/math]−2 Experiment

This experiment performed highly precise measurements of the magnetic dipole moment of muons to test the theoretical predictions of the currently accepted model of elementary particles. Blinding is done by injecting a secret code into all of the data that would undergo analysis, so that the scientists involved would not make specific choices in the analysis in a way that makes the final value agree with the theoretical prediction. The secret code was kept in a physical locker, the opening of which was highly publicized in the announcement event. Once the data was "unscrambled", the result shows that there is indeed a sizeable deviation of the measured value from the theoretical prediction.

Why is the Muin [math]\displaystyle{ g }[/math]−2 Experiment Shifting Time?

A short video about this process.

[Example Name]

[Example Description]

[Description of misconception.]

[Resolution.]

[Description of misconception.]

[Resolution.]

Useful Resources



[[address] Discussion Slides Template]

[[address] The discussion slides for this lesson.]


[[address] [Title]]

[[address] [Description.]]

[[address] [Title]]

[[address] [Description.]]


[[address] [Title]]

[[address] [Description.]]

[[address] [Title]]

[[address] [Description.]]


Recommended Outline

Before Class

[Any steps that need to be taken before class.]

During Class

[n] Minutes [Activity.]
[n] Minutes [Activity.]

Lesson Content

Activity 1

[Brief description of and motivation for the activity]

[Common misconception or thing to look out for.]

[Thing you really need to look out for!]

[Title]

[Useful tip, guideline, or other background.]


Instructions

[n] Minutes [Activity.]
[n] Minutes [Activity.]

Discussion Questions

[Question 1]

[Possible misconception that may need to be corrected and clarified.]

[Intended answer to the above question.]

[Question 2]

[Possible misconception that may need to be corrected and clarified.]

[Intended answer to the above question.]

[[[Previous Lesson]|⟨]]

[[[Previous Lesson]|Previous]]

[[[Previous Lesson]]]

[[[Next Lesson]|⟩]]

[[[Next Lesson]|Next]]

[[[Next Lesson]]]


Retrieved from "https://sensibility.gpe.fyi/index.php?title=10.2_Blinding&oldid=5940"