12.2 Grill the Guest
More actions
Confront a working scientist about their work using course concepts.
The Lesson in Context
This is the first of several lessons involving guests. We typically invite a social scientist to share a recent published study of theirs, to be read by students ahead of time and discussed in groups during section. Questions and comments generated during the section will then be collected, a few of which will be posed to the guest during the live plenary. Students are encouraged to base their questions on particular course concepts introduced thus far.
We have a different guest and paper every year. So, the substance of this lesson can change quite substantially. Much of what's written here is an example based on what was used in UC Berkeley's Spring 2023 iteration of the course.
Takeaways
After this lesson, students should
- Be able to critically examine a piece of contemporary research and pose meaningful questions to gauge the validity and limitations of its results.
- Be able to flexibly invoke and synthesize course concepts in a real world dialogue on science and society.
The definitions for this lesson very depending on your paper and guest. These examples were used at UC Berkeley in Spring 2023.
Disfluency
- Filled Pauses
- Unfilled Pauses
Referent
Novel
Discourse-new
Principle of Contrast
Determiners
Lexical Category
Prosody
Online Word Recognition
Eye-tracking Studies
<restricted>
Useful Resources
Recommended Outline
Before Class
- Someone invites a guest speaker at least one month in advance, consulting the whole teaching staff for ideas.
- Send the article by the guest to students several days in advance and remind them to read it before the discussion section. A briefly annotated version should be made to make it easier for students to skim it and pick out the important information.
- Update the division of course topics to make sense for your version of the course.
During Class
| 7 Minutes | Short review of the paper. |
| 23 Minutes | Go through the first round of discussion. |
| 50 Minutes | Go through the second round of discussion. |
After Class
- Go through the questions the students submitted and put the best ones on a copy of the question card template. Make sure to print these and cut them out so you can pass them off to the students during plenary.
- Pose any remaining questions to the guest in writing and send a thank you email.
Lesson Content
Paper Review
Do a short review of the paper. The students should have read the annotated version of it. But, it's likely that at least some students haven't fully grasped it yet.
Discussion Round 1
Have the students discuss and come up with answers to the following questions in small groups:
- What was the motivation for this study?
2023 Answer
Prior research with adults suggests that disfluencies help facilitate sentence comprehension by signalling the presence of something new to the discourse. Researchers wanted to know if toddlers did something similar and if it plays any role in their language development.
- What are the hypotheses of this study?
2023 Answer
Whether children use indicators of processing difficulties in order to anticipate likely referents for an upcoming noun.
- What experiment(s) did the researchers perform to test these hypotheses?
2023 Answer
Two experiments main experiments involving toddlers of different ages.
- First: 2 years 4 months - 2 years 8 months.
- Second: 1 year 4 months - 2 years 2 months. (Done in two groups.)
- What are the conclusions of this study?
2023 Answer
That young children have learned that disfluencies contain information, attend to disfluencies in speech, and can make use of the information contained in disfluencies during comprehension in order to infer speaker intention.
Discussion Round 2
Have the students discuss and come up with answers to the following questions in small groups:
- What instruments were used?
2023 Answer
Primarily a Tobii 1750 eye-tracker. The eye-tracker keeps track of what the toddlers are looking at. The children don't have to wear anything to be eye-tracked. It was calibrated by having the children fixate on a shrinking dot located successively at one of five different screen locations.
- What was measured by these instruments? What's the signal? What are some sources of noise?
2023 Answer
The eye-tracker measured where the toddlers were looking on the screen. The software tracked the fraction of the time the child spent looking at the object after it was labelled.
- Signal: Where the toddler was looking.
- Noise: Anything that makes the tracker misidentify this.
- What is this used as a proxy for?
2023 Answer
What the child was actually paying attention to. This, in turn, is used as a proxy for the toddler's language acquisition. Eye tracking can be used a proxy for many different things depending on the specifics of the experimental setup. It's not a perfect proxy. But, it's one of the few ways to get at what's going on in the heads of really young toddlers.
- What intervention was performed to demonstrate causation?
2023 Answer
Participants were exposed to audio with and without disfluencies to see if their attention differed. The presence of the disfluency in the audio was the intervention.
- What are some confounding factors that the researchers controlled for? How did they control for them?
2023 Answer
How interesting the objects and their labels were.
- Novel and known objects were paired by color/shape. Labels were paired to be have similar sorts of sounds that were as easy to learn.
- An additional “look…” was added to the third presentation of the object in both fluent and disfluent audio to ensure the toddlers were prompted at the same amount of time before the window of interest.
- Prior work shows that 81% of the time, “thee” is followed by a suspension of speech.
- Disfluencies were preceded by “thee.”
- Parents wore headphones playing music to mask auditory stimuli.
- They were instructed to keep their gaze downwards during the experiment.
- What are some confounding factors that the researchers might not have controlled for?
2023 Answer
Were there unknown systematic factors between the novel and known objects? Whether the toddlers were learning the disfluencies as part of the objects' names.
- This was accounted for only in the analysis by seeing whether or not the toddlers paid less attention to the objects as the trials went on.
- Maybe parents that have the time to bring kids into studies also have more time to spend with their kids in general?
- Lots of other possible systematic biases that could affect how externally valid the study is.
- It is hard to recruit toddlers. This is not a lot of them.
- The sample is small enough that there could be systematic differences by random chance.
- What was at least one alternative hypothesis for the study's findings?
2023 Answer
Children pay more attention overall to the display during disfluencies. Children interpreted some component of the disfluency as a novel object label.
Developing Questions
Assign the groups numbers from 1 to 6. Give the groups the following instructions.
- Based on your group number, you will be given a subset of course concepts to develop questions about.
- You and your group must write at least two questions and submit them via the form on the right.
- Further instructions are provided in the form.
- Links to the form and concepts are also in today's section of the syllabus.
The relevant course concepts and group assignments are detailed below.
Group/Category 1: Getting at Reality
- 1.2 Shared Reality and Modeling
- Raft vs. pyramid
- Evaluation of models
- Scientific realism vs. anti-realism
- Operationalism, conventionalism, and realism
- 2.1 Senses and Instrumentation
- Validation of instruments through interactive exploration, triangulation with other instruments, and comparison with direct senses
Group/Category 2: Error Analysis
- 2.2 Systematic and Statistical Uncertainty
- Measurement proxies as sources of systematic and statistical uncertainty
- 3.1 Probabilistic Reasoning & 3.2 Calibration of Credence Levels
- [math]\displaystyle{ p }[/math]-values and statistical significance
- Error bars and confidence intervals
Group/Category 3: Research Quality and Refinement
- 4.1 Signal and Noise & 4.2 Finding Patterns in Random Noise
- Signal-to-noise ratio
- [math]\displaystyle{ p }[/math]-hacking
- Look elsewhere effect
- HARKing (hypothesizing after results are known)
- Effect size (as distinct from statistical significance)
- 5.1 False Positives and Negatives
- Thresholds between positive and negative detections
- Trade offs between false positives and negatives
- 5.2 Scientific Optimism
- Iterative progress
Group/Category 4: Untangling Causation
- 6.1 Correlation and Causation
- Causation as correlation under intervention
- Randomized controlled trials
- Spurious correlations
- 6.2 Hill's Criteria (Causation in the Messy Real World)
- Natural experiments
- Hill's criteria for causation
- Causal networks
Group/Category 5: Understanding Mechanisms
- 7.2 Emergent Phenomena
- Global effects that arise through the interaction of small pieces (rather than general causation)
- Explanation at different scales
- Scientific reductionism
- 8.1 Orders of Understanding
- Orders of magnitude
- Multiple causes of comparable importance
- Orders of importance of causes
- Refinement of models using higher order descriptions
Group/Category 6: Research Practice
Blind analysis is a common and easy topic to criticize on, however the students should be reminded that it is not yet widespread in many fields of science, and perfectly valid scientific results can still be obtained without it.
- 10.1 Confirmation Bias
- Selective exposure
- Biased assimilation
- Strategies for reducing confirmation bias (AOT, etc.)
- 10.2 Blinding
- Techniques for blind analysis
- Preregistration
- Registered replication
- Adversarial collaboration
- Peer review
- 11.1 Pathological Science
- Langmuir's pathological science indicators
- 11.2 When Is Science Suspect
- The validity and reliability of social science metrics
- External validity
- The difficulty of creating and applying social science metrics in cross-cultural contexts
- How researchers' biases and backgrounds shape their research agendas</restricted>