|
|
| (21 intermediate revisions by the same user not shown) |
| Line 1: |
Line 1: |
| [[File:Topic Cover - 12.2 Grill the Guest.png|thumb]]
| | {{Cover|12.2 Grill the Guest}} |
|
| |
|
| Confront a working scientist about their work using course concepts.
| | Students' scientific critical thinking skills are put to the test by evaluating a recent scientific study by a faculty member and "grilling" them on whether each of the concepts in this course has been carefully considered. |
|
| |
|
| {{Navbox}}
| | == The Lesson in Context == |
|
| |
|
| == Useful Links == | | <!-- Always begin section with a description of this lesson in relation to the course as a whole. --> |
| | This is the first of several lessons involving guests. We typically invite a social scientist to share a recent published study of theirs, to be read by students ahead of time and discussed in groups during section. Questions and comments generated during the section will then be collected, a few of which will be posed to the guest during the live plenary. Students are encouraged to base their questions on particular course concepts introduced thus far. |
| | {{BoxWarning|We have a different guest and paper every year. So, the substance of this lesson can change quite substantially. Much of what's written here is an example based on what was used in UC Berkeley's Spring 2023 iteration of the course.}} |
| | == Takeaways == |
|
| |
|
| * [https://docs.google.com/document/d/1Vu588U1L04MpXg876tEzTofN7NSKGnTYHhrsbkdkGTk/edit?usp=sharing Three Column Overview of the Week]
| | <tabber> |
| * [[:File:Group Question Assignment.pdf|Group Question Assignment]]
| |
| * [https://docs.google.com/forms/d/1H3Lfk0i4g8EO0byEY-tSREPsrBfAXA3cyrW5CopY6aw/edit?usp=drivesdk Question Collection Google Form Template] {{Todo|Google form link must be updated every year.}}
| |
| * [https://docs.google.com/document/d/1poTfTn6ilLO3_tMOzy_VSN_b2VYoVSKvm3z8jlWlXwY/edit?usp=drivesdk Question Card Template]
| |
| * [https://docs.google.com/presentation/d/1_IMaZzRLXov-r_WDBpoJpacYTi5NSOz16ONofx9tDbg/edit?usp=drivesdk Lesson Slides (2023 Master)]
| |
|
| |
|
| === Readings and Assignments ===
| | |-|Learning Goals= |
| | |
| * [[:File:Toddlers Use Speech Disfluencies to Predict Speakers' Referential Intentions - Kidd, White, Aslin.pdf|Toddlers Use Speech Disfluencies to Predict Speakers' Referential Intentions (2023 Paper)]] {{Todo|Paper must be updated every year.}}
| |
| | |
| {{Caution|There is no lecture video for this lesson currently as it varies so much depending on the guest.}} {{Todo|Add demo lecture to show how this runs despite the differences in guests.}}
| |
| | |
| == Learning Goals ==
| |
|
| |
|
| After this lesson, students should | | After this lesson, students should |
| | <!-- Learning goals are written as a numbered list. --> |
| # Be able to critically examine a piece of contemporary research and pose meaningful questions to gauge the validity and limitations of its results. | | # Be able to critically examine a piece of contemporary research and pose meaningful questions to gauge the validity and limitations of its results. |
| # Be able to flexibly invoke and synthesize course concepts in a real world dialogue on science and society. | | # Be able to flexibly invoke and synthesize course concepts in a real world dialogue on science and society. |
|
| |
|
| == Context ==
| | |-|Definitions= |
| | |
| This is the first of several lessons involving guests. We typically invite a social scientist to share a recent published study of theirs, to be read by students ahead of time and discussed in groups during section. Questions and comments generated during the section will then be collected, a few of which will be posed to the guest during the live plenary. Students are encouraged to base their questions on particular course concepts introduced thus far. {{Caution|We have a different guest and paper every year. So, the substance of this lesson can change quite substantially.}}
| |
| | |
| {{Caution|Blind analysis is a common and easy topic to criticize on, however the students should be reminded that it is not yet widespread in many fields of science, and perfectly valid scientific results can still be obtained without it.}}
| |
| | |
| == Recommended Outline ==
| |
| | |
| === Before Class ===
| |
| | |
| * Someone invites a guest speaker at least one month in advance, consulting the whole teaching staff for ideas.
| |
| * Send the article by the guest to students several days in advance and remind them to read it before the discussion section. A briefly annotated version should be made to make it easier for students to skim it and pick out the important information.
| |
| * Update the division of course topics to make sense for the current year.
| |
| * Review PlayPosit and discussion questions and ask faculty, Gabriel, or Emlen any questions you have.
| |
| | |
| === During Class ===
| |
| | |
| * (7 min) Short [[#Paper Review|review]] of the paper.
| |
| * (23 min) Go through the [[#Discussion Round 1|first round of discussion]].
| |
| * (50 min) Go through the [[#Discussion Round 2|second round of discussion]].
| |
| | |
| === After Class ===
| |
| | |
| * Go through the questions the students submitted and put the best ones on a copy of the [https://docs.google.com/document/d/1poTfTn6ilLO3_tMOzy_VSN_b2VYoVSKvm3z8jlWlXwY/edit?usp=drivesdk question card template]. Make sure to print these and cut them out so you can pass them off to the students during plenary.
| |
| * Pose any remaining questions to the guest in writing and send a thank you email.
| |
| | |
| == Lesson Content ==
| |
| | |
| === Paper Review ===
| |
| | |
| Do a '''short''' review of the paper. The students should have read the annotated version of it. But, it's likely that at least some students haven't fully grasped it yet.
| |
| | |
| === Discussion Round 1 ===
| |
| | |
| Have the students discuss and come up with answers to the following questions in small groups:
| |
| # What was the motivation for this study? {{Answer|'''2023 Answer:'''
| |
| #* Prior research with adults suggests that disfluencies help facilitate sentence comprehension by signalling the presence of something new to the discourse.
| |
| #* Wanted to know if toddlers did something similar and if it plays any role in their language development.}}
| |
| # What are the hypotheses of this study? {{Answer|'''2023 Answer:'''
| |
| #* Whether children use indicators of processing difficulties in order to anticipate likely referents for an upcoming noun.}}
| |
| # What experiment(s) did the researchers perform to test these hypotheses? {{Answer|'''2023 Answer:'''
| |
| #* Two experiments main experiments involving toddlers of different ages.
| |
| #** First: 2 years 4 months - 2 years 8 months.
| |
| #** Second: 1 year 4 months - 2 years 2 months. (Done in two groups.)
| |
| #* The children were seated on their parents' lap while having their eyes tracked.
| |
| #* Participants were presented objects that were either novel or familiar with accompanying fluent or disfluent audio. The experiment tested how much attention the toddlers paid to the objects given the audio.}}
| |
| # What are the conclusions of this study? {{Answer|'''2023 Answer:'''
| |
| #* That young children have learned that disfluencies contain information, attend to disfluencies in speech, and can make use of the information contained in disfluencies during comprehension in order to infer speaker intention.}}
| |
| | |
| === Discussion Round 2 ===
| |
| | |
| Have the students discuss and come up with answers to the following questions in small groups:
| |
| # What instruments were used? {{Answer|'''2023 Answer:'''
| |
| #* Primarily a Tobii 1750 eye-tracker.
| |
| #* The eye-tracker keeps track of what the toddlers are looking at. The children don't have to wear anything to be eye-tracked.
| |
| #* It was calibrated by having the children fixate on a shrinking dot located successively at one of five different screen locations.}}
| |
| # What was measured by these instruments? What's the signal? What are some sources of noise? {{Answer|'''2023 Answer:'''
| |
| #* The eye-tracker measured where the toddlers were looking on the screen.
| |
| #* The software tracked the fraction of the time the child spent looking at the object after it was labelled.
| |
| #** '''Signal:''' Where the toddler was looking.
| |
| #** '''Noise:''' Anything that makes the tracker misidentify this.}}
| |
| # What is this used as a proxy for? {{Answer|'''2023 Answer:'''
| |
| #* What the child was actually paying attention to.
| |
| #* This, in turn, is used as a proxy for the toddler's language acquisition.
| |
| #* Eye tracking can be used a proxy for many different things depending on the specifics of the experimental setup.
| |
| #* It's not a perfect proxy. But, it's one of the few ways to get at what's going on in the heads of really young toddlers.}}
| |
| # What intervention was performed to demonstrate causation? {{Answer|'''2023 Answer:'''
| |
| #* Participants were exposed to audio with and without disfluencies to see if their attention differed. The presence of the disfluency in the audio was the intervention.}}
| |
| # What are some confounding factors that the researchers controlled for? How did they control for them? {{Answer|'''2023 Answer:'''
| |
| #* How interesting the objects and their labels were.
| |
| #*: Novel and known objects were paired by color/shape. Labels were paired to be have similar sorts of sounds that were as easy to learn.
| |
| #* The length of the audio given the disfluency.
| |
| #*: An additional “look…” was added to the third presentation of the object in both fluent and disfluent audio to ensure the toddlers were prompted at the same amount of time before the window of interest.
| |
| #* The use of “the” before disfluencies in the real world.
| |
| #** Prior work shows that 81% of the time, “thee” is followed by a suspension of speech.
| |
| #** Disfluencies were preceded by “thee.”
| |
| #* Cues by the parents.
| |
| #** Parents wore headphones playing music to mask auditory stimuli.
| |
| #** They were instructed to keep their gaze downwards during the experiment.}}
| |
| # What are some confounding factors that the researchers might not have controlled for? {{Answer|'''2023 Answer:'''
| |
| #* Were there unknown systematic factors between the novel and known objects?
| |
| #* Whether the toddlers were learning the disfluencies as part of the objects' names.
| |
| #*: This was accounted for only in the analysis by seeing whether or not the toddlers paid less attention to the objects as the trials went on.
| |
| #* How toddlers were recruited. Are they from similar backgrounds?
| |
| #** Maybe parents that have the time to bring kids into studies also have more time to spend with their kids in general?
| |
| #** Lots of other possible systematic biases that could affect how externally valid the study is.
| |
| #* Sample size.
| |
| #** It is hard to recruit toddlers. This is not a lot of them.
| |
| #** The sample is small enough that there could be systematic differences by random chance.
| |
| #* This is only tested with one language. Maybe this is just a feature of English? Or is cultural?
| |
| #* Lots of other possible confounds as well…}}
| |
| # What was at least one alternative hypothesis for the study's findings? {{Answer|'''2023 Answer:'''
| |
| #* Children pay more attention overall to the display during disfluencies.
| |
| #* Children interpreted some component of the disfluency as a novel object label.}}
| |
| | |
| === Developing Questions ===
| |
| | |
| Assign the groups numbers from 1 to 6. Give the groups the following instructions.
| |
| * Based on your group number, you will be given a subset of course concepts to develop questions about.
| |
| * You and your group must write at least two questions and submit them via the form on the right.
| |
| * Further instructions are provided in the form.
| |
| * Links to the form and concepts are also in today's section of the syllabus.
| |
| The relevant course concepts and group assignments are detailed below.
| |
| | |
| ==== Group/Category 1: Getting at Reality ====
| |
| | |
| * [[1.2 Shared Reality and Modeling]]
| |
| ** Raft vs. pyramid
| |
| ** Evaluation of models
| |
| ** Scientific realism vs. anti-realism
| |
| ** Operationalism, conventionalism, and realism
| |
| * [[2.1 Senses and Instrumentation]]
| |
| ** Validation of instruments through interactive exploration, triangulation with other instruments, and comparison with direct senses
| |
| | |
| ==== Group/Category 2: Error Analysis ====
| |
| | |
| * [[2.2 Systematic and Statistical Uncertainty]]
| |
| ** Measurement proxies as sources of systematic and statistical uncertainty
| |
| * [[3.1 Probabilistic Reasoning]] & [[3.2 Calibration of Credence Levels]]
| |
| ** ''p''-values and statistical significance
| |
| ** Error bars and confidence intervals
| |
| | |
| ==== Group/Category 3: Research Quality and Refinement ====
| |
| | |
| * [[4.1 Signal and Noise]] & [[4.2 Finding Patterns in Random Noise]]
| |
| ** Signal-to-noise ratio
| |
| ** ''p''-hacking
| |
| ** Look elsewhere effect
| |
| ** HARKing (hypothesizing after results are known)
| |
| ** Effect size (as distinct from statistical significance)
| |
| * [[5.1 False Positives and Negatives]]
| |
| ** Thresholds between positive and negative detections
| |
| ** Trade offs between false positives and negatives
| |
| * [[5.2 Scientific Optimism]]
| |
| ** Iterative progress
| |
| | |
| ==== Group/Category 4: Untangling Causation ====
| |
| | |
| * [[6.1 Correlation and Causation]]
| |
| ** Causation as correlation under intervention
| |
| ** Randomized controlled trials
| |
| ** Spurious correlations
| |
| * [[6.2 Hill's Criteria]] (Causation in the Messy Real World)
| |
| ** Natural experiments
| |
| ** Hill's criteria for causation
| |
| ** Causal networks
| |
| | |
| ==== Group/Category 5: Understanding Mechanisms ====
| |
| | |
| * [[7.2 Emergent Phenomena]]
| |
| ** Global effects that arise through the interaction of small pieces (rather than general causation)
| |
| ** Explanation at different scales
| |
| ** Scientific reductionism
| |
| * [[8.1 Orders of Understanding]]
| |
| ** Orders of magnitude
| |
| ** Multiple causes of comparable importance
| |
| ** Orders of importance of causes
| |
| ** Refinement of models using higher order descriptions
| |
|
| |
|
| ==== Group/Category 6: Research Practice ==== | | <!-- Definitions must be written with the Definition and Subdefinition templates. The first Definition should have the "first=yes" flag at the end. --> |
| | {{BoxWarning|The definitions for this lesson very depending on your paper and guest. These examples were used at UC Berkeley in Spring 2023.}} |
| | {{Definition|Disfluency|Pauses that disrupt normal speaking patterns.}} |
| | {{Subdefinition|Filled Pauses|When the pause is filled with an utterance like "uh" or "um."}} |
| | {{Subdefinition|Unfilled Pauses|When no utterance fills the pause. Note that unfilled pauses are not always disfluencies.}} |
| | {{Definition|Referent|The object that a given set of sounds refers to.}} |
| | {{Definition|Novel|An object or word that has not ever been encountered before.}} |
| | {{Definition|Discourse-new|An object or word that has not previously come up in a given discourse. These may or may not be novel.}} |
| | {{Definition|Principle of Contrast|The idea that novel objects will be attached to novel words.}} |
| | {{Definition|Determiners|Words that identify or distinguish a referent without describing or modifying it. For example, the word "the."}} |
| | {{Definition|Lexical Category|Type of word.}} |
| | {{Definition|Prosody|The patterns of stress and intonation in someone's speech.}} |
| | {{Definition|Online Word Recognition|Word recognition that's happening live (in the moment) as opposed to figuring out words afterwards.}} |
| | {{Definition|Eye-tracking Studies|A class of studies that measure where people look or how their gaze shifts. These are common throughout psychology, psycholinguistics, and cognitive science research as a proxy for people's behavior, attention, and cognition.}} |
| | <br /> |
|
| |
|
| * [[10.1 Confirmation Bias]]
| | </tabber> |
| ** Selective exposure
| |
| ** Biased assimilation
| |
| ** Strategies for reducing confirmation bias (AOT, etc.)
| |
| * [[10.2 Blinding]]
| |
| ** Techniques for blind analysis
| |
| ** Preregistration
| |
| ** Registered replication
| |
| ** Adversarial collaboration
| |
| ** Peer review
| |
| * [[11.1 Pathological Science]]
| |
| ** Langmuir's pathological science indicators
| |
| * [[11.2 When Is Science Suspect]]
| |
| ** The validity and reliability of social science metrics
| |
| ** External validity
| |
| ** The difficulty of creating and applying social science metrics in cross-cultural contexts
| |
| ** How researchers' biases and backgrounds shape their research agendas
| |
| {{BoxCaution|Blind analysis is a common and easy topic to criticize on, however the students should be reminded that it is not yet widespread in many fields of science, and perfectly valid scientific results can still be obtained without it.}}
| |
|
| |
|
| {{NavCard|prev=12.1 Wisdom of Crowds and Herd Thinking|next=13.1 Denver Bullet Study}} | | {{#restricted:{{Private:12.2 Grill the Guest}}}} |
| | {{NavCard|chapter=Lesson plans|text=All lesson plans|prev=12.1 Wisdom of Crowds and Herd Thinking|next=13.1 Denver Bullet Study}} |
| [[Category:Lesson plans]] | | [[Category:Lesson plans]] |
