Quiz questions

Fact or value

Relevant topic(s): 1.1 Introduction and When Is Science Relevant

In an international forum on climate change, the following statements are made by various parties across the globe. Identify if each of these statements is a claim of fact or a claim of value.

1. Leader of a major first world country: “The current change in global climate is not caused by human industrial activity.” (Fact/Value)
2. A climate activist: “A world whose economy is built around environmental consciousness rather than corporate greed is a world that’s worth living in.” (Fact/Value)
3. An economist: “By my conservative estimate, the clean energy sector will create 1 million jobs within the next two years.” (Fact/Value)
4. An industry lobbyist: “More than half the population would prioritize maintaining job security over reducing the risks of climate change.” (Fact/Value)

Genetic engineering conference

Relevant topic(s): 1.2 Shared Reality

At a conference on genetic engineering, two panelists, Dr. Hallstein and Dr. Yamazawa, are having a heated debate over the effectiveness of gene editing of mosquitos in reducing the spread of malaria. Dr. Hallstein thinks the approach can drastically reduce disease spread, whereas Dr. Yamazawa argues that it has negligible effect. How can the attendees of the conference decide who is correct in reality? Choose one.

1. They cannot. Both panelists are correct to themselves, and they should agree to disagree.
2. The attendees should take a vote. The claim receiving more than 50% of the votes is the correct one.
3. They should defer to the most reputable senior scientist at the conference.
4. They cannot yet, but they can continue to gather more evidence to see which opinion is more likely to be correct.

Instruments you use

Relevant topic(s): 2.1 Senses and Instrumentation

1. List two instruments for perception that you use regularly, which extend your observation beyond direct perception through your senses.
2. Choose one instrument from the two you wrote above. By making reference to the concept of interactive exploration (as described by Ian Hacking in the reading about instrumentation), give one reason why you trust it.
3. List one possible problem that could cause this instrument to lead you awry. Explain briefly how you would know that it misled you about reality.

Close encounters of the third kind

Relevant topic(s): 2.1 Senses and Instrumentation

You’re casually walking down Telegraph Avenue when you bump into an alien from a different planet. It turns out that they don’t have ears to hear sounds, but do have eyes to see. What is an instrument you could give them to show that sounds exist, and how could they use interactive exploration to confirm that sounds are indeed real and not suspicious human trickery?

Sleep studies

Relevant topic(s): 3.1 Systematic and Statistical Uncertainty

Many psychological studies are performed on university undergraduates who volunteer as experimental subjects in exchange for academic credit or cash. Suppose one such study attempts to measure the average amount of sleep humans have per week by asking 10 random students on Sproul Plaza to recall their sleep times in the past week.

1. Name one source of systematic uncertainty in such a measurement.
2. How would you improve the study to reduce this systematic uncertainty?

Red pen black pen

Relevant topic(s): 2.2 Causation and Correlation

Suppose you want to find out whether writing with a red pen causes people to write faster. Which of the following would be the most effective test of this hypothesis? (Not necessarily perfect, just most effective.) Assume, for each study, that the measure of speed is how many letters (i.e. alphabet letters) they write in 5 minutes, when asked to write a description of their day.

1. Randomly choose 1,000 undergraduates, give each participant a red pen and a black pen, let them choose which pen to use, and ask them to write about their day.
2. Take 200 undergraduates, randomly hand out 100 red pens and 100 black pens, so that each participant has one pen, and have them write about their day with the pen they are given.
3. Randomly choose two cities in different parts of the world, randomly choose 3,000 people in each of the two cities, give the participants in one city red pens and the participants in the other city black pens to write about their day.
4. Pick 10,000 random people from all around the world and ask whether they tend to write with a red pen or a black pen. Then have them write for five minutes about their day.
5. Find a pair of identical twins who grew up together and randomly give one twin a red pen and the other twin a black pen to write about their day.

Self-driving car

Relevant topic(s): 3.2 Signal and Noise

1. In a newly developed self-driving car, the onboard artificial intelligence (AI) uses several cameras mounted on various parts of the car (and no other instruments) to view the surroundings and make decisions. To this AI driving system, is each of the following a signal, a source of noise, or neither?
   1. The constant static sound from the car radio (Signal/Noise/Neither)
   2. The dust buildup on the camera lens (Signal/Noise/Neither)
   3. A pedestrian about to cross the road (Signal/Noise/Neither)
   4. Reflections of the street lamps' light on a wet road (Signal/Noise/Neither)
   5. A stop sign (Signal/Noise/Neither)
2. It is well known that driving on a rainy night is very dangerous. In the language of signal-to-noise ratio, why is driving on a rainy night more dangerous than on a sunny day?

Signal-to-noise ratio

Relevant topic(s): 3.2 Signal and Noise

Which of the following statements about signal-to-noise ratios (SNRs) is/are correct?

1. Detecting if there is an elephant in my living room has a low SNR, because there is no elephant in my living room.
2. There is a higher SNR for detecting an elephant in my living room compared to detecting an elephant in my neighbour's living room through a pair of binoculars.
3. For observing the surface of the moon, the SNR is higher at night than in the daytime because the signal is stronger at night.
4. For a predator, a camouflaged animal has a lower SNR because there is higher noise in its surroundings.
5. Your roommate is talking loudly on the phone while you are studying, and your mind keeps being drawn towards their conversation. You play loud music to drown out their voice. This lowers the SNR.

Twin adoption

Relevant topic(s): 4.1 Finding Patterns in Random Noise

Identical twins separately adopted by different families are extremely valuable to biologists and psychologists in experiments that determine whether a physical trait or behavior is due to nature or nurture. One psychologist is trying to determine whether being raised in a high-income family (earning more than $400,000 a year) causes bipolar disorder. They study 10,000 pairs of adult identical twins that were separated at birth and adopted by families from a range of financial backgrounds.

1. If causation is equivalent to “correlation under intervention”, what simple observational result would demonstrate that “being raised in a high-income family causes bipolar disorder”?
2. Describe how each of the criteria for a randomized controlled trial is satisfied by such a study:
   1. Intervention
   2. Control condition
   3. Random assignment
3. From these 10,000 pairs of twins, this psychologist did not find that being raised in a high-income family causes bipolar disorder. One of the other psychologists suggested, “it is so hard for us to get 10,000 pairs of identical twins. Why don’t we use the same 10,000 pairs to study whether being raised in a high-income family causes any other type of mental disorder, and then only publish the result that confirms a causation?” If you were the first psychologist, should you say this is a good idea? Why or why not?

Crush

Relevant topic(s): 4.2 False Positives and Negatives

Lynn has a crush on her friend Harriet, and she suspects but doesn’t know if Harriet returns her interest. If a false positive is Lynn thinking Harriet likes her when she doesn’t…

1. What’s a false negative in this situation, and what are the costs of a false negative?
2. What are the costs of a false positive in this situation?

NFL concussions

Relevant topic(s): 5.2 Hill's Criteria

Suppose you find out that former National Football League (NFL) players who had more concussions are more likely to have chronic traumatic encephalopathy (CTE), which is a form of long-term brain damage causing various cognitive and behavioral problems. One candidate hypothesis is that concussions cause CTE.

1. Why would a randomized controlled trial be difficult or impossible to perform?
2. Choose two forms of evidence listed below​, and say what question you would need to answer to gather each form of evidence (in trying to find out whether concussions cause CTE). We have given you the first answer to the first criterion as an example. Answer two of the following:

   Example: Dose-response curve (a.k.a. Biological gradient)

   Does a higher frequency of playing football correlate with higher rates of CTE?

   Plausible Mechanism

   Specificity

   Consistency Across Different Contexts

   Temporality (temporal sequence of cause and effect)

3. What is an alternative hypotheses for the correlation between concussions and CTE in former NFL players?

Honorary mentions

Relevant topic(s): 6.1 Probabilistic Reasoning

Medical journals usually have a standard of only publishing results that have a p-value less than 0.05 (i.e. probability of the same result simply due to random noise is less than 5%). Suppose a reputable journal recently published a special edition, Honorary Mentions, which includes all the articles with a p-value less than 0.10 but not quite below the 0.05 standard. As a reader, which of the following attitudes should you take about articles in Honorary Mentions?

1. Only a small fraction of those results are probably correct, and it’s hard to tell which ones, so I shouldn’t believe any of the results there.
2. Since the results are published by a reputable journal, I should believe them until another article proves them wrong.
3. The scientists must not have done a good enough job to bring the p-value down to below 0.05. I shouldn’t believe any of the results there.
4. The results there are more likely correct than not, so they are still somewhat useful, but you wouldn’t want to bet your life on them.
5. These are essentially the “rejected” articles, so I shouldn’t believe any of them.

Rival pollsters

Relevant topic(s): 6.2 Calibration of Credence Levels

Two groups of pollsters regularly publish their predictions of local election results in the US, along with the credence levels associated with the predictions. Their predictions over the past 4 years are summarized as follows.

Polling Firm A

Credence level	Number of predictions made with this credence level	Number of predictions with this credence level that came true
50-60%	29	16
60-70%	30	19
70-80%	33	26
80-90%	24	20
90-100%	32	31
Total true predictions		102

Polling Firm B

Credence level	Number of predictions made with this credence level	Number of predictions with this credence level that came true
50-60%	29	23
60-70%	32	26
70-80%	25	25
80-90%	22	21
90-100%	36	35
Total true predictions		130

1. Which firm is better calibrated?
   1. Polling Firm A
   2. Polling Firm B
2. For the less well calibrated pollster, are they underconfident or overconfident?
   1. Underconfident
   2. overconfident
3. Justify your answers using the table of numbers above.

Mrs. Hewlett's math test

Relevant topic(s): 6.2 Calibration of Credence Levels

Mrs. Hewlett’s class took a math test.

• Joe doesn’t know how to count, so he thought he wouldn’t get any questions right. He didn’t bother trying, writing in “Joe” for every answer. He did not get any right.
• Allison was great at math, but not sure of herself. She thought she’d only get 85% of the problems correct, but really she got a perfect score.
• Joy reasoned that if she always chose C she would get about 20% correct. Sadly it was not a multiple choice test, so she got none correct.
• Benjamin by and large had a pretty good sense of his own abilities. He thought he would get about 80% correct, but actually only got about 75% correct.

1. Who is best calibrated?
   1. Joe
   2. Allison
   3. Joy
   4. Benjamin
2. Whose calibration is most underconfident?
   1. Joe
   2. Allison
   3. Joy
   4. Benjamin

The 🍌phone

Relevant topic(s): 7.1 Orders of Understanding

You’re a manufacturer of banana-shaped satellite phones. The main components of the satellite phone currently have the following costs:

Battery	$10.69
Screen	$96.50
Processor and memory	$910.10
Sensors, holding material, and assembly	$1,050.88
Triple camera	$103.50
Miscellaneous	$20.01

Why is this phone so expensive? Give the first-order cause(s). Select all that apply.

1. Battery
2. Screen
3. Processor and memory
4. Sensors, holding material, and assembly
5. Triple camera
6. Miscellaneous

Jefe Pesos

Relevant topic(s): 7.2 Fermi Problems

Jefe Pesos, an (hypothetical) American billionaire, earned US$100 billion (that’s US$100,000,000,000) in the first year of the Covid-19 pandemic. Suppose he could spend all this money. Would he have been able to pay for the basic food costs for the poorest 10% of children in the US, for whom hunger is a constant concern, for one year? Use Fermi estimation. (Please show your working and write down all your assumptions. You will not be marked by the accuracy of your estimates, but by the process of your calculation.)

Base-rate neglect

Relevant topic(s): 8.1 Heuristics and Biases I

1. Which of the following statements engage(s) in valid reasoning? (That is, the conclusion follows from the premise, and it does not commit the fallacy of base-rate neglect.) Select all that apply.
   1. 98% of Vietnamese people have some level of lactose intolerance. Hong Thuy, our new flatmate from Vietnam, is therefore most likely lactose intolerant.
   2. Less than 1% of all pet bite injuries are due to cats. If you have a pet cat, you are therefore very unlikely to be bitten by it.
   3. This test for colorectal cancer has a 12% false positive rate. If you don’t have colorectal cancer, the test will likely report as negative.
   4. 60% of patients who are admitted to the hospital for a highly contagious (hypothetical) disease were already vaccinated against this disease. You are more likely to end up in the hospital if you are vaccinated than if you are not vaccinated.
2. For one of the statements above that commits base-rate neglect, name a piece of information about the base rates that would render the conclusion incorrect.

Heuristics and biases

Relevant topic(s): 8.1 Heuristics and Biases I, 8.2 Heuristics and Biases II

In each of the following scenarios, which heuristic is most likely at play? (Availability heuristic, representativeness heuristic / conjunction fallacy, anchoring heuristic, peak-end rule)

1. After watching a video compilation of escalator accidents, Kylie now has a fear of escalators, believing that they are prone to deadly mechanical failures.
2. In a government budget meeting, a politician insists that low-income people are not the source of criminal offences, but that more budget should instead be spent to solve a much bigger problem—crimes committed by poor migrant workers.
3. While choosing an occupational path after his graduation, Andrew thinks, “wood logging sounds like a rather dangerous job, but it’s nowhere near as dangerous as being a police officer, otherwise I would’ve heard about logging accidents more often on the news.”
4. Write a couple more about status quo bias, fundamental attribution error, and temporal discounting.

EmDrive

Relevant topic(s): 9.1 Pathological Science

EmDrive is a proposed device that generates thrust just by reflecting microwaves internally, surprisingly violating the conservation of momentum. It garnered widespread attention from the scientific and engineering community, many of whom have devoted resources to developing and validating such a device. In 2016, NASA's Jet Propulsion Laboratory announced that they observed a small apparent thrust from an EmDrive that they had constructed. The result was published after a year of peer review by 5 referees. However, other researchers had since failed to reproduce this result, and most did not see an effect larger than the experimental margin of error itself. In 2021, the 2016 study was found to have omitted to take into account the effect due to the Earth's magnetic field, rendering the result illusory.

Langmuir’s criteria for pathological science:

1. The effect is produced by a barely detectable cause, and the magnitude of the effect is substantially independent of the intensity of the cause.
2. The effect is barely detectable or has very low statistical significance.
3. Claims of great accuracy.
4. Involving fantastical theories contrary to experience.
5. Criticisms are met with ad hoc excuses.
6. Ratio of supporters to critics rises to near 1:1, then drops back to near zero.

1. It is not always clear when something is a case of pathological science. Name one of Langmuir’s indicators (just write the number) that is present in this example and explain your answer.
2. Name one of Langmuir’s indicators (just write the number) that is NOT present in this example and explain your answer.

Seuss' sneetches

Relevant topic(s): 9.2 When Is Science Suspect

There are two communities of sneetches living in the same town—those with a star pattern on their bellies and those without. The star-bellied sneetches have done lots of research on starless sneetches. Of the following studies, knowing nothing else, which should you be most skeptical about?

1. A study on the swimming speed of starless sneetches
2. A study on the sophistication of speech of starless sneetches
3. A study on the starless sneetches’ susceptibility to sun-sensitive sneezes
4. A study on the starless sneetches’ satisfaction with street sweeping services
5. A study on the starless sneetches’ sleep schedules

Confirmation bias

Relevant topic(s): 10.1 Confirmation Bias

In 1997, Keith Stanovich and Richard West developed a measure called the Argument Evaluation Test (AET) for how well one can evaluate the quality of an argument independent of one’s prior beliefs. They first asked individuals how much they agreed or disagreed with each of the 23 topic statements (concerning gun control, taxes, crime, etc.). Then, they asked the individuals to rate the quality of a series of arguments made by a fictional person, Dale, for or against the same 23 topics statements. Stanovich and West have found strong evidence for confirmation bias in this measure.

1. What pattern of data would Stanovich and West observe to demonstrate that participants engage in confirmation bias?
2. Is this an example of (circle one):
   1. Biased assimilation
   2. Selective exposure

Blind analysis

Relevant topic(s): 10.2 Blind Analysis

Christa is trying to see which of two medications works better. She divides her patients into two groups, a treatment group and a control group. Which of the following variants (none of which are ideal) employs blind analysis?

1. Both Christa and the patients know who is in which group. Christa looks at the labelled data as she makes statistical choices, such as which outliers to exclude.
2. Christa knows which patients are in which group, but the patients themselves do not know. Christa looks at the labelled data as she makes statistical choices, such as which outliers to exclude.
3. Both Christa and the patients know who is in which group. However, Christa’s data is not labelled when she makes choices during statistical analysis, such as which outliers to exclude.
4. Neither Christa nor the patients know who is in which group. Christa looks at the labelled data as she makes statistical choices, such as which outliers to exclude.
5. Neither Christa nor the patients know who is in which group. Christa wears a blindfold while she performs statistical analysis.

Singular and general causation

Relevant topic(s): 11.1 Singular and General Causation

From the following statements, select the one(s) that is/are about general causation.

1. The El Dorado wildfire in 2020 was caused by a smoke bomb at a gender reveal party.
2. Ariel’s back injury was caused by lifting heavy weights with bad posture at the gym earlier today.
3. Inhalation of radon from radioactive minerals underground is the second leading cause of lung cancer.
4. A reduction in gun violence can be achieved by requiring gun owners to first take and pass a safety and competency test.
5. Chantelle’s award of a special exploratory grant enabled her discovery of a new molecule.

Wisdom of crowds

Relevant topic(s): 12.1 Wisdom of Crowds and Herd Thinking

In which of the following scenarios would you expect the “wisdom of crowds” effect to have the largest positive impact on the result (i.e. to get closer to the true answer)? Estimating the heaviest recorded weight of a pumpkin in a large college classroom by having the students shout their own guesses one after another

1. Estimating the number of traffic lights in San Francisco by selecting a random sample of the US population, having them discuss the question in groups, and then averaging their individual guesses
2. Estimating the height of the empire state building by selecting a random group of university students, and having them discuss the question and agree on a collective guess
3. Estimating the largest recorded weight of a tiger by selecting a random sample of the world population, and having them discuss the question in groups and agree on a collective guess
4. Estimating the height of the tallest tree in the US by selecting a large, random sample of the US population and averaging their individual guesses

High-speed rail

Relevant topic(s): 12.2 Denver Bullet Study

Suppose you are a government official tasked with developing a plan for a new high-speed railway system across your country. Several foreign countries, including China, France, Germany, and Japan, have already offered development plans using their own high-speed rail technology under significant foreign investment. You have also thought about developing the system entirely domestically to boost the national economy, but this will certainly take longer than foreign development plans.

You remember Prof. Campbell’s lecture on the Denver bullet study from your days at Berkeley and hope to use it to make a decision by integrating facts and values.

1. Name two factors or features of the high-speed rail plans that will play a significant role in the decision making process.
2. Name two stakeholders and two types of experts for this situation.
3. Briefly explain how the Denver bullet study technique produces or outputs a final decision on the optimal high-speed railway plan.