12.1 Wisdom of Crowds and Herd Thinking

Useful Links

• Jury Bias and Composition Papers
  • Reducing the Effects of Juror Bias (Kaplan and Miller, 1978)
  • Effects of Victim Attractiveness, Care and Disfigurement on the Judgements of American and British Mock Jurors (Kerr, Bull, MacCoun and Rathborn, 1985)
  • The Emergence of Extralegal Bias During Jury Deliberation (MacCoun, 1990)
  • Bias in Jurors vs Bias in Juries: New Evidence from the SDS Perspective (Kerr, Niedermeier and Kaplan, 1999)
• Estimates Survey Template
• Estimates Spreadsheet Example
• Three Column Overview of the Week
• Lesson Slides (2023 Master)
• Website Page

Readings and Assignments

• PlayPosit Video
• The Social Psychology of the Wisdom of Crowds

Learning Goals

After this lesson, students should

1. Not take for granted that consensus offers the best conclusions.
2. Take seriously (but not as absolute!) the consensus of a group which has reasoned about a question in a careful, appropriate way.
3. Take seriously (but not as absolute!) the average of a large group's independent estimates of a number, under appropriate conditions.
4. Identify shared biases in a given group which may increase the odds of problematic herd thinking rather than helpful wisdom of crowds.

Definitions

• Wisdom of Crowds

  Sometimes groups make better judgments than individuals. This happens when:

  • Judgments are genuinely independent, preventing herd thinking.
  • Members of the group do not share the same biases.
  • There are enough people in the group to balance out random biases or fluctuations (analogous to the need for an adequate sample size).
  • Works especially well when estimating a quantity, where errors may be large but are not systematic.

• Herd Thinking

  Sometimes groups make worse judgments than individuals. This happens when:

  • Judgments of individuals are influenced by the judgments of others, leading to groupthink and sometimes polarization.
  • Members of the group share biases, which can be exaggerated by discussion and cannot be decreased by averaging judgments.

• Meta-analysis

  A statistical analysis that aggregates the results from many independent studies addressing the same question. Each individual study is expected to have some error independent of the other studies. By combining these results this aggregate statistical error can hopefully be eliminated.

Examples

• Within a group like a family, the parents may have chosen each other partly due to shared opinions and beliefs, and taught these to their children. The experience of strong consensus and absence of dissent within the family can cause the members to become more confident and more extreme in their beliefs (herd thinking), even if these opinions are quite different from a broader consensus in society. Moreover, it can be difficult for members to break out, since disagreeing with the group can be seen as disloyal, unethical, defiant, and ungrateful.
• In Congress, most committees include both Republicans and Democrats. Ideally, this leads to groups whose members have different biases, helping to prevent herd thinking and leading to more considered policy proposals.
• In science, the ideal is a large community of people with varied biases who pursue investigations independently, and share their findings and ideas periodically. These features help the enterprise of science take advantage of the virtues of wisdom of crowds.
  • Insofar as the members of the scientific community share the same biases - for example, that they are not demographically representative of humanity at large - this can reduce the efficacy. For this reason, representation in science is important not only for ethical reasons of equity, but also for epistemic reasons.

Common Misconceptions

• "Everyone agrees, so it must be true."

  Sometimes everyone in a group agrees due to shared biases, sometimes combined with a charismatic or dominating figure who has convinced everyone else. Consensus can be a good indicator of accuracy, especially when biases are distributed and independent judgments are included, but it isn't a guarantee, especially when the group is missing key information.

• "I trust my guess about how many pages are in this book more than the average of my guess with other people's guesses, because even if I don't have more information than other people, it's MY guess and you should stick with your own guess."

  If the goal is accuracy, more information is generally better. When other people's guesses don't share biases, as is typical with numeric guesses, averages of many independent guesses tend to be more accurate than any given randomly selected guess. Of course, if one person is an expert in some way, it may be worth weighting their answer more highly or even trusting their expertise.

Context

This lesson discusses when groups of people make better or worse judgments than individuals. It leads into the last part of the course, which revolves around group decision making, e.g. as a society.

Before

2.2 Systematic and Statistical Uncertainty

Individual judgments can deviate from the truth due to systematic bias and random fluctuation. Reducing shared bias and increasing sample size are ways to improve the effects of Wisdom of Crowds.

9.2 Biases

Conformity to the group consensus, illustrated by the Asch experiment, and obedience to a perceived authority in a group, illustrated by the Milgram experiment, are psychological tendencies that affect group decision making, tending to increase herd thinking.

10.1 Confirmation Bias

Confirmation bias can exacerbate other biases in group decision making, such as the motivation to make judgments that conform to the group consensus or agree with the opinion of the authority figure - again, increasing problematic herd thinking.

After

13.1 Denver Bullet Study

When making group decisions using the Denver Bullet Study method, it is important to survey each expert (on matters of fact) and each stakeholder (on matters of value) individually and independently so as to reduce "herd thinking" effects.

13.2 Deliberative Polling

In a somewhat opposite approach, deliberative polling encourages moderated discussion between all participants punctuated by dialogue with the relevant experts, before participants answer a poll to make informed decisions in society. Conformity with other participants as well as "obedience" to (in the sense of taking advice from) the expert panelists are essential features. The result tends to be a convergence of opinions towards moderation on divisive issues.

Recommended Outline

Before Class

• Review PlayPosit and lesson plans. If you have any questions, please ask!
• Update the Google Form in the warm-up questionnaire.

During Class

• (5 min) Come up with some fun way to assign the roles of spokesperson and notetaker (e.g. earliest birthday in the year, lives furthest from campus). Remind them of the responsibilities of these roles.
• (25 min) Questionnaire and presentation/slides.
• (30 min) Jury discussion and activity.
• (15 min) Optional: The Fermi problem if sufficient time remains.

After Class

• Go through the results from the warm-up questionnaire and make visualizations to show in plenary.

Lesson Content

Warm-up Polls

For this discussion section, students will start by filling out a Google form which asks for their best estimate of four quantities (should take 3-5 mins). The form should connect to a spreadsheet which shows how the class is or isn't aggregating live. Depending on the particular setup, this may take some spreadsheet finagling live during class. An example spreadsheet that was used in spring 2023 can be found here.

Wisdom of Crowds Estimates

The first poll demonstrates the wisdom of crowds and what happens when there's a systematic bias in people's judgments. It has the following four numerical questions. Make sure to copy the Google form and provide it to the students with a link and/or QR code. Make sure the students give their best guesses for each question without discussing with any classmates or looking any answers up. Don't reveal the answers until everyone's submitted their responses.

1. What is the heaviest recorded weight for a feline as of 2013?

2. What was the average weight for American men in 1960?

3. What is the heaviest recorded weight for a rabbit as of 2013?

4. What was the average weight for American women in 2010?

Famous Artist Poll

A painting made by... someone.

This poll demonstrates the power of herd thinking in amplifying shared biases. Students are shown the painting on the right and asked which of the following artists created it.

1. Claude Monet
2. Andy Warhol
3. Armand Guillaumin
4. Eugene Boudin

The poll is done in two rounds. In both cases, say the names in order and count how many students raise their hands for each one. Pause for a moment to give students a chance to get their hands up if they're slow (and to see what other people are doing).

1. Ask the students to make their best guess, but don't require students to raise their hand.
2. Force every student to make a guess. Tell the students that they all need to raise their hand for some choice even if they don't know the answer.

The correct answer is Armand Guillamin. However, it's likely the case that most people will pick Monet, because he is more widely known than the other options. This is an example of the negative outcome of herd thinking, where a group of people independently and collectively select the wrong answer because they all share the same prior bias (picking the famous painter). In the first case the bias only comes from people's tendency to pick familiar names. In the second, it may get amplified by seeing other more people raise their hands and then wanting to join in.

Structure of Groups Poll

The final warm-up is a quick poll (via hand raising is fine) to ask the students what the best way to answer each of the following questions is. Should it be done by deciding separately and averaging, by discussing as a group, or through some other means?

1. How many starlings are in a murmuration?

2. Is the center of mass between the Moon and Earth inside or outside of the Earth?

3. A jury deliberation in a criminal trial?

Takeaways

• The "wisdom of crowds" allows for the removal of statistical errors.
  • When there's no "systematic bias" in the opinions of a group, then independent guesses without discussion can average out individuals' "statistical biases."
  • Groups can come to surprising correct answers even without any expertise.
  • When there's expertise, but not systematic bias, often the "error bars" on the answer get smaller, but the average remains the same.
• The "wisdom of crowds" can become "herd thinking" in the case of systematic errors. But, it can also sometimes help.
  • When there's no systematic bias, but the guesses aren't independent (i.e. the group discusses their answer), there's often "herding" around the earlier answers.
  • When there's a systematic bias that is shared by many (but not all) members of the group, then discussion can sometimes dampen it.
  • This depends a lot on the group dynamics we looked at in our lesson on conformity.
• But what about when expertise is required, the decision is multi-faceted, and there may also be biases present? In that case, things get more complicated...

Jury Discussion

What is a Jury?

Many of the students may be international or haven't served on a jury before. So, we start by giving them a quick reminder of juries' most important features (for the purposes of this course).

• It is meant to be a group of peers randomly selected from your community.
• A jury is charged with the responsibility of deciding whether, on the facts of the case, a person is guilty or not guilty of the offence for which he or she has been charged.
• The jury is supposed reach its verdict by considering only the evidence introduced in court and the directions of the judge.
• Attorneys can typically veto potential jurors that they think may have some bias against their side.

Jury Group Dynamics Discussion Questions

We start with some discussion questions about shared biases. These should be answerable given the takeaways from the previous activity. Remind the students that juries typically discuss the merits of the case as a group so as to come to a consensus on whether the defendant is guilty.

1. What are the kinds of biases that might be present in a jury?
2. In what situations might the biases in a jury get amplified by deliberation?
3. In what circumstances might biases get averaged out?

Given the dynamics of conformity and obedience, the exact number of people and explicit decision making processes may also play a role in the decisions they make.

• A jury of 6 is constitutional, but a jury of 5 is not.

  (Williams v. Florida, 1970) (Ballew v. Georgia, 1978)

• In a jury of 6, a decision must be unanimous.

  (Burch v. Louisiana, 1979)

• In a jury of 12, a majority of 9:3 or 10:2 can be constitutional for non-capital offenses.

  (Johnson v. Louisiana and Apodaca v. Oregon, 1972)

Given this, have the students answer the following questions.

1. How might the dynamics of a jury of 6 with 1 dissenter (believes not guilty) be the same as or different from one of 12 with 2 dissenters?
2. How might jury composition dampen or amplify shared or individual biases?

How Juries Help

How juries help.

Kaplan & Miller (1978): Argued that deliberation helps juries reach a just evaluation of cases by averaging across different biases. The jury does this by arguing/debating the merits of the case. Strong/weak refers to the merits in the figure. The study varied obnoxiousness of attorney and which case was strong. Deliberation helps juries reach a just evaluation of cases by averaging across different biases. (Regardless of how obnoxious the attorney is, as long as they present the evidence.)

For more info, see this paper.

How Juries Hurt

How juries hurt. This figure is not actually from any of the papers linked on this page. It was kidnapped in cold blood from slides Rob MacCoun used in 2016.

Group Polarization: Group deliberation tends to exaggerate shared biases. For example… MacCoun (1985, 1990): Juries tended to overweight a completely irrelevant consideration: the defendant's attractiveness.

Why did juries overweight attractiveness? Because it's a shared bias.

For more info, see this paper and this paper.

How Juries Do Both

How juries do both.

Kerr, Niedermeier, & Kaplan (1999): Groups magnify biases when evidence is ambiguous. Groups reduce bias when evidence is really strong or weak.

For more info, see this paper.

Jury Overview Discussion Questions

1. What are the flaws of the jury system?
2. Which components of the jury system are similar to herd thinking? What about wisdom of crowds?
3. How do the flaws of the jury system diverge from the flaws of wisdom of crowds directly?

Building an Ideal Jury

Have the students discuss the following in small groups.

• Considering wisdom of crowds, herd thinking, and the other effects of group dynamics on decision making, how would you design a better jury system?
• Consider the whole process from selection of people (if you even have this?) to the final decision that's made. Assume there are no limits to your resources and control!

Lesson Content Overflow

Warm-up Questionnaire

For this discussion section, students will start by filling out a Google form which asks for their best estimate of four quantities (should take 3-5 mins). While the UGSI pulls up the responses to this poll, the GSI will continue on to a poll which demonstrates the concept of herd thinking. After the poll, the UGSI will take over screen sharing to show the summary of Google poll responses, while the GSI gives the correct answer for each question. The aggregate of the Google poll responses should demonstrate the wisdom of crowds.

Instructions

1. A series of polls with concept review.
   1. (4 min) Google poll: Prepare Google Forms (either one per section or one for the whole class) with questions from here. Provide the link to students (e.g. QR code) and tell them to just give their best guesses for each question without discussing with any classmates. Don't reveal the answers until after the next question.

   2. 

      A painting made by... someone.

      (5 min) In-person poll: students select which painter they think made the painting shown in the image on the right (A, B, or C).
      1. First have the students raise their hands for the answers. It's likely that students that don't know the answer just won't guess at all. But, they should be able to see how many people are voting for which options.
      2. Second, force the students to answer. Tell the students that every student needs to raise their hand for some choice even if they don't know the answer. Have them all go through it again.
2. Collect the Google Form responses as a Google Sheet. Produce an average value and standard deviation for each of the questions. Write the correct answers next to the average values and reveal these results to the students.
3. (8 min) Explain wisdom of crowds—sometimes groups make better judgments than individuals. This happens when:
   1. Judgments are genuinely independent, preventing herd thinking.
   2. Members of the group do not share the same biases.
   3. There are enough people in the group to balance out random biases or fluctuations (analogous to the need for an adequate sample size).
   4. Works especially well when estimating a quantity, where errors may be large but are not systematic.
4. (8 min) Explain herd thinking—Sometimes groups make worse judgments than individuals. This happens when:
   1. Judgments of individuals are influenced by the judgments of others, leading to group-think and sometimes polarization.
   2. Members of the group share biases, which can be exaggerated by discussion and cannot be decreased by averaging judgments.

Jury Composition and Deliberation

We use the composition and decision making process of juries as an example of a case where wisdom of crowds can be applied and misapplied.

Discussion Questions

See slides for graphs and more detailed discussion.

1. (5 min) How do the nuances of jury composition affect a jury's likelihood of coming to a verdict?
2. (5 min) Would a 6 person jury where 1 person believes not guilty come to the same conclusion as a 12 person jury where 2 people believe not guilty?
3. (5 min) How might deliberation lead to fairer results or polarize group decision making? Refer to the conditions for wisdom of crowds or herd thinking.

Redesigning the System

1. (4 min) Have a brief whole class discussion where you ask the students to consider what the flaws with the current jury system are and how it diverges from wisdom of crowds.
2. (6 min) In small groups, ask the think through the following question.
   1. Considering mainly wisdom of crowds, how would you design a better jury system? Consider the whole process from selection of people (if you even have this?) to the final decision that's made.
3. (5 min) Bring the whole class back together to share their solutions.

Fermi Problem

(Optional) As a final point, give the students an especially challenging Fermi problem to solve collectively. This is to emphasize that some problems do benefit from having deliberation at certain stages.

Instructions

1. (11 min) Have the students work out step 1 as groups.
2. (4 min) Have the students make their individual estimates and then average them together.

Step 1: The Tree

IKEA mainly produces two types of industrial bags. There are the blue bags that people can purchase and the yellow bags that are only used in the store. Besides the colors, these two types of bags are identical. As a group, figure out the quantities you need to estimate in order to determine the following:

1. How many blue (purchasable) bags does IKEA produce per year?
2. How many yellow (only for use in store) bags does IKEA produce per year?
3. Are there more blue or yellow IKEA bags?

Step 2: Estimates

Now, using wisdom of crowds, have every person in the group estimate each quantity independently without discussion. Then, average the final values together to get a result that's (hopefully) better than if the estimates were just determined individually.

Discussion Question

1. In terms of wisdom of crowds and herd thinking, why should the first step be collaborative, while the second step is individual?

Changemaker

"Members of a homogeneous group rest somewhat assured that they will agree with one another; that they will understand one another's perspectives and beliefs; that they will be able to easily come to a consensus. But when members of a group notice that they are socially different from one another, they change their expectations. They anticipate differences of opinion and perspective. They assume they will need to work harder to come to a consensus. This logic helps to explain both the upside and the downside of social diversity: people work harder in diverse environments both cognitively and socially. They might not like it, but the hard work can lead to better outcomes."

How does homogeneity in groups contribute to herd thinking? What benefit does diversity therefore provide?