11.1 Pathological Science
More actions
How to catch bad science.
The Lesson in Context
This lesson teaches students to be cautious of bad science of all kinds and aware of the signs thereof. We ask students to study a few established examples of pathological science in history dressed as science merely in form. The emphasis is on well-intentioned researchers "falling in love" with their own ideas, finding excuses to justify them even when reality has turned out to be contrary to them.
Although blatant pseudoscience such as flat earth, astrology, creationism, and alternative medicine are also included, what deserves particular attention is seemingly genuine science done by seemingly genuine people that is nevertheless incorrect, possibly due to the researchers' own hubris.
- Everyone in principle has access to the same shared reality. If a certain (amazing) result by one research group cannot seem to be replicated by many other groups, it is a good sign that it does not accurately describe the shared reality.
- Overstating the confidence level of a scientific result is a sign of bad science.
- It is expected that patterns arise from random noise. Good science can also turn out to be wrong just by chance, but stubbornly sticking with the original result would turn it into bad science.
- In the spirit of personal persistence and iterative progress, setbacks such as a wrong or disappointing result should be accepted as an inevitable part of this progress.
- Though not yet widely employed in all fields of science, various blind analysis techniques can help reduce the possibility that a scientific result may be contaminated by subtle analysis choices made by the researchers that are (often subconsciously) motivated by the desire for a certain anticipated result.
- Bad science is particularly problematic when it concerns the study of human subgroups, as it may be motivated by or may perpetuate preexisting inequitable power structures in society.
Takeaways
After this lesson, students should
- Be able to distinguish roughly among the following:
- Good science that gets the wrong answer.
- Fraudulent science.
- Pathological science.
- Poorly-done science.
- Pseudo-science.
- Feel comfortable using Langmuir's Pathological Science Indicators to assess scientific articles.
- Be able to identify what is wrong in cases of fraudulent, pathological, poorly-done, and pseudo-science.
Spectrum of Poor Research
- Good Science
- Poorly-done Science
- Pathological Science
- Pseudo-science
- Fraudulent Science
Langmuir's Pathological Science Indicators
- The effect is produced by a barely detectable cause, and the magnitude of the effect is substantially independent of the intensity of the cause.
- The effect is barely detectable, or has very low statistical significance. Claims of great accuracy.
- Claims of great accuracy.
- Involving fantastic theories contrary to experience.
- Criticisms are met with ad hoc excuses.
- Ratio of supporters to critics rises to near 50%, then drops back to near zero.
- Conclusion-motivated design & analysis.
There is no set number of Langmuir's criteria that determines whether something is pathological science or not. The criteria serve as a useful guide, but detecting pathological science is ultimately something of an art.
This paper's result could not be replicated by others in the field, therefore the authors have committed pathological science.
Useful Resources
Recommended Outline
Before Class
Each student will have read one of the three articles that we discuss in this section, as well as an article by Langmuir. (Article A, Article B, Article C, Langmuir's reading) Assign each group to one of the three articles one week ahead of time, and then send another reminder to read them one or two days before the lesson. Also make sure to print the handout.
During Class
| 5 Minutes | Introduce the lesson and go over the plan for the day. Make sure people have groups, spokespeople, etc. |
| 15 Minutes | Review the definitions for this lesson. |
| 15 Minutes | Students review assigned articles in small groups. |
| 20 Minutes | Jigsaw groups so each group has two students who read each article present to the others. |
| 20 Minutes | Run the whole-class discussion. |
Lesson Content
Activity 1
[Brief description of and motivation for the activity]
[Common misconception or thing to look out for.]
[Thing you really need to look out for!]
[Title]
[Useful tip, guideline, or other background.]
Instructions
| [n] Minutes | [Activity.] |
| [n] Minutes | [Activity.] |
Discussion Questions
[Question 1]
[Possible misconception that may need to be corrected and clarified.]
[Intended answer to the above question.]
[Question 2]
[Possible misconception that may need to be corrected and clarified.]
[Intended answer to the above question.]