SCIF1000 Chap.3 Bias, Multiple Testing and Reproducibility
Bias, Multiple Testing and Reproducibility
Repeated tests create repeated opportunities for false positives. This chapter computes the probability of at least one false positive across an independent family, then connects the arithmetic to selective reporting, HARKing and replication. Exploration remains useful, but it must be labelled and followed by independent testing.
The practical repairs are transparent analysis families, honest timing, complete reporting, shared methods and replication.
The complement calculation begins with the chance that one null test does not cross the threshold. Under independence, raising that value to the number of tests gives the chance that none cross. Subtracting from one gives the probability of at least one false positive somewhere in the family.
The event and assumptions must accompany the number so it is not misread as the probability that a particular hypothesis is false.
Multiplicity extends beyond a visible list of outcomes. Alternative exclusions, transformations, time points, subgroups and models can all create opportunities to select a favourable pattern.
Defining the analysis family from the scientific claim, rather than after seeing which result survives, makes the correction and interpretation more defensible. Dependence may change the exact calculation but not the need for transparency.
Selective reporting hides the size of the search process. HARKing changes the apparent timing by presenting a result-driven explanation as a prior prediction.
Both can make the final report look cleaner and the evidence stronger than it was. The repair is not to ban exploration. It is to separate discovery from confirmation and show readers the complete route that generated the highlighted result.
Reproducibility improves through specific protections. Preregistration records planned questions and decisions. Shared methods and materials permit inspection.
Complete reporting restores invisible outcomes. Independent replication tests whether the pattern survives new data. Each protection addresses a different failure, so a strong answer matches the repair to the problem rather than calling generally for scientists to avoid bias.
What this chapter covers
- 01
Per-test and familywise error
- 02
Complement probability across independent tests
- 03
Selective reporting and the file drawer
- 04
HARKing and exploratory findings
- 05
Preregistration, transparency and replication
Worked example · free
Calculate risk across eight tests
- +1Find the chance that one test is not significant: 0.95.
- +1Raise 0.95 to the eighth power to find the chance that none are significant: 0.6634.
- +1Take the complement: 0.3366, or about 33.7 per cent, and state the null and independence assumptions.
Key terms
- Familywise error
- The probability of at least one false positive across a defined family of tests.
- Selective reporting
- Showing favourable analyses or outcomes while leaving other relevant results invisible.
- HARKing
- Presenting a result-driven hypothesis as if it had been specified before analysis.
- Replication
- Repeating a study or test in new data to judge whether a finding is reproducible.
Bias, Multiple Testing and Reproducibility FAQ
Why does running more tests increase risk?
Each null test creates another chance to cross the threshold by chance. Under independence, the complement formula calculates how quickly the chance of at least one false positive rises.
Is exploratory analysis bad science?
No. Exploration can discover useful patterns and generate hypotheses. The problem arises when exploratory results are presented as prior predictions or treated as independently confirmed without validation.
How can a study improve reproducibility?
Record questions and analysis choices before results where appropriate, report the full analysis path, share enough method detail for inspection, and test important findings in independent data.
Can a multiple-testing correction rescue a weak study design?
No. A correction addresses the increased opportunity for false positives across a defined family of tests; it does not repair biased sampling, poor measurement, uncontrolled confounding, selective missingness or an unsuitable comparison. A corrected result can therefore be statistically cautious yet scientifically uninformative. Diagnose the design and the analysis separately, then explain how their limitations interact.
The most defensible report identifies the intended population, measurement process, test family and analytic choices, applies an appropriate multiplicity strategy, and still limits the conclusion to what the design can support.
What does a failed replication tell us about an earlier finding?
It should update confidence, but it does not automatically prove fraud or show that the original effect never occurs. Compare populations, procedures, measures, implementation, statistical precision and analysis choices before interpreting the difference. Ask whether both studies estimate the same quantity and whether either result is too imprecise to distinguish meaningful disagreement from sampling variation.
A replication can expose a fragile condition, an exaggerated initial estimate or a genuine context dependency. Report those alternatives explicitly and use further evidence to separate them rather than declaring success or failure from one threshold alone.
Assessment move
Recompute the familywise formula for several test counts until the complement logic is automatic. Pair each calculation with a precise verbal interpretation. Build a diagnosis table for multiple testing, selective reporting and HARKing, then match each problem to a specific repair.
When reading a result, ask how many outcomes, exclusions, subgroups and models were examined and whether the highlighted hypothesis existed before inspection.
