DSC6008 Chap.6 Randomization Tests and Monte Carlo Evidence
Randomization Tests and Monte Carlo Evidence
Let the design generate the null
A randomization test asks which assignments could have occurred under the actual experiment. Under a sharp null of no treatment effect, outcomes stay fixed while treatment labels are reassigned according to that mechanism.
Recomputing the chosen statistic over valid assignments creates a reference distribution without assuming a parametric error law.
State what the sharp null makes possible
Only one potential outcome is observed for each unit. The sharp null claims that treatment changes no unit's outcome, so the missing potential outcome under the alternative assignment can be replaced by the observed value.
This imputation is why outcomes may remain fixed while labels move. A null of zero average effect is weaker and does not generally provide the same unit-level imputation.
Match every shuffle to the assignment rule
Complete randomization with fixed treatment counts permits balanced reallocations across all eligible units.
Blocked randomization permits reallocations only inside each block, while a matched-pair design permits a swap or sign change within each pair.
An unrestricted shuffle after a restricted design includes allocations that were never possible and produces the wrong reference distribution.
Choose the statistic before generating assignments
The statistic should express the treatment contrast and handle the design structure.
A difference in means may suit a balanced two-group experiment, while a block-adjusted contrast may be needed when assignments were restricted. The identical formula is applied to the observed and reassigned data.
Changing the statistic after seeing which option gives a smaller tail probability abandons its calibration.
Measure the exact tail honestly
For a two-sided test, count assignments whose statistic is at least as extreme as the observed value under the chosen definition of extremity. The observed assignment belongs to the reference set and must not be discarded from its own tail.
The exact p-value is a proportion over all permitted assignments, so it can be audited through the assignment count before any interpretation is attempted.
Approximate large spaces with Monte Carlo sampling
When enumeration is impractical, sample valid assignments uniformly and estimate the tail fraction. A plus-one calculation includes the observed assignment and prevents a finite simulation from reporting zero.
Record the draw count and seed. The approximate standard error, based on the estimated probability and number of draws, shows whether simulation noise is small enough for the conclusion.
Conclude with the tested null and design
Rejection provides evidence against the sharp no-effect statement under the implemented assignment. Non-rejection does not establish that every unit effect is zero.
Report the observed contrast, the assignment restrictions, whether the reference was exact or simulated and the tail definition. This keeps a design-based number attached to the claim it can actually support.
What this chapter covers
- 01
Sharp nulls
- 02
Assignment mechanisms
- 03
Permutation statistics
- 04
Tail probabilities
- 05
Monte Carlo error
- 06
Blocked shuffles
Worked example · free
Estimate a randomization tail probability
- 2Add one extreme result for the observed assignment.
- 1Use 999 + 1 = 1000 in the denominator.
- 2Compute (23 + 1) / 1000 = 0.024.
Key terms
- Sharp Null
- A null claim specifying no treatment effect for every experimental unit.
- Permutation Statistic
- A statistic recomputed after a treatment-label reassignment allowed by the design.
- Monte Carlo Error
- Numerical uncertainty caused by estimating a reference probability with finitely many simulations.
Randomization Tests and Monte Carlo Evidence FAQ
What makes a permutation valid?
It must be an assignment that the original design could have produced. A completely randomized experiment permits one set of reallocations, while paired or blocked designs require treatment labels to move only within pairs or blocks. Unrestricted shuffling changes the design and answers a different question.
Why include the observed assignment?
The observed allocation belongs to the set of possible allocations under the design. Including it through the plus-one correction gives a conservative finite-simulation estimate and prevents an impossible reported p-value of zero when none of the sampled reallocations is more extreme.
How many simulations are enough?
Enough that simulation noise is small relative to the inferential distinction being made. The standard error of an estimated tail probability is approximately the square root of p(1-p)/B. Near a decision boundary, increasing B is more important than when the conclusion is far from it.
Assessment move
Start with a physical description of the assignment mechanism. List what remains fixed under the null, what labels can move and where they are allowed to move. Only then choose the statistic and tail. Recompute a tiny example by hand before trusting a Monte Carlo implementation.
Working through Randomization Tests and Monte Carlo Evidence in DSC6008? Sia is AskSia’s AI Statistics tutor — ask any DSC6008 Randomization Tests and Monte Carlo Evidence question and get a clear, step-by-step explanation grounded in how DSC6008 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
