RS517 Chap.2 Experimental Designs and Validity
Experimental Designs and Validity
Assignment determines inference
True experiments randomly allocate interventions, quasi-experiments study intervention changes without random allocation, and observational studies measure exposures that were not assigned.
These families do not rank every study, but they set different ceilings on causal language.
Internal threats are rival explanations
History, maturation, testing, regression, selection, attrition, instrumentation and statistical problems matter only when their mechanisms are connected to the outcome.
A concurrent comparison, repeated baseline, standardised measurement or sensitivity analysis addresses a named route; a smaller p-value does not.
Transport is separate
Internal validity asks whether the estimate is credible within the study. External validity asks whether it travels to target people, settings, interventions and follow-up.
A tightly controlled efficacy trial can be internally strong and deliberately narrow, while a pragmatic study trades some control for ordinary delivery.
Start with how the comparison was generated
Random allocation creates groups comparable in expectation on measured and unmeasured baseline factors, but concealment, adherence, attrition and outcome assessment still matter.
Cluster allocation can protect against contamination while reducing the number of independent units. Crossover designs use each participant as a comparison but require stable conditions, adequate washout and attention to period effects.
A simple before-after study has no concurrent counterfactual, so history, maturation and regression to the mean remain live explanations for change.
Translate validity labels into mechanisms
Internal validity improves when the design blocks plausible alternative causes.
Selection bias asks why participants enter or remain; performance bias asks whether care differs beyond the intended intervention; detection bias asks whether outcome measurement depends on group knowledge; attrition bias asks whether loss changes the contrast. External validity is a separate question about transport to other people, settings, staff and delivery conditions.
A narrowly controlled trial can estimate its internal contrast well and still need further evidence for routine care.
Read quasi-experiments through their assumptions
Interrupted time series needs enough observations before and after the intervention to separate a level or slope change from ordinary variation. Difference-in-differences needs a credible parallel-trends argument, not merely one baseline value.
A natural experiment requires an assignment event plausibly independent of potential outcomes. Matching can improve balance on recorded variables but cannot randomise an unmeasured cause. Name both the comparison and the assumption that turns it into evidence.
Revision drill
For one service change, sketch a before-after study, a controlled before-after study, an interrupted time series and a cluster trial.
Write one threat each design controls and one it leaves. Decide what additional data could reveal the remaining threat. Design choice then becomes a trade-off between inference, feasibility and ethics rather than a ranking learned by rote.
What this chapter covers
- 01
True experimental and randomised designs
- 02
Quasi-experimental comparisons
- 03
Observational exposure designs
- 04
Internal validity threats and controls
- 05
Efficacy and effectiveness
- 06
External validity and transportability
Worked example · free
Audit a before-after result
- 4Name history, measurement or regression mechanisms.
- 4Choose a concurrent or time-series improvement.
Key terms
- Random Allocation
- Assignment to groups by chance to support exchangeability within a study.
- Quasi-experiment
- An intervention study without random assignment to the comparison conditions.
- History Threat
- An external event during the study that offers another explanation for change.
- Regression To Mean
- The tendency for unusually extreme selected values to be less extreme on repetition.
- Transportability
- The extent to which an effect estimate applies in a specified target setting.
Experimental Designs and Validity FAQ
What is the difference between random sampling and random allocation?
Random sampling controls selection from a population frame and supports representation of that frame. Random allocation assigns enrolled participants to study conditions and supports internal group comparability. A trial may allocate well but recruit narrowly, while a survey may sample broadly without supporting a treatment-effect claim.
Can adjustment make a quasi-experiment equivalent to a trial?
Adjustment can reduce measured baseline differences and improve a quasi-experimental comparison, but it cannot guarantee balance on unmeasured causes or recreate the protection of random allocation. Strong answers describe the design, measured adjustment, pre-trend evidence and residual confounding rather than upgrading the label.
How should external validity be discussed?
Compare the study and target across participants, intervention delivery, comparator practice, outcome measurement and follow-up. Avoid calling a controlled efficacy study unrealistic without acknowledging its purpose. State what can travel directly and which evidence is still needed for ordinary practice.
Exam move
For each design, draw who assigns exposure, when outcomes are measured and where a comparator enters. Then list the strongest remaining rival explanation and the control that would address it. Practise changing the conclusion verb as the design changes from randomised to quasi-experimental to observational.
Working through Experimental Designs and Validity in RS517? Sia is AskSia’s AI Statistics tutor — ask any RS517 Experimental Designs and Validity question and get a clear, step-by-step explanation grounded in how RS517 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.