The Hong Kong Polytechnic University · FACULTY OF STATISTICS

RS517 Chap.2 Experimental Designs and Validity

- one subject, every graph, every model, every mark
6 Chapters3-page Bible
Our own words - no uploaded lecturer files
Updated for this semester
Chapter 2 of 10 · RS517

Experimental Designs and Validity

Assignment determines inference

True experiments randomly allocate interventions, quasi-experiments study intervention changes without random allocation, and observational studies measure exposures that were not assigned.

These families do not rank every study, but they set different ceilings on causal language.

Internal threats are rival explanations

History, maturation, testing, regression, selection, attrition, instrumentation and statistical problems matter only when their mechanisms are connected to the outcome.

A concurrent comparison, repeated baseline, standardised measurement or sensitivity analysis addresses a named route; a smaller p-value does not.

Transport is separate

Internal validity asks whether the estimate is credible within the study. External validity asks whether it travels to target people, settings, interventions and follow-up.

A tightly controlled efficacy trial can be internally strong and deliberately narrow, while a pragmatic study trades some control for ordinary delivery.

Start with how the comparison was generated

Random allocation creates groups comparable in expectation on measured and unmeasured baseline factors, but concealment, adherence, attrition and outcome assessment still matter.

Cluster allocation can protect against contamination while reducing the number of independent units. Crossover designs use each participant as a comparison but require stable conditions, adequate washout and attention to period effects.

A simple before-after study has no concurrent counterfactual, so history, maturation and regression to the mean remain live explanations for change.

Translate validity labels into mechanisms

Internal validity improves when the design blocks plausible alternative causes.

Selection bias asks why participants enter or remain; performance bias asks whether care differs beyond the intended intervention; detection bias asks whether outcome measurement depends on group knowledge; attrition bias asks whether loss changes the contrast. External validity is a separate question about transport to other people, settings, staff and delivery conditions.

A narrowly controlled trial can estimate its internal contrast well and still need further evidence for routine care.

Read quasi-experiments through their assumptions

Interrupted time series needs enough observations before and after the intervention to separate a level or slope change from ordinary variation. Difference-in-differences needs a credible parallel-trends argument, not merely one baseline value.

A natural experiment requires an assignment event plausibly independent of potential outcomes. Matching can improve balance on recorded variables but cannot randomise an unmeasured cause. Name both the comparison and the assumption that turns it into evidence.

Revision drill

For one service change, sketch a before-after study, a controlled before-after study, an interrupted time series and a cluster trial.

Write one threat each design controls and one it leaves. Decide what additional data could reveal the remaining threat. Design choice then becomes a trade-off between inference, feasibility and ethics rather than a ranking learned by rote.

In this chapter

What this chapter covers

  • 01

    True experimental and randomised designs

  • 02

    Quasi-experimental comparisons

  • 03

    Observational exposure designs

  • 04

    Internal validity threats and controls

  • 05

    Efficacy and effectiveness

  • 06

    External validity and transportability

Worked example · free

Audit a before-after result

Q [8 marks]. Practice note: the marks used here are not part of the university's published assessment scheme. Falls decline after a ward introduces training. Identify three rival explanations and one design improvement.
The three design families differ primarily in who controls exposure assignment and therefore which causal alternatives remain plausible.The three design families differ primarily in who controls exposure assignment and therefore which causal alternatives remain plausible.True experimentrandom assignmentcomparison controlQuasi-experimentintervention observedassignment not randomObservationalexposure measuredno assigned treatmentInference weakens as alternative explanations become harder to exclude
The three design families differ primarily in who controls exposure assignment and therefore which causal alternatives remain plausible.
  • 4Name history, measurement or regression mechanisms.
  • 4Choose a concurrent or time-series improvement.
The decline may reflect changing patient mix, seasonal staffing, altered reporting or regression after an unusually poor baseline. A comparable concurrent ward plus several pre-intervention time points would separate common history and baseline trend more effectively than one before-after contrast.
Sia tip — Translate every validity label into a mechanism that could move the outcome in this study.
Glossary

Key terms

Random Allocation
Assignment to groups by chance to support exchangeability within a study.
Quasi-experiment
An intervention study without random assignment to the comparison conditions.
History Threat
An external event during the study that offers another explanation for change.
Regression To Mean
The tendency for unusually extreme selected values to be less extreme on repetition.
Transportability
The extent to which an effect estimate applies in a specified target setting.
FAQ

Experimental Designs and Validity FAQ

What is the difference between random sampling and random allocation?

Random sampling controls selection from a population frame and supports representation of that frame. Random allocation assigns enrolled participants to study conditions and supports internal group comparability. A trial may allocate well but recruit narrowly, while a survey may sample broadly without supporting a treatment-effect claim.

Can adjustment make a quasi-experiment equivalent to a trial?

Adjustment can reduce measured baseline differences and improve a quasi-experimental comparison, but it cannot guarantee balance on unmeasured causes or recreate the protection of random allocation. Strong answers describe the design, measured adjustment, pre-trend evidence and residual confounding rather than upgrading the label.

How should external validity be discussed?

Compare the study and target across participants, intervention delivery, comparator practice, outcome measurement and follow-up. Avoid calling a controlled efficacy study unrealistic without acknowledging its purpose. State what can travel directly and which evidence is still needed for ordinary practice.

Study strategy

Exam move

For each design, draw who assigns exposure, when outcomes are measured and where a comparator enters. Then list the strongest remaining rival explanation and the control that would address it. Practise changing the conclusion verb as the design changes from randomised to quasi-experimental to observational.

Working through Experimental Designs and Validity in RS517? Sia is AskSia’s AI Statistics tutor — ask any RS517 Experimental Designs and Validity question and get a clear, step-by-step explanation grounded in how RS517 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.

A+Everything unlocked
Unlocks this Bible + all 24 of your subjects at The Hong Kong Polytechnic University, plus 1,000+ more Bibles across the AskSia library.
Sia - your RS517 tutor, unlimited, worked the way the exam marks it
The full 3-page Bible + practice bank with worked solutions
Chrome extension - sync your LMS so Sia knows your deadlines
Bilingual EN / Chinese on every Bible and every Sia answer
$0.99 Trial
30-day money-back · cancel in one tap · how it works