The University of Manchester · FACULTY OF STATISTICS

SOST10021 Chap.8 Measurement Robustness and Bias

- one subject, every graph, every model, every mark
7 Chapters3-page Bible
Our own words - no uploaded lecturer files
Updated for this semester
Chapter 8 of 10 · SOST10021

Measurement Robustness and Bias

Measurement Robustness and Bias tests whether a conclusion survives credible threats and analytical choices. Reliability concerns consistency, while validity asks whether the intended construct is represented. Selection, non-response, wording, recall, social desirability, misclassification and missingness can shift estimates through different mechanisms.

Confounding and reverse causation offer alternatives to a causal story. Random variation limits precision but is only one part of total error. Useful evaluation names the mechanism, predicts its likely consequence and proposes a remedy that targets the failure instead of attaching a generic caveat.

In this chapter

What this chapter covers

  • 01

    Reliability

  • 02

    Validity

  • 03

    Selection bias

  • 04

    Question wording

  • 05

    Missing data

  • 06

    Confounding

  • 07

    Sensitivity analysis

Worked example · free

Route one group difference through a bias diagnosis

Q [8 marks]. A study reports an observed difference between groups. Use the diagnostic tree to identify four distinct checks before accepting the conclusion. The weighting here is for diagnostic practice and is not a published University scheme.
The decision tree routes one observed difference toward four checks: selection, measurement, confounding and random variation, each requiring a different remedyThe decision tree routes one observed difference toward four checks: selection, measurement, confounding and random variation, each requiring a different remedy.observed differenceselectionwho is included?measurementsame meaning?confoundingthird cause?random variationhow uncertain?
The decision tree routes one observed difference toward four checks: selection, measurement, confounding and random variation, each requiring a different remedy.
  • 2Check selection by asking who entered, responded and remained in the analysis.
  • 2Check measurement by comparing wording, coding and meaning across groups.
  • 2Check confounding and temporal order before using a causal verb.
  • 2Assess random variation with design-aware uncertainty. The allocation is an original revision aid.
The conclusion should survive separate checks for selection, measurement, confounding and random variation. Each threat requires different evidence, so one sensitivity result cannot certify every part of the study.
Sia tip — Name the mechanism, state which values it could change and select a remedy that acts on that exact pathway.
Glossary

Key terms

Measurement error
Measurement error is the difference between a recorded value and the quantity a procedure intends to capture.
Reverse causation
Reverse causation occurs when the proposed outcome may influence the proposed exposure rather than only the reverse.
Sensitivity analysis
A sensitivity analysis tests how conclusions change under plausible alternative assumptions or decisions.
FAQ

Measurement Robustness and Bias FAQ

How are reliability and validity different?

Reliability concerns consistency under comparable conditions, while validity concerns whether the measure represents the intended construct. A procedure can repeatedly produce the same value and still measure the wrong concept, or capture a valid concept noisily.

What makes a bias critique useful?

A useful critique identifies who or what is affected, explains how recorded values or inclusion change, predicts the likely direction when possible and proposes a targeted check. Generic statements that all research has bias do not evaluate this result.

Can statistical adjustment establish causality?

Adjustment can address measured confounders under modelling assumptions, but it cannot automatically fix unmeasured factors, selection, measurement error or reverse causation. Causal interpretation depends on design, timing, comparison strategy and subject knowledge as well as a fitted model.

What does a robustness check establish?

A robustness check asks whether a conclusion survives one specified alternative, such as another threshold, coding rule or model. Stability increases confidence against that particular dependency, not against every possible error. Name what changed, compare the result, and state precisely which interpretation is strengthened or weakened.

Study strategy

Exam move

Organise evaluation into four columns: selection, measurement, confounding and random variation. For each study or table, write the strongest plausible threat in each column, then rank them by ability to change the conclusion. Convert the leading threat into a mechanism chain showing who is affected, how a value changes and which estimate moves.

Match it to a remedy such as better frame coverage, neutral wording, validation, longitudinal timing, a comparison group or a sensitivity analysis. Practise rewriting causal sentences as descriptive associations, then list the design evidence needed to restore a stronger verb. Distinguish missing, refused, not applicable and genuine zero categories and inspect whether exclusions differ across groups.

When several data sources agree, compare their populations and error structures before calling the result triangulated. Vary defensible thresholds and recodes without selecting only the most dramatic output. Label exploration and seek confirmation in new evidence. Keep a log of critiques that failed because the remedy did not address the problem.

End each evaluation with the consequence: narrowed scope, weaker precision, alternative explanation or need for another design. This prevents limitations from becoming detached lists and makes robustness reasoning advance the answer. Collect examples of leading, ambiguous and double-barrelled questions, then rewrite each neutrally and predict how responses might move.

Construct a missing-data table with refusal and not-applicable categories separated. Compare complete-case results with a sensitivity scenario and state what assumption drives the difference. For causal claims, draw arrows between exposure, outcome and plausible common causes, adding time order. Choose one measured adjustment and one unmeasured threat.

Read two studies that reach similar conclusions and test whether their sources fail independently or repeat the same coverage problem. Practise a robustness paragraph that begins with the strongest evidence, introduces the most consequential threat and explains whether the conclusion survives, narrows or changes. Finish by naming the specific additional observation that would most improve the judgement.

Working through Measurement Robustness and Bias in SOST10021? Sia is AskSia’s AI Statistics tutor — ask any SOST10021 Measurement Robustness and Bias question and get a clear, step-by-step explanation grounded in how SOST10021 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.

A+Everything unlocked
Unlocks this Bible + all 41 of your subjects at The University of Manchester, plus 1,000+ more Bibles across the AskSia library.
Sia - your SOST10021 tutor, unlimited, worked the way the exam marks it
The full 3-page Bible + practice bank with worked solutions
Chrome extension - sync your LMS so Sia knows your deadlines
Bilingual EN / Chinese on every Bible and every Sia answer
$0.99 Trial
30-day money-back · cancel in one tap · how it works