PES6251 Chap.9 Reliability, Validity and Measurement
Reliability, Validity and Measurement
Consistency does not guarantee correctness
Reliability addresses stability, internal consistency and agreement under specified conditions. Validity addresses whether evidence supports the intended interpretation and use for a population and context. Systematic error can preserve repeatability while shifting every score, whereas random error weakens consistency.
Instrument choice begins with a construct definition and then examines content, criterion and construct evidence rather than choosing the most familiar questionnaire or device.
Ask which variation matters
Test-retest reliability examines stability when the construct should not change, inter-rater reliability examines observer agreement and internal consistency examines coherence among scale items.
A strong coefficient in one category does not certify the others. Highly similar items can produce impressive internal consistency while representing too little of the construct.
Name the occasions, raters, items and population behind the estimate so the reader can judge whether it addresses the proposed use.
Validity evidence follows the intended decision
Content evidence asks whether the measure represents the full construct. Criterion evidence compares it with a meaningful external measure, and construct evidence tests relationships predicted by theory.
A screening tool needs classification evidence around a threshold, while a monitoring tool needs sensitivity to change. Evidence for separating high from low performers may not show that the same scale detects modest improvement.
Define the interpretation first and gather evidence for that use.
Integrate complementary errors deliberately
Self-report can capture activity mode, purpose and context but faces recall and social-desirability error. Devices can record timing and intensity consistently yet miss resistance work or the meaning of behaviour.
Combining them is useful when the integration rule is planned and discrepancies are interpretable. Standardise administration, placement, language and observer position, and record deviations. Then state whether each error adds random noise, shifts scores systematically or differs between groups.
What this chapter covers
- 01
Test-retest stability
- 02
Inter-rater agreement
- 03
Internal consistency
- 04
Content and criterion evidence
- 05
Systematic and random error
- 06
Subjective and objective activity measures
Worked example · free
Separate stable error from useful measurement
- 2Assess repeatability.
- 3Identify systematic error.
- 3Explain how intended use changes the verdict.
Key terms
- Test-retest Reliability
- Score stability across repeated occasions when the measured construct should remain unchanged.
- Inter-rater Reliability
- The extent to which observers applying the same procedure produce comparable scores or categories.
- Internal Consistency
- The degree to which items intended to form a scale behave coherently with one another.
- Criterion Evidence
- Support obtained by comparing scores with a meaningful external measure concurrently or predictively.
- Systematic Error
- A consistent measurement shift that can preserve repeatability while damaging correctness.
Reliability, Validity and Measurement FAQ
Can a measure be reliable but invalid?
Yes. A device or scale can give nearly identical readings every time while being consistently too high, too low or focused on the wrong construct. Reliability is necessary for stable interpretation but is not sufficient. Validity evidence must address the intended meaning, population, context and decision.
Which reliability estimate should I report?
Choose the estimate that matches the suspected source of variation. Use test-retest evidence for stability across occasions, inter-rater evidence for observer agreement and internal consistency for coherence among scale items. One coefficient does not certify all measurement properties, and conditions behind the estimate should be stated.
Are objective activity measures always better than questionnaires?
No. Devices can record movement consistently but may miss activity type, resistance work, context or purpose. Self-report captures context and intention but faces recall and social-desirability error. The better measure is the one whose strengths fit the exact research question and whose limitations are acknowledged.
Assessment move
For every instrument, write the construct, intended interpretation, population, administration and scoring. Then match one reliability question and one validity question to that use. Practise identifying whether an error is systematic or random and predict how it would change an association, group difference or absolute classification.
Do not accept a reliability coefficient without checking the relevant source of variation, interval and participant range. For validity, assemble evidence about content, response process, internal structure and relationships with other variables, then state which interpretation that evidence supports.
Compare the likely measurement error with the smallest meaningful change so that statistical consistency does not conceal poor practical discrimination.
Working through Reliability, Validity and Measurement in PES6251? Sia is AskSia’s AI Statistics tutor — ask any PES6251 Reliability, Validity and Measurement question and get a clear, step-by-step explanation grounded in how PES6251 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
