PSY2041 Chap.2 Reliability and Validity
Reliability and Validity
Reliability and Validity separates score consistency from support for interpretation. Reliability asks whether scores reproduce across relevant conditions, including occasions, raters, items or equivalent forms. Validity asks whether evidence supports the meaning and use assigned to the scores.
A test may reliably produce the same biased or irrelevant result, so reliability is necessary but never sufficient for validity. The strongest answers match a claim with the comparison that could test it and keep the population, timing and purpose visible.
What this chapter covers
- 01
Observed scores, stable signal and measurement error
- 02
Test-retest reliability for stability across suitable intervals
- 03
Interrater reliability for scoring agreement
- 04
Internal consistency across items intended to measure one construct
- 05
Equivalent forms reliability across alternate versions
- 06
Content, criterion and construct sources of validity evidence
- 07
Why consistency supports but does not guarantee valid use
Worked example · free
Match three claims with three evidence designs
- 2Use internal-consistency evidence to examine whether items intended for the same construct work together.
- 2Use test-retest evidence across the stated short interval, provided genuine change is not expected.
- 2Use a prospective criterion design relating the initial score to independently measured later help-seeking.
Key terms
- Test-retest reliability
- Consistency of scores across occasions when the underlying construct should remain stable.
- Interrater reliability
- Consistency of scores assigned by different evaluators to the same performance.
- Internal consistency
- The degree to which items intended to measure one construct function coherently.
- Equivalent forms
- Alternate test versions designed to produce comparable scores.
- Criterion validity
- Evidence from the relationship between a test score and a relevant external outcome.
- Construct evidence
- Evidence that score patterns behave as theory predicts for the intended construct.
Reliability and Validity FAQ
Can a test be reliable but invalid?
Yes. A procedure can reproduce the same score while consistently measuring the wrong attribute, using a biased method or supporting an inference beyond its evidence. Reliability controls instability; validity evaluates whether the proposed meaning and use are justified.
Which type of reliability should I report?
Choose the type that matches the potential source of inconsistency. Use test-retest for time, interrater for scorers, internal consistency for item coherence and equivalent-forms evidence for alternate versions. One type does not automatically establish the others.
Is validity a property of the test itself?
Validity is best framed as support for a score interpretation and use in a population and context. Evidence that works for one decision may not justify another. State the intended inference rather than writing the absolute claim that an instrument is valid.
Exam move
Create a four-row reliability table with comparison, suitable use, threat and remedy. For validity, start from the claim and write the external observation or theoretical pattern that would support it. Practise scenarios where a result establishes one evidence type but leaves another untested.
In extended responses, define reliability and validity, state that consistency is necessary but insufficient, then apply the relationship to the facts. Keep conventional thresholds subordinate to design, population and purpose. Finish every result sentence with its scope: stable across this interval, agreement among these raters, coherent in this sample or predictive of this criterion.
Revise by converting advertisements about tests into evidence designs. Circle each claim, identify what would be compared and decide whether the relevant threat concerns time, raters, items, forms, content representation, an external criterion or theoretical relationships. Do not begin with a coefficient name; begin with the comparison that the claim requires.
Use a two-column error log headed evidence obtained and evidence still missing. A strong item correlation might support coherence while leaving dimensionality, stability and criterion relations unresolved. A large test-retest coefficient may support stability across that interval while saying nothing about scorer agreement.
For validity practice, write one content, criterion and construct question about the same proposed scale. Then change the population or purpose and explain why earlier evidence may no longer settle the new interpretation. Sketch the true-score and error partition and use it to explain attenuation without claiming that reliability is validity.
When reviewing a numerical result, ask how restriction of range, genuine change, memory, rater training or heterogeneous items could affect it. End each session with a five-sentence response: define consistency, define validity support, name their logical relationship, apply the named evidence and state the bounded inference. This routine preserves the distinction even when the scenario combines several attractive statistics.
Review mistakes by labelling whether you confused an evidence source, overstated its scope or treated a conventional threshold as proof. Correcting the category of error makes later scenarios easier to diagnose.
Working through Reliability and Validity in PSY2041? Sia is AskSia’s AI Psychology tutor — ask any PSY2041 Reliability and Validity question and get a clear, step-by-step explanation grounded in how PSY2041 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
