EDST5443 Chap.9 Trustworthy Assessment Judgements
Trustworthy Assessment Judgements
Trustworthy judgement is built through an argument connecting a defined purpose, relevant evidence, defensible interpretation and fair decision. This chapter examines validity, reliability and fairness as design questions rather than final technical labels.
It shows how sampling, task conditions, criteria, moderation, accessibility and consequences affect what can reasonably be claimed, and it uses failure analysis to distinguish consistency from accuracy and standardisation from equitable access.
What this chapter covers
- 01
Clarifying the purpose and claim an assessment is intended to support
- 02
Validity as the defensibility of an interpretation and use
- 03
Reliability as consistency sufficient for the stakes and decision
- 04
Fairness through access to the intended construct without irrelevant barriers
- 05
Sampling enough relevant evidence while recognising practical constraints
- 06
Using criteria, exemplars and moderation to surface judgement reasoning
- 07
Checking task, process, impact and response for threats to trustworthiness
- 08
Balancing competing qualities rather than treating one metric as decisive
Worked example · free
Diagnose a trustworthy judgement failure
- +1State the mismatch: recall evidence does not adequately represent collaborative inquiry performance.
- +1Explain that consistent scoring cannot repair weak validity for the intended claim.
- +1Propose relevant evidence, such as an inquiry product plus an individual account of reasoning and contribution.
- +1Use clear criteria, accessible conditions and moderation proportionate to the stakes.
Key terms
- Validity argument
- Reasoning and evidence supporting a proposed interpretation and use of assessment results.
- Reliability
- The degree of consistency in evidence or judgement required for a particular decision and its stakes.
- Fairness
- Assessment conditions that provide equitable access to the intended construct and avoid irrelevant disadvantage.
- Construct
- The knowledge, capability or quality that an assessment is intended to represent.
- Moderation
- Collaborative processes used to compare interpretations, apply standards and strengthen defensibility of judgements.
- Sampling
- Selection of tasks, occasions or evidence intended to represent a broader area of learning.
Trustworthy Assessment Judgements FAQ
Does reliability require identical tasks for everyone?
No. Consistency concerns the dependability needed for a decision. Appropriate adjustments or varied ways of accessing a task can reduce irrelevant barriers while preserving the intended construct.
Can an authentic task still be invalid?
Yes. Authentic appearance does not ensure that the evidence represents the intended learning or that criteria support the proposed interpretation. Purpose, sampling, conditions and consequences still require examination.
What does moderation accomplish?
Moderation makes judgement reasoning discussable, helps participants interpret criteria and standards and can improve consistency. It does not remove professional judgement or guarantee fairness unless the task and evidence are also defensible.
How much evidence is enough?
The answer depends on the breadth of the claim, variability of performance and consequences of the decision. Broader or higher-stakes claims usually require stronger sampling and scrutiny than a low-stakes instructional check.
Assessment move
Use a claim-evidence-inference-decision chain to analyse every assessment example. Write the intended claim in a single sentence, list exactly what learners produce, explain how that evidence supports the claim and state the resulting decision. Then stress-test each link.
Ask whether important content or performance has been under-sampled, whether irrelevant skills dominate, whether conditions differ in consequential ways and whether the decision exceeds the evidence. Compare validity, reliability and fairness through tensions rather than isolated definitions.
A tightly controlled task may improve scoring consistency while narrowing the construct; a rich task may better represent performance while requiring stronger criteria and moderation. Explain how the design manages the tension for its actual purpose and stakes. When reviewing a rubric, test whether criteria focus on the intended learning and whether quality descriptions can guide comparable judgement.
Use exemplars to discuss interpretations, not to impose one surface form. For an assessment sequence, distribute evidence across occasions where appropriate and specify how adjustments preserve the intended construct. In critical reflection, analyse consequences as part of trustworthiness. Consider who can show learning, what behaviours the task encourages and how results will be used.
Conclude with a calibrated claim: identify what the judgement supports, its limitations and the safeguards needed before a more consequential use. Create a threat register for one assessment with separate entries for construct coverage, irrelevant demand, sampling, assessor interpretation, access and consequences.
Rank each threat by likely impact on the intended decision, then attach a proportionate safeguard and evidence for reviewing it. Rehearse moderation with a borderline and an unusual response rather than only obvious samples. Record where assessors disagree, whether the source is the task, descriptor or interpretation, and what clarification follows.
This produces a more precise trustworthiness argument than a general promise that rubrics or double marking will make judgement objective.
Working through Trustworthy Assessment Judgements in EDST5443? Sia is AskSia’s AI Education tutor — ask any EDST5443 Trustworthy Assessment Judgements question and get a clear, step-by-step explanation grounded in how EDST5443 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
