DDES9911 Chap.8 Assessment and Evaluation
Assessment and Evaluation
Assessment judges learner performance, while evaluation examines whether the simulation and learning strategy are effective. This chapter builds an evidence chain from target performance through observable traces and transparent judgement to feedback or progression. It compares technical and non-technical skills, automatic and human evidence, proxy risks, assessor reliability and quality control for consequential decisions.
What this chapter covers
- 01
Assessment versus evaluation
- 02
Technical skill evidence
- 03
Non-technical skill evidence
- 04
Automatic and human measures
- 05
Rubrics and judgement rules
- 06
Reliability and moderation
- 07
Program quality control
Worked example · free
Assess incident command with several valid paths
- 1Define the target capabilities without reducing them to completion time.
- 1Capture a decision timeline and selected scenario consequences.
- 1Use observable rubric criteria and short learner rationales.
- 1Moderate uncertain results and review scenario bias before revising standards.
Key terms
- Assessment
- Judgement of learner performance against a stated outcome or standard.
- Evaluation
- Investigation of whether a learning strategy, simulation or program is effective.
- Performance trace
- A recorded action, decision, outcome or explanation relevant to a learning claim.
- Proxy measure
- An indirect measure used in place of the target capability.
- Moderation
- Review that supports consistent and defensible judgements across assessors or cases.
Assessment and Evaluation FAQ
How are assessment and evaluation different?
Assessment asks how well the learner performed against an outcome or standard. Evaluation asks whether the learning strategy, simulation and delivery worked. The same event may inform both, but a poor result can reflect learner need, interface design, scenario alignment or operational failure.
When are automatic metrics useful?
Automatic metrics are useful when they directly represent meaningful behaviour such as sequence, precision or a critical omission. Time, clicks and path length become weak proxies for judgement or teamwork unless the design explains their relevance and combines them with contextual evidence.
How can rubric reliability improve?
Write criteria as observable behaviours, provide examples and non-examples, calibrate assessors on shared cases and moderate uncertain results. Reliability supports consistency, but the criteria must still represent the target capability or the process will consistently judge the wrong thing.
Can several metrics be combined into one score?
They can, but only after each metric has a defensible relationship to the outcome and the weighting rule is explained. A composite score can conceal a critical omission when strong results elsewhere compensate for it. Inspect the component evidence, define any non-compensable safety or quality conditions, test the scoring rule on contrasting cases and report uncertainty instead of presenting convenient precision as fact.
Assessment move
Write an evidence chain for each outcome: target performance, observable trace, interpretation rule, decision and feedback. Mark every proxy and describe how it could be gamed. Combine system evidence with expert judgement only when each contributes something distinct. For rubrics, test whether two assessors can identify the same behaviour and whether the behaviour represents the authentic task.
Separate learner decisions from program evaluation, then list the contextual evidence needed before attributing a poor result to one or the other. Create a measure inventory divided into direct evidence, proxy evidence, self-report and contextual information. For each item, write the claim it can support and one claim it cannot.
This protects time, score, confidence or click counts from becoming broader evidence than they deserve. If several measures will be combined, identify any critical error that cannot be compensated by strength elsewhere. Rehearse rubric calibration with two contrasting cases. Ask independent reviewers to locate the observable behaviour behind each criterion, assign a judgement and explain their evidence.
Differences may reveal ambiguous wording, missing context or a genuine boundary case. Decide whether training, criterion revision or moderation is the appropriate response. Add a fairness review that examines interface access, language, scenario familiarity and assessor expectation. A consistent pattern may come from the program rather than the capability.
Preserve scenario version and assistance status so reviewers can investigate. Finish by drawing two arrows from the evidence: one to learner feedback or progression, another to program evaluation. Name the decision owner on each route. The same event can inform both, but the analysis and consequence must remain distinct.
Working through Assessment and Evaluation in DDES9911? Sia is AskSia’s AI Education tutor — ask any DDES9911 Assessment and Evaluation question and get a clear, step-by-step explanation grounded in how DDES9911 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.