PES6252 Chap.10 Observer Error, Reliability and Coach Development
Observer Error, Reliability and Coach Development
A corrupted record looks exactly like a sound one
Nothing in the method chapters works if the record is wrong, and the dangerous property of a wrong record is that it is indistinguishable from a right one by inspection. A drifting coder produces a complete sheet. A biased coder produces plausible totals. A coach who knows he is being filmed produces a session.
The only way to know is to have built the checks in beforehand, which is why this chapter belongs before data collection rather than after it.
Five classic threats are named, and none of them is a mark of carelessness: published studies report guarding against all five, which is an admission that all five happen to competent people working carefully.
The five, grouped by where the error enters
Drift and complexity are properties of a coding system meeting a human being.
Drift is the observer gradually changing the coding rules or interpreting them differently over time, with definitions loosening or tightening across a long collection period and two observers possibly drifting in opposite directions. Complexity is a category set larger than working memory: more categories means more decisions per second and accuracy falls as load rises, with live coding far more error prone than video.
Bias and cheating are properties of the coder relationship to the study: what a coder expects of the coach, or of the study, tips the borderline decisions, and completing a sheet later from memory is the everyday version, far commoner than fabrication. Reactivity is the only one that is not about the observer at all, and the only one where the data are faithfully recorded and still do not describe ordinary coaching.
What this chapter covers
- 01
Why a corrupted record cannot be identified by looking at it
- 02
Drift: definitions moving, and invisible to the person moving them
- 03
Complexity: category load, live coding, and the cost of simplifying
- 04
Bias on borderline calls, and why it is worse than random error
- 05
Reactivity, habituation, and the sessions you plan to discard
- 06
Cheating in its ordinary form: sheets completed from memory
Diagnose four observation failures and triage them
- 4Name each of the four failures.
- 4Say what is recoverable and what is not, with the reason.
- 4Name the two habits that would have prevented three of the four.
Key terms
- Observer Bias
- The steady tilting of borderline decisions by what a coder already expects of the coach or of the study, reduced by hiding whose session it is.
- Intra Observer Reliability
- The agreement of one coder with themselves on the same footage at two times, which is the direct test for drift.
- Habituation
- Filming a coach and squad repeatedly before any analysed session, so that the change caused by being watched has faded before data collection begins.
Observer Error, Reliability and Coach Development FAQ
Why is drift the threat that costs most?
Because of what it does to a before and after design, which is the commonest shape in this field. If a coder definition of post instruction loosens across eight weeks, the second observation shows a higher rate whatever the coach did, and the study reports exactly the change the intervention was meant to produce while the coder has no sense of having changed anything.
That is why the reliability check belongs in the middle of the collection period rather than at the end, and why the check costs one session of double coding and saves the whole study.
Is it better to use a simpler instrument if agreement is poor?
Only if the question survives the simplification. More categories means more decisions per second and accuracy falls as load rises, so a smaller set will raise agreement, but simplifying may cost validity and produce a record two coders agree about that no longer captures the thing being studied. The test is the research question.
If the question is how much feedback a coach gives, merging feedback categories is fine; if it is what kind, merging them deletes the answer and the correct response is more coder training.
Assessment move
Code the same five minutes of footage twice, a week apart, without looking at your first sheet. The disagreement between you and yourself is your own drift rate, it takes twenty minutes to measure, and it is the fastest way to stop treating reliability as an administrative requirement rather than as a property of your own attention.
Working through Observer Error, Reliability and Coach Development in PES6252? Sia is AskSia’s AI Education tutor — ask any PES6252 Observer Error, Reliability and Coach Development question and get a clear, step-by-step explanation grounded in how PES6252 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
