MGMT7250 Chap.3 Appraising Professional Expertise
Appraising Professional Expertise
Professional expertise is valuable because it identifies patterns, constraints and plausible mechanisms before other evidence is available. It is risky when confidence substitutes for calibration. This chapter treats expert judgement as a revisable prior, distinguishes directional bias from unwanted variation, and tests whether a panel contributes independent information.
It provides practical elicitation moves: state a range, name the reference class, explain assumptions and identify the observation most likely to change the view.
Expertise develops most reliably in environments with stable patterns, repeated decisions and prompt, accurate feedback. Length of service alone is weak evidence when outcomes are delayed, ambiguous or selectively remembered.
Ask how the practitioner learned that prior judgements were correct and whether failed cases remained visible. That answer helps determine how tightly to weight the prior.
Bias and noise require different responses. Bias pushes judgements systematically in one direction, so adding colleagues exposed to the same training or incentive may reproduce the error.
Noise is unwanted variation across judges or occasions; independent estimates and structured rubrics can reduce it. A panel should therefore estimate privately before discussion, preserve first-round reasons and show what new information caused convergence.
Elicit a range, confidence, reference class, causal story and change condition.
These elements make the judgement testable and separate experienced pattern recognition from status. After other evidence arrives, document the direction and reason for the update without inventing false numerical precision. A forecast log can improve calibration across repeated decisions, provided review is designed for learning rather than punishment.
The central question is not whether an expert is right in the abstract, but what the expertise contributes to this claim and which independent evidence can reveal its boundary.
What this chapter covers
- 01
Expert judgement as a prior
- 02
Bias versus noise
- 03
Calibration and valid feedback
- 04
Independent panel estimates
- 05
Reference classes and base rates
- 06
Explicit belief updating
Worked example · free
Audit a unanimous forecast
- 1Identify the shared briefing as a dependent information source.
- 2Separate agreement from accuracy and ask for reference cases.
- 1Seek a representative pilot or disconfirming customer evidence.
- 1Update modestly rather than averaging confidence mechanically.
Key terms
- Prior Judgement
- A Prior Judgement is the explicit belief held before new evidence is acquired.
- Bias
- Bias is a systematic tendency for judgements or estimates to move away from the target in one direction.
- Noise
- Noise is unwanted variation across judges, occasions or repeated measurements.
Appraising Professional Expertise FAQ
Does experience make an expert calibrated?
Only when the domain provides repeated, timely and valid feedback. Experience in a changing or low-feedback environment can reinforce confidence without improving accuracy, so calibration evidence and reference cases still matter.
Why does averaging reduce noise but not shared bias?
Independent errors can cancel when estimates are combined. A common directional error remains in every estimate and survives the average. Shared training, incentives or information therefore need separate appraisal before aggregation.
What should an expert state with a forecast?
Ask for a range, confidence, reference class, assumptions and the observation that would most change the view. These details make the judgement testable and expose whether certainty rests on evidence or authority.
Assessment move
Practise with forecasts from work or public commentary. Label the claim, confidence, reference class and change condition. Then distinguish possible bias from noise and decide whether additional judges would contribute independent information. Finish with an explicit update sentence that names the new evidence and direction of movement without inventing a precise posterior.
For a panel case, draw an information-dependence map before averaging estimates. Identify shared briefings, common databases, hierarchy and incentives. Rehearse an elicitation sequence in which members estimate independently, disclose assumptions, inspect a base rate and revise. Keep the first and second estimates so the source of movement stays visible.
Create three calibration cases from the same domain and compare confidence with outcomes. Finally, name a domain boundary where the practitioner’s experience should receive less weight and explain which evidence would compensate. Build a short forecast log with dates, ranges, assumptions and eventual outcomes.
Sort missed forecasts into directional bias, excessive confidence and random variation, then prescribe a different correction for each. Repeat one estimate a week later without viewing the first to test within-person noise. Practise asking for the strongest disconfirming case in neutral language.
Conclude by designing a panel protocol that protects junior independence, records revisions and distinguishes convergence caused by evidence from convergence caused by authority.
Working through Appraising Professional Expertise in MGMT7250? Sia is AskSia’s AI Management tutor — ask any MGMT7250 Appraising Professional Expertise question and get a clear, step-by-step explanation grounded in how MGMT7250 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.