ECON7880 Chap.8 Model Evaluation and Project Decisions
Model Evaluation and Project Decisions
Preserve both error directions
A confusion matrix keeps true positives, false positives, false negatives and true negatives separate.
This chapter derives precision, recall and accuracy from the correct denominators and connects each measure to a threshold, workload and error cost.
Finish the project with a decision
The group project applies CRISP-DM to a selected dataset and ends in a short presentation.
A defensible storyline states the business question, target, preparation boundary, validation design, model comparison, operating point and deployment recommendation.
Label counts before calculating ratios
Put actual class on rows and predicted class on columns, then mark all four cells. Precision uses all predicted positives in its denominator, while recall uses all actual positives.
Accuracy combines correct predictions and can look strong when negatives dominate. Keeping raw counts beside ratios makes workload and mistakes visible and allows every metric to be reconstructed during questions.
Use curves for threshold comparison
An ROC curve follows true-positive rate against false-positive rate as the threshold changes.
A precision-recall curve emphasises positive predictions and is often more informative for rare events. Area summaries describe ranking over many thresholds, but the best curve overall need not dominate in the narrow capacity region available to the organisation.
Mark the selected operating point and print its confusion counts.
Connect evaluation to capacity and cost
A review team may handle only a fixed number of cases. Choose a threshold that produces a feasible queue, then report precision in that queue, recall across all positives and both error counts.
If missing a positive is much more costly than an unnecessary review, the organisation may accept lower precision to gain recall. The trade-off belongs in the conclusion rather than being hidden behind accuracy.
Build the presentation around decisions
Use CRISP-DM as the story: business need, row and target, preparation boundary, candidate methods, validation evidence, operating point and deployment implication.
The published presentation includes question time, so reserve detail for assumptions most likely to change the result. A limitation is useful when paired with a next check, pilot rule or monitoring trigger rather than offered as a vague disclaimer.
What this chapter covers
- 01
Confusion-matrix counts
- 02
Precision and recall
- 03
Accuracy under imbalance
- 04
ROC and precision-recall curves
- 05
Capacity-based thresholds
- 06
CRISP-DM project defence
Worked example · free
Evaluate a capacity-matched classifier
- 2Label all four counts.
- 4Use the correct denominators.
- 3Connect metrics to capacity and errors.
Key terms
- Precision
- The share of predicted-positive cases that are actually positive at a stated threshold.
- Recall
- The share of actual-positive cases found by the model at a stated threshold.
- Operating Point
- A selected threshold and its resulting combination of workload, true positives and errors.
Model Evaluation and Project Decisions FAQ
Why can accuracy mislead on rare events?
Predicting the common negative class can create high accuracy while finding few positive cases. Inspect confusion counts, precision, recall and workload at the chosen threshold.
What does an ROC curve compare?
It plots true-positive rate against false-positive rate across thresholds. The curve assesses ranking, but a business still must select an operating point from error costs, review capacity and expected workload.
What should the project conclusion contain?
State whether evidence supports deployment, a pilot, more data or stopping. Include the chosen threshold, expected workload, main error, limitation and monitoring trigger, then name the person or process responsible for reviewing those signals after launch. A conclusion without ownership, review timing and escalation is not deployable or ready for a responsible pilot.
Specify what result would pause, expand or end the pilot, and when that decision will be made from observed outcomes.
Exam move
Build one confusion matrix from raw counts and calculate every metric with labelled denominators. Change the threshold and predict the direction of precision, recall and queue size before recomputing. Sketch an ROC point and explain why a curve cannot choose the business policy by itself.
Then rehearse a two-minute project conclusion that names the question, split, baseline, selected threshold, workload, error trade-off, limitation and next organisational action. Keep the raw counts ready so every reported ratio can be defended during questions. Prepare answers to three challenges: why the split matches deployment, why the baseline is fair and why the chosen threshold fits capacity.
Turn the main limitation into a pilot measurement or stopping rule. Time the explanation so enough of the published presentation remains for questions rather than compressing the final recommendation. Prepare one backup table with the untouched test counts, baseline and threshold workload so every headline can be reconstructed.
Working through Model Evaluation and Project Decisions in ECON7880? Sia is AskSia’s AI Economics tutor — ask any ECON7880 Model Evaluation and Project Decisions question and get a clear, step-by-step explanation grounded in how ECON7880 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
