The University of Manchester · FACULTY OF ECONOMICS

ECON31031 Chap.5 Binary Model Accuracy, ROC Curves and Link Tests

- one subject, every graph, every model, every mark
7 Chapters4-page Bible
Our own words - no uploaded lecturer files
Updated for this semester
Chapter 5 of 9 · ECON31031

Binary Model Accuracy, ROC Curves and Link Tests

Binary Model Accuracy, ROC Curves and Link Tests covers how to judge a fitted binary model. Fitted probabilities become predicted outcomes through a cut-off, usually 0.5, and the resulting two-by-two hit and miss table yields the hit count, the Kuipers score, precision, recall, the F1 score and the phi coefficient. Likelihood-based pseudo-R-squared measures and the Brier score summarise fit in sample.

Varying the cut-off traces the ROC curve, and the area under it, computed by the trapezium rule, is graded from poor to excellent by rules of thumb. When successes are rare, precision, recall and F1 are better guides. Finally, the link test adds powers of the fitted index to check for misspecification. The module's benefits application illustrates every measure.

At the 0.5 cut-off all three fitted models predict almost everyone to be a success, so hit counts near 0.69 sit beside Kuipers scores close to zero, and the AUC with many cut-offs is only about 0.63. Reading several measures together is therefore the skill being examined, not computing any single one.

In this chapter

What this chapter covers

  • 01

    In-sample fitted values and out-of-sample prediction

  • 02

    Hit and miss tables at a cut-off

  • 03

    Hit count, Kuipers score, precision, recall and F1

  • 04

    Pseudo-R-squared measures and the Brier score

  • 05

    ROC curves and the AUC by the trapezium rule

  • 06

    Imbalanced outcomes and the phi coefficient

  • 07

    The link test and generalised RESET

Worked example · free

Compute an AUC with the trapezium rule

Q [4 marks]. Sorted ROC points for a binary model are (0, 0), (0.1, 0.5), (0.3, 0.8), (0.6, 0.95) and (1, 1), given as (false positive rate, true positive rate). Compute the area under the curve and grade it with the rules of thumb in the notes. This practice weighting is assigned by AskSia and is not an official assessment scheme.
Each dot pairs the false positive rate with the true positive rate at one cut-off for twelve invented people.Each dot pairs the false positive rate with the true positive rate at one cut-off for twelve invented people. Joining the sorted dots and summing trapezia gives an area under the curve of 0.889, well above the dashed chance line.cut-off 0.6cut-off 0.4000.50.511AUC 0.889chance linefalse positive ratetrue positive rate
Each dot pairs the false positive rate with the true positive rate at one cut-off for twelve invented people. Joining the sorted dots and summing trapezia gives an area under the curve of 0.889, well above the dashed chance line.
  • 3The trapezia contribute 0.1 × 0.25 = 0.025, 0.2 × 0.65 = 0.130, 0.3 × 0.875 = 0.2625 and 0.4 × 0.975 = 0.390.
  • 1The sum is 0.8075, which falls in the 0.8 to 0.9 band labelled good.
The AUC is about 0.81, a good classifier by the rules of thumb.
Sia tip — Sort the points by false positive rate before summing, and use the average of the two true positive rates as each trapezium's height.
Glossary

Key terms

Confusion matrix
The two-by-two table of predicted against observed outcomes, also called the hit and miss table.
Kuipers score
The hit rate minus the false alarm rate for a set of binary predictions.
Brier score
The mean squared difference between binary outcomes and their fitted probabilities.
Link test
A misspecification check that adds powers of the fitted index to a binary model and tests them.
FAQ

Binary Model Accuracy, ROC Curves and Link Tests FAQ

Why can a high hit count be misleading?

When one outcome dominates, a model that predicts the common outcome for almost everyone scores a high hit count while separating the groups badly. Kuipers, precision, recall and the AUC reveal that weakness, as the benefits example in the notes shows.

What does an AUC of 0.63 mean?

It means the curve sits only modestly above the diagonal chance line, so the model separates successes from failures weakly. The rules of thumb in the notes call 0.5 to 0.7 poor, 0.7 to 0.8 fair, 0.8 to 0.9 good and above 0.9 excellent; the module's benefits models score about 0.63.

What does the link test detect?

It detects that powers of the fitted index improve the model, which signals misspecification such as a missing nonlinearity or the wrong link. It does not say which problem is present, only that the current specification is rejected.

What is the difference between precision and recall?

Precision is the share of predicted successes that are real successes, while recall is the share of real successes that the model predicts. Raising the cut-off usually increases precision and lowers recall, and the F1 score combines the two into one number between zero and one.

What does the Brier score measure?

It is the mean squared difference between observed zero-one outcomes and fitted probabilities, so lower is better. A model that assigns 0.5 to everyone scores 0.25, and applying the same formula to zero-one predictions instead of probabilities gives one minus the hit count.

Why is the ROC curve built from several cut-offs?

A single cut-off gives only one point, so the curve reduces to two straight segments. Using many cut-offs between zero and one traces how the true and false positive rates trade off, which gives a fuller picture and a more informative area under the curve.

Study strategy

Exam move

Build a hit and miss table from scratch for a handful of invented probabilities and outcomes, then compute every measure from it until the formulas are automatic. Draw the ROC curve for several cut-offs and calculate the AUC by hand. Memorise the AUC bands and the reason imbalance matters.

For the link test, practise the likelihood ratio version: restricted and unrestricted log-likelihoods, number of restrictions, statistic and verdict. When you meet an output table, read it in a fixed order: hit count, Kuipers, precision and recall, then pseudo-R-squared and the AUC, and finish with a sentence on whether the model separates the groups.

That order prevents the most common misreading, which is praising a high hit count in imbalanced data.

Working through Binary Model Accuracy, ROC Curves and Link Tests in ECON31031? Sia is AskSia’s AI Economics tutor — ask any ECON31031 Binary Model Accuracy, ROC Curves and Link Tests question and get a clear, step-by-step explanation grounded in how ECON31031 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.

A+Everything unlocked
Unlocks this Bible + all 41 of your subjects at The University of Manchester, plus 1,000+ more Bibles across the AskSia library.
Sia - your ECON31031 tutor, unlimited, worked the way the exam marks it
The full 4-page Bible + practice bank with worked solutions
Chrome extension - sync your LMS so Sia knows your deadlines
Bilingual EN / Chinese on every Bible and every Sia answer
$0.99 Trial
30-day money-back · cancel in one tap · how it works