Intermediate level · Module 2: Cross-Validation and Model Selection · Lesson 3 of 3
Metrics and Thresholds from Real Costs
Choose the metric and decision threshold that minimise the real cost of errors.
In this lesson you will
- Read precision–recall and ROC curves
- Choose a threshold that minimises expected cost
Key concepts
precision–recall curveROC curve and AUCexpected costdecision threshold
The lesson
Different errors cost different amounts. Missing an agent who runs out of cash loses customers and fees; sending cash unnecessarily costs a trip. Estimate each cost, even roughly.
Precision–recall curves show the trade-off as the threshold changes and are informative when positives are rare. ROC curves and their area (AUC) summarise ranking quality across thresholds.
Choose the threshold that minimises expected cost on validation data, then report the resulting precision and recall in plain terms.
Step by step
- Estimate the cost of a missed shortage and a false alarm.
- Plot the precision–recall curve.
- Calculate expected cost for several thresholds.
- Choose the threshold with the lowest cost.
Practice activity
Complete the cost-based model and threshold choice from the practical activity.
Expected result: A threshold chosen by expected cost, explained in business terms.
Check your understanding
-
When are precision–recall curves especially useful?
Show answer
When the positive class is rare.
-
How should a threshold be chosen?
Show answer
By minimising expected cost, or by meeting the required precision or recall, on validation data.
Taught live by an instructor — online or in person. Free course lessons are for self-study and do not lead to a certificate — see registration options.