Advanced level · Module 5: MLOps and Monitoring · Lesson 2 of 4
Monitoring Models in Production
Detect when inputs, predictions or performance change, even before true outcomes are known.
In this lesson you will
- Monitor input and prediction distributions
- Measure performance as delayed labels arrive
Key concepts
input driftprediction driftdelayed labelsalert thresholds
The lesson
Monitor three things: inputs (are features distributed as in training?), predictions (has the forecast distribution shifted?) and performance (how accurate were forecasts once real prices were known?).
True outcomes often arrive late — next month's price is known only next month. Input and prediction drift give earlier warnings.
Set alert thresholds that matter, such as error 50% above the backtest level for two months, and make sure alerts reach someone who will act.
Step by step
- Compute feature distribution statistics for training data.
- Compare them with simulated new data.
- Compute forecast errors as actual prices arrive.
- Define alert thresholds.
Practice activity
Detect the simulated price shock through monitoring and document the alerts.
Expected result: Monitoring outputs that detect the shock, with clear alert rules.
Check your understanding
-
Why monitor input drift when labels are available later?
Show answer
It gives an early warning before performance can be measured.
-
What makes an alert useful?
Show answer
It reflects a meaningful change and reaches someone who will act.
Taught live by an instructor — online or in person. Free course lessons are for self-study and do not lead to a certificate — see registration options.