Intermediate level · Module 2: Cross-Validation and Model Selection · Lesson 1 of 3
Cross-Validation
Get a more reliable estimate of performance by training and evaluating on several different splits.
In this lesson you will
- Run k-fold and stratified cross-validation
- Interpret the mean and spread of scores
Key concepts
k-foldstratified k-foldcross_val_scorescore variance
The lesson
A single train/validation split can be lucky or unlucky. k-fold cross-validation splits the training data into k parts, trains on k−1 parts and validates on the remaining one, k times, so every row is used for validation once.
Stratified k-fold keeps class proportions in each fold, which matters for imbalanced problems such as rare cash shortages.
Report the mean score and its spread. Scores of 0.81 ± 0.01 and 0.82 ± 0.08 tell different stories: the second model is less predictable.
Step by step
- Run 5-fold cross-validation on the pipeline.
- Switch to StratifiedKFold.
- Print the mean and standard deviation of the scores.
- Compare with a single split's result.
Practice activity
Estimate the agent pipeline's performance with stratified 5-fold cross-validation.
Expected result: A mean and spread for the chosen metric, with a note on how it differs from a single split.
Check your understanding
-
Why use cross-validation instead of one split?
Show answer
It gives a more reliable estimate by averaging over several splits.
-
When is stratification important?
Show answer
When classes are imbalanced.
Taught live by an instructor — online or in person. Free course lessons are for self-study and do not lead to a certificate — see registration options.