Machine Learning (ML) · Intermediate level

Intermediate level · Module 3: Hyperparameter Tuning · Lesson 2 of 3

Honest Estimates after Tuning

Keep performance estimates honest when tuning, using a held-out test set or nested cross-validation.

In this lesson you will

  • Explain why tuning makes validation scores optimistic
  • Use nested cross-validation or a final test set for honest estimates

Key concepts

optimistic biasnested cross-validationheld-out test setreporting

The lesson

The best cross-validation score from a search is optimistic: you chose it because it was the best among many tries, partly by luck.

Report final performance on a test set that played no part in tuning. When data is too small to spare a test set, nested cross-validation tunes in inner folds and evaluates in outer folds.

Report the search, the chosen settings and the final test result together, so readers can judge the work.

Step by step

  1. Compare the best search score with the test score.
  2. Run a small nested cross-validation.
  3. Explain the difference between the estimates.
  4. Write the final results paragraph.

Practice activity

Report the tuned agent model's final performance honestly.

Expected result: A results paragraph with the search summary and a single test-set result.

Check your understanding

  1. Why is the best search score optimistic?

    Show answer

    It was selected as the best of many attempts, partly due to chance.

  2. What does nested cross-validation do?

    Show answer

    Tunes in inner folds and evaluates in outer folds to give an honest estimate.

Taught live by an instructor — online or in person. Free course lessons are for self-study and do not lead to a certificate — see registration options.