Machine Learning (ML) · Advanced level

Advanced level · Module 1: Ensemble Methods · Lesson 2 of 4

Gradient Boosting

Build strong models by adding trees that correct previous errors, and control overfitting with learning rate and early stopping.

In this lesson you will

  • Explain how boosting differs from bagging
  • Tune gradient boosting with learning rate and early stopping

Key concepts

boostinglearning rateearly stoppingHistGradientBoosting / XGBoost / LightGBM

The lesson

Boosting builds trees one after another, each fitted to the errors of the trees before it. Many small corrections add up to a very accurate model — gradient boosting often wins on tabular data.

Boosting can overfit if it keeps going. A small learning rate with many trees, limited tree depth and early stopping on a validation set keep it in check.

Libraries such as XGBoost and LightGBM, and scikit-learn's HistGradientBoosting, implement fast versions that handle large data and missing values.

Step by step

  1. Train HistGradientBoosting with default settings.
  2. Add early stopping and a lower learning rate.
  3. Tune max_depth or max_leaf_nodes.
  4. Compare with the random forest.

Practice activity

Tune a gradient boosting model for the food-insecurity data and compare it with the random forest.

Expected result: A tuned boosting model with a fair comparison and documented settings.

Check your understanding

  1. How does boosting differ from bagging?

    Show answer

    Boosting trains models sequentially to correct earlier errors; bagging trains them independently and averages.

  2. What does early stopping do?

    Show answer

    Stops adding trees when validation performance stops improving.

Taught live by an instructor — online or in person. Free course lessons are for self-study and do not lead to a certificate — see registration options.