Free course · Lesson 5 of 5

Model Optimization: Loss Functions, Optimizers & Learning Rate

In this lesson you will

Read a training loss curve
Choose basic remedies for overfitting

Training a deep learning model is an optimization problem. The loss function measures how wrong the model's predictions are. The optimizer (like Adam) adjusts the weights to minimize the loss. The learning rate controls how big each adjustment is — too high and the model jumps around; too low and it learns painfully slowly. Choosing the right combination is key to fast, accurate training.

Loss functions: CrossEntropyLoss for classification, MSELoss for regression.
Adam optimizer: adapts learning rates per-parameter — works well out of the box for most tasks.
Learning rate schedulers: reduce the learning rate over time (e.g., ReduceLROnPlateau) for fine-grained convergence.
Monitor training with TensorBoard or simple accuracy/loss plots in Jupyter Notebooks.

💡 Practical Skill: Start with Adam and a learning rate of 0.001. If the loss plateaus, use a scheduler to reduce the rate. Use Google Colab to experiment with different configurations quickly.

Worked exampleLoss Function & Optimizer Setup (PyTorch)

Python

# Setting up loss function and optimizer in PyTorch# Install: pip install torch import torchimport torch.nn as nnimport torch.optim as optim # --- Define a simple neural network ---class SimpleNN(nn.Module):    def __init__(self):        super().__init__()        self.fc1 = nn.Linear(10, 32)   # input: 10 features        self.fc2 = nn.Linear(32, 1)    # output: 1 value     def forward(self, x):        x = torch.relu(self.fc1(x))        x = self.fc2(x)        return x model = SimpleNN() # --- Step 1: Choose a loss function ---# For binary classification (e.g., flooded / not flooded)criterion = nn.BCEWithLogitsLoss() # --- Step 2: Choose an optimizer ---optimizer = optim.Adam(    model.parameters(),    lr=0.001  # learning rate: 0.001 is a great default for Adam) # --- Step 3: Optional learning rate scheduler ---scheduler = optim.lr_scheduler.ReduceLROnPlateau(    optimizer,    mode='min',     # reduce when loss stops decreasing    factor=0.5,     # cut learning rate in half    patience=5      # wait 5 epochs before reducing) print("\u2705 Loss function: BCEWithLogitsLoss (binary classification)")print("\u2705 Optimizer: Adam with lr=0.001")print("\u2705 Scheduler: ReduceLROnPlateau (patience=5)") # --- Simulate one training step ---# Dummy data: batch of 8 samples, each with 10 featuresinputs = torch.randn(8, 10)        # random inputlabels = torch.randint(0, 2, (8, 1)).float()  # random labels (0 or 1) # Forward passoutputs = model(inputs)loss = criterion(outputs, labels) # Backward pass & optimization stepoptimizer.zero_grad()  # clear previous gradientsloss.backward()        # compute gradientsoptimizer.step()       # update weights print(f"\nSample training step completed!")print(f"   Loss: {loss.item():.4f} (should decrease over epochs)")print(f"   Learning rate: {optimizer.param_groups[0]['lr']}")

Practice exercise

Describe what a loss curve looks like when the learning rate is too high, and list two ways to reduce overfitting.

Key takeaways

A rising or jumping loss usually means the learning rate is too high
Augmentation, dropout and early stopping reduce overfitting
Change one setting at a time and record the result

In the full program Module 5: Training Neural Networks Well See how this topic is taught in depth, with practical activities and an assessment.

Free self-study lesson — no certificate. For the full instructor-led program, see registration options.