Deep Learning (DL) · Beginner level

Beginner level · Module 3: Sequence Models for Text and Time · Lesson 2 of 3

Recurrent Networks and LSTMs

Process sequences step by step with recurrent networks, and see how LSTMs keep longer memory.

In this lesson you will

  • Explain how an RNN carries a hidden state through a sequence
  • Train an LSTM classifier

Key concepts

hidden staterecurrent networkvanishing gradientsLSTM

The lesson

A recurrent neural network (RNN) reads a sequence one step at a time, keeping a hidden state that summarises what it has seen so far. The final state can be used to classify the whole sequence.

Simple RNNs struggle to remember information from many steps back because gradients shrink as they flow backwards (vanishing gradients).

LSTMs add gates that decide what to keep, forget and output, so they remember longer. In PyTorch, nn.LSTM processes a batch of sequences in one call.

Step by step

  1. Build a model with an embedding layer, an LSTM and a classifier.
  2. Train it on the SMS data.
  3. Evaluate it on test messages.
  4. Inspect misclassified messages.

Practice activity

Complete the SMS classification practical activity.

Expected result: A trained LSTM classifier with test results and an error analysis.

Check your understanding

  1. What does an RNN's hidden state hold?

    Show answer

    A summary of the sequence seen so far.

  2. What problem do LSTMs reduce?

    Show answer

    Vanishing gradients, which make long-range memory hard.

Taught live by an instructor — online or in person. Free course lessons are for self-study and do not lead to a certificate — see registration options.