Deep Learning (DL) · Advanced level

Advanced level · Module 1: Attention and Transformers · Lesson 2 of 4

The Transformer Architecture

See how attention, feed-forward layers, residual connections and positional information form a transformer.

In this lesson you will

  • Describe encoder and decoder transformers
  • Explain positional encoding and the role of residual connections

Key concepts

encoder and decodermulti-head attentionpositional encodinglayer normalisation

The lesson

A transformer layer combines multi-head self-attention and a feed-forward network, each wrapped with residual connections and layer normalisation. Many layers are stacked.

Attention ignores order by itself, so positional information is added to each token's embedding.

Encoder models such as BERT read the whole input and suit classification; decoder models such as GPT generate text one token at a time; encoder–decoder models such as T5 suit translation and summarisation.

Step by step

  1. Label the parts of a transformer layer diagram.
  2. Match tasks to encoder, decoder or encoder–decoder models.
  3. Inspect the configuration of a pre-trained model.
  4. Count its layers and parameters.

Practice activity

Write a one-page explanation of a transformer layer for a classmate who knows CNNs.

Expected result: An accurate explanation linking attention, positional information and residual connections.

Check your understanding

  1. Why do transformers need positional information?

    Show answer

    Attention alone does not know the order of tokens.

  2. Which type of transformer suits text classification?

    Show answer

    An encoder model.

Taught live by an instructor — online or in person. Free course lessons are for self-study and do not lead to a certificate — see registration options.