Deep Learning (DL) · Advanced level

Advanced level · Module 1: Attention and Transformers · Lesson 3 of 4

Pre-trained Models and Tokenisers

Load pre-trained transformers and their tokenisers, and understand what they can and cannot handle.

In this lesson you will

  • Load models and tokenisers from a model hub
  • Assess language coverage and licences

Key concepts

model hubtokenisermultilingual modelsmodel licences

The lesson

Libraries such as Hugging Face Transformers provide thousands of pre-trained models with matching tokenisers. The tokeniser must be the one the model was trained with.

Multilingual models cover many languages, but coverage varies: widely spoken languages work well, while languages with little online text — including many in South Sudan — may be split into many meaningless pieces and handled poorly.

Read each model's card for its training data, intended use, limitations and licence before using it.

Step by step

  1. Load a small multilingual model and its tokeniser.
  2. Tokenise sentences in English, Arabic and a local language.
  3. Compare token counts and splits.
  4. Read the model card and note limitations.

Practice activity

Choose a pre-trained model for the SMS feedback task and document why.

Expected result: A justified choice covering language coverage, size, limitations and licence.

Check your understanding

  1. Why must the matching tokeniser be used?

    Show answer

    The model was trained on that tokeniser's tokens.

  2. What sign shows a language is poorly covered?

    Show answer

    Words split into many small, meaningless pieces.

Taught live by an instructor — online or in person. Free course lessons are for self-study and do not lead to a certificate — see registration options.