Data Science · Advanced level

Advanced level · Module 4: Text Data and Open-Ended Responses · Lesson 1 of 4

Cleaning and Exploring Text

Prepare text for analysis and find frequent words and phrases.

In this lesson you will

  • Clean and tokenise text
  • Summarise frequent words and phrases

Key concepts

tokenisationstop wordsn-gramsspelling variation

The lesson

Text analysis starts with cleaning: lower-casing, removing punctuation, handling spelling variations and splitting text into words (tokens).

Remove very common words (stop words) and count frequent words and two-word phrases (bigrams) to see what people talk about.

Keep mixed languages and local terms in mind: a standard stop-word list may remove words that matter, and key terms may have several spellings.

Step by step

  1. Clean the feedback messages.
  2. Count the most frequent words.
  3. Count frequent bigrams.
  4. List spelling variants to merge.

Practice activity

Produce a first exploration of the feedback text.

Expected result: Frequency tables and notes on cleaning decisions and language issues.

Check your understanding

  1. What is a bigram?

    Show answer

    A pair of consecutive words.

  2. Why be careful with standard stop-word lists?

    Show answer

    They may remove meaningful words, especially in mixed-language text.

Taught live by an instructor — online or in person. Free course lessons are for self-study and do not lead to a certificate — see registration options.