Advanced level · Module 5: Data Analysis with pandas · Lesson 2 of 4
Cleaning and Transforming Data
Fix missing values, inconsistent spellings, wrong types and duplicates.
In this lesson you will
- Handle missing values and duplicates
- Standardise text and convert types
Key concepts
NaNfillna/dropnadrop_duplicatesastype and to_datetime
The lesson
Real datasets have inconsistent spellings ('Juba', 'juba ', 'JUBA'), missing values and duplicates. Analysis on dirty data gives wrong answers, so cleaning comes first.
Use string methods on whole columns (df['county'].str.strip().str.title()), replace() to map known misspellings, drop_duplicates() for repeated rows and isna() to find missing values. Decide case by case whether to fill, drop or flag missing data.
Keep a cleaning log: what you changed, why and how many rows were affected. It makes your results trustworthy and repeatable.
Step by step
- Count missing values and duplicates.
- Standardise county names and map known misspellings.
- Remove exact duplicates.
- Record each step and its row counts in a cleaning log.
Practice activity
Clean a messy health-facility dataset with inconsistent county spellings, missing values and duplicated rows; document every cleaning decision.
Expected result: A clean dataset plus a written cleaning log.
Check your understanding
-
Why document data-cleaning decisions?
Show answer
They affect results, and others must be able to understand and reproduce them.
Taught live by an instructor — online or in person. Free course lessons are for self-study and do not lead to a certificate — see registration options.