Advanced level · Module 3: Working with Larger and Spatial Data · Lesson 1 of 4
Efficient Data Processing
Work with data that strains memory by choosing types, file formats and chunked processing.
In this lesson you will
- Reduce memory use with dtypes and columnar formats
- Process large files in chunks
Key concepts
memory usagecategorical dtypeParquetchunked reading
The lesson
Check memory with df.info(memory_usage="deep"). Converting repeated text to the category dtype and using smaller numeric types can cut memory dramatically.
Parquet stores data by column, compressed and with types preserved; it loads much faster than CSV and lets you read only the columns you need.
For files larger than memory, read in chunks (read_csv with chunksize), aggregate each chunk and combine the results.
Step by step
- Measure a large DataFrame's memory use.
- Convert columns to efficient types.
- Save as Parquet and compare load times.
- Aggregate a large CSV in chunks.
Practice activity
Prepare the large transaction file for efficient analysis and record the gains.
Expected result: Documented memory and time improvements and a working chunked aggregation.
Check your understanding
-
Why is Parquet faster than CSV?
Show answer
It is columnar, compressed and keeps types, so less data is read and parsed.
-
When is chunked processing needed?
Show answer
When data does not fit in memory at once.
Taught live by an instructor — online or in person. Free course lessons are for self-study and do not lead to a certificate — see registration options.