Data Science · Intermediate level

Intermediate level · Module 5: Reporting and Dashboards · Lesson 1 of 3

Reproducible Analysis

Organise analysis so that anyone can regenerate every number from the raw data.

In this lesson you will

  • Structure an analysis project for reproducibility
  • Run the full pipeline from raw data with one command

Key concepts

raw vs processed datascripts and notebooksone-command rebuildversion control

The lesson

Keep raw data unchanged, write every cleaning and analysis step in code and save outputs separately. Then a new data delivery or a corrected error needs only a rerun.

Organise projects into folders — data/raw, data/processed, notebooks or scripts, outputs — with a README explaining how to run them.

A single command, such as a script that runs each step in order, should rebuild everything. Use Git to track changes to code and documents.

Step by step

  1. Reorganise the project into the folder structure.
  2. Move manual steps into code.
  3. Write a run script for the whole pipeline.
  4. Rebuild on a classmate's computer.

Practice activity

Make the education analysis rebuild from raw data with one command.

Expected result: A classmate regenerates all outputs from the raw files using the README.

Check your understanding

  1. Why keep raw data unchanged?

    Show answer

    So every step can be repeated and checked from the original source.

  2. What is the test of reproducibility?

    Show answer

    Someone else can regenerate the outputs from the raw data.

Taught live by an instructor — online or in person. Free course lessons are for self-study and do not lead to a certificate — see registration options.