Intermediate level · Module 5: Testing Changes and Causal Thinking · Lesson 2 of 3
Designing Fair Tests
Design pilots that can show whether a change works.
In this lesson you will
- Compare before-after and comparison-group designs
- Design a pilot with a fair comparison group
Key concepts
before-aftercomparison grouprandom assignmentpilot
The lesson
A before-after comparison is simple but other things change over time — seasons, prices, competitors.
A comparison group that does not receive the change shows what would have happened anyway. Randomly choosing which customers or branches receive the change makes groups comparable.
Agree the metric, duration and decision rule before the pilot starts, so results are not interpreted to suit expectations.
Step by step
- Define the change to test.
- Choose the metric and duration.
- Choose and assign groups.
- Write the decision rule in advance.
Practice activity
Design the SMS reminder pilot.
Expected result: A pilot design with groups, metric, duration and decision rule.
Check your understanding
-
Why use a comparison group?
Show answer
It shows what would have happened without the change.
-
Why set the decision rule before the pilot?
Show answer
To avoid interpreting results to suit expectations.
Taught live by an instructor — online or in person. Free course lessons are for self-study and do not lead to a certificate — see registration options.