Causal reasoning · 5/6
Design a small experiment before seeing results
Specify assignment, outcomes, follow-up and stopping rules.
About 15–20 minutes. The following cases are fictional.
Record your first judgment
A team sends subject A to existing customers and B to new ones. It checks daily and stops when B leads. Is that a fair comparison?
Understand the judgment process
Subject and customer type are entangled. Randomize A or B within each customer type and combine results as prespecified. Randomization makes groups comparable in expectation, not identical in every small sample.
Choose a primary outcome, a useful effect size and a follow-up period in advance. For example, clicks within seven days per delivered email, with unsubscribe monitoring. Email opens can depend on client software; connect the metric to the actual goal.
Stopping at the first favorable difference alters false-positive risk. Formal testing needs a prespecified analysis or an appropriate sequential method. A small low-risk pilot may test feasibility, but a tiny lead is not robust evidence.
Track assignment, receipt, crossover and missingness. Analysis by original assignment usually addresses the effect of assignment. Restricting analysis to users can undo randomization. Prespecified harm rules may stop a trial, with transparent reporting.
Checks you can perform
- Define participants, assignment units and strata.
- Specify the intervention contrast and record other changes.
- Prespecify the outcome, follow-up, analysis and sample rationale.
- Set harm monitoring, stopping and rollback rules; account for everyone assigned.
Guided practice
Rewrite the email experiment: assignment, primary outcome, period, stopping rule and scope.
Explore the explanation
Randomize within new and existing customers; measure seven-day clicks per delivery; complete the prespecified sample and follow-up; apply a predefined unsubscribe harm threshold and report stopping; interpret a pilot with uncertainty and practical importance. Seven days is this example’s choice, not a universal rule.
Apply it in a new context
A learner studies with music one day and in silence the next. What else matters?
Compare after attempting
Material difficulty, time, order, fatigue and carryover. Repeat randomized conditions using comparable materials and a common delayed test. A personal result does not automatically generalize.
Preserve a revision record
No universal sample size is provided. Requirements depend on variation, the target difference and the analysis.
Check whether you supplied inspectable evidence, a feasible next step and revision conditions. Revealing an answer is not mastery.
Source and scope
NIST/SEMATECH — General design principles
The source provides conceptual or methodological background. Cases, steps and exercises are authored here; they are not the original experiment or evidence of this course’s effectiveness.
Completion is a personal record, not proof of mastery or certification.