10 / 12 · Make a fair comparison
Better results, or different people?
See why a difference alone does not tell you what caused it.
Builds on An estimate is not an exact address
Go to practice ↓A question to keep in mind
One class uses a new exercise app and scores higher. Does that prove the app caused it?
The classes might already differ in prior knowledge, study time, or teaching. A factor that differs between the groups can offer another explanation for their scores.
Random assignment gives the comparison a better starting point: chance decides who uses each method. Keep the tasks, time, and scoring comparable too. Random assignment helps with causal comparison; random sampling is about which wider group you can describe.
Work through an example
| Method | Passed | Total |
|---|---|---|
| New | 16 | 20 |
| Old | 12 | 20 |
- Before collecting results, assign the 40 volunteers randomly to two groups of 20.
- With comparable tasks and scoring, observe 16 passes versus 12.
- The observed rates are 80% and 60%. Random assignment helps interpretation, but uncertainty and the study population still matter.
Your turn
0 / 3Which design better isolates the effect of two study methods?
Read solution · does not award completion
Randomly assign volunteers, then use comparable tasks, time, and scoring.
A fair comparison needs more than two numbers. How groups were formed and measured matters.
In the table, calculate the two pass rates and their difference.
Read solution · does not award completion
- New method rate: 80 %
- Old method rate: 60 %
- Difference in rates: 20 points
16/20 = 80%, 12/20 = 60%, a 20-percentage-point observed difference.
Which process is about selecting people, and which is about assigning methods?
Read solution · does not award completion
- Draw names at random from all registered learners. → Random sampling: selecting people
- Use a random allocation to decide which volunteers use each method. → Random assignment: allocating methods
Selecting participants concerns who the results may describe. Assigning treatments concerns the comparison between methods. A study may do one, both, or neither.
Apply this lesson to an AI answer · synthetic teaching example
A higher score does not identify the cause
Synthetic classroom example: students chose whether to use an AI tutor. Their later average scores were 82 for users and 74 for non-users. No baseline scores were recorded.
The AI tutor raised scores by 8 points.
What can this comparison establish?
Think first, then inspect the reasoning
- 82 − 74 = 8 is a difference between two groups, not a before-and-after change for the same students.
- Students selected their own group. Prior ability, motivation or study time could differ, so the comparison does not isolate the tutor's effect.
- A stronger study would record the starting level, assign the intervention randomly where appropriate, use the same assessment, and report sample sizes and uncertainty.
A conclusion supported by the information
AI-tutor users averaged 8 points more in this sample. This comparison alone does not show that the tutor caused the difference.
When an assistant changes ‘associated with’ into ‘caused’, look for the assignment method and baseline data. Mark missing information as missing.
Bring it back to your own work
Design a small comparison of two study routines. State who joins, how routines are assigned, and one outcome measured the same way for everyone.
Statistical literacy: directory, audit and transfer assessment
How do I retain independent judgment when using AI here?
Keep your own answer before consulting an AI candidate. Check sources, calculations, inference and conditions; explain adoption or rejection. Then solve a new-context task independently and apply the checks to this lesson.
Practice the verification workflow and keep a recordMy learning status
Mark statuses manually; visits do not establish mastery. Languages share statuses while lessons keep their own answers. This backup contains statuses only, not answers or notes.