11 / 12 · Make a fair comparison
What does a small p-value actually say?
Read a p-value as a question under an assumption, not the probability that a claim is true.
Builds on Better results, or different people?
Go to practice ↓A question to keep in mind
If p is about 0.04, does that mean the new method has a 96% chance of being better?
No. You cannot subtract p from 1 to get the probability that the new method is better. A p-value asks about possible test results while treating a starting model as true. It does not give probabilities to the competing explanations.
For example, assume a model in which the methods have no real difference, together with the test's other assumptions. Under a fixed test rule, how likely is a test statistic this extreme or more extreme? That probability is the p-value. 'More extreme' is defined by the chosen test, not by changing the rule after seeing the result.
In the example below, count all runs that are at least as extreme, not just exact copies of the observation. A small p-value signals an unusual result under the assumed model. It does not tell you the effect's size, its practical importance, or whether another explanation is correct.
Work through an example
- Fix the no-difference model, assumptions, and what counts as 'at least as extreme' before looking at the result.
- In these invented teaching counts, 40 of 1,000 runs meet that rule. Every run uses the same starting model; 40 is a count of qualifying results, not a count of false hypotheses.
- 40/1000 = 4%, so the simulation gives a rough p-value of 0.04. This calculation keeps the starting model assumed.
Your turn
0 / 3Which interpretation of a small p-value is appropriate?
Read solution · does not award completion
The chosen test result is unusual under the assumed model.
Do not reverse 'probability of a result under a model' into 'probability of a model given a result'. It is the conditional-probability distinction from lesson 3.
A very large experiment finds a tiny improvement with a small p-value. What else is needed before deciding whether it is useful?
Read solution · does not award completion
The effect size, uncertainty, study design, and practical costs or benefits.
Read the full result: how big, how uncertain, measured how, and useful for what. A threshold alone cannot answer those questions.
Bring it back to your own work
Complete this sentence: 'A small p-value does not tell me ____. To decide what it means here, I would also ask ____.'