Back to Glossary
Glossary
P-Value
The probability of seeing results as extreme as yours if there were no real difference.
What is a P-Value?
A p-value is the probability of observing results as extreme as yours (or more extreme) if there were no real difference between variants.
A low p-value (below 0.05) means it's unlikely the results are due to random chance, so we call them "statistically significant."
Common Misinterpretation
❌ Wrong interpretation
"p = 0.03 means there's a 3% chance the result is false."
✓ Correct interpretation
"p = 0.03 means if there were no real difference, we'd see results this extreme only 3% of the time."
Significance Thresholds
| P-Value | Interpretation |
|---|---|
| < 0.001 | Highly significant |
| < 0.01 | Very significant |
| < 0.05 | Significant (standard threshold) |
| ≥ 0.05 | Not significant |
Limitations
- Doesn't tell you the probability B is actually better
- Doesn't tell you the size of the effect
- Can be "hacked" by checking results repeatedly