Back to Glossary
Glossary
Confidence Interval
A range of values that likely contains the true difference between variants.
What is a Confidence Interval?
A 95% confidence interval is a range calculated from your data. If you repeated the experiment many times, 95% of these intervals would contain the true difference.
How to Interpret It
- Interval doesn't cross zero: Statistically significant difference
- Interval crosses zero: Not significant — could go either way
- Narrow interval: More precise estimate
- Wide interval: Less certain — need more data
Example
A 95% CI of [+2%, +8%] for the difference means:
- The observed improvement is likely between 2% and 8%
- Since it doesn't include 0, the result is significant
- B is likely better than A
Confidence vs Credible Interval
A confidence interval (frequentist) is about the procedure — 95% of intervals from repeated experiments would contain the truth.
A credible interval (Bayesian) is about this specific experiment — there's a 95% probability the truth is in this range.