Credible Interval
A range where the true value has a specified probability of falling, typically 95%.
What is a Credible Interval?
A 95% credible interval is a range where there's a 95% probability the true lift falls. This is a direct probability statement about where the truth lies.
For example, a credible interval of [+5%, +25%] means there's a 95% chance the true lift is between 5% and 25%.
Credible vs Confidence Interval
Credible interval (Bayesian): "There's a 95% probability the true value is in this range."
Confidence interval (Frequentist): "If we repeated this experiment many times, 95% of intervals would contain the true value."
The credible interval is more intuitive — it directly tells you about the current experiment.
What Width Tells You
- Narrow interval: High certainty about the true lift
- Wide interval: More uncertainty — need more data
Decision Making
If the entire credible interval is positive (e.g., [+5%, +25%]), you can be confident B is better. If it spans zero (e.g., [-5%, +15%]), there's still uncertainty.