Bayesian A/B Testing
A statistical approach that calculates the probability one variant is better than another, allowing you to make decisions at any time.
What is Bayesian A/B Testing?
Bayesian A/B testing is a statistical method that directly answers the question: "What is the probability that Variant B is better than Variant A?"
Unlike frequentist methods that give you a p-value (which is often misinterpreted), Bayesian analysis gives you an intuitive probability you can act on immediately.
Key Benefits
- Intuitive results: "97% chance B is better" is easier to understand than "p = 0.03"
- Check anytime: You can look at results whenever you want without inflating false positives
- Decision-focused: Directly tells you which variant to pick and with what confidence
- Risk quantification: Shows expected loss if you make the wrong choice
How It Works
Bayesian analysis uses a Beta-Binomial model to estimate the true conversion rate of each variant. As more data comes in, the estimate becomes more precise.
The key output is P(B > A) — the probability that Variant B has a higher true conversion rate than Variant A. When this probability exceeds 95%, you can confidently pick B.
When to Use Bayesian
- You want to monitor results continuously
- You need actionable recommendations (not just "significant" or "not")
- You want to understand the risk of making a wrong decision
- You prefer intuitive probability statements
runab Default
runab uses Bayesian analysis by default because it provides clearer, more actionable results for most users.