Glossary

A/B Testing Terms

Understand the key concepts behind A/B testing statistics. Learn the difference between Bayesian and Frequentist approaches, and what each metric means for your tests.

Statistical Methods

Bayesian A/B Testing

A statistical approach that calculates the probability one variant is better than another.

Frequentist A/B Testing

Traditional hypothesis testing using p-values and confidence intervals.

Bayesian Metrics

Probability to Win

The likelihood that a variant has a higher true conversion rate.

Expected Lift

The most likely improvement of one variant over another.

Credible Interval

A range where the true value has a specified probability of falling.

Expected Loss

The expected cost of choosing the wrong variant.

Frequentist Metrics

P-Value

The probability of seeing results as extreme by random chance.

Confidence Interval

A range likely to contain the true difference between variants.

Confidence Level

The threshold used to determine statistical significance.

Relative Lift

The percentage improvement of one variant over another.

Ready to run your first A/B test?

runab makes A/B testing simple. Just add one line of code.

Get started free