Frequentist A/B Testing

What is Frequentist A/B Testing?

Frequentist A/B testing is the traditional statistical approach used in scientific research. It answers the question: "If there were no real difference, how likely would we see results this extreme?"

This probability is called the p-value. If the p-value is below 0.05 (5%), the result is considered "statistically significant."

Key Metrics

• P-value: Probability of seeing these results by chance
• Confidence interval: Range where the true difference likely falls
• Statistical significance: Whether p-value is below the threshold (usually 0.05)

Important Caveats

• No peeking: Looking at results multiple times inflates false positives
• Pre-set sample size: You should decide how long to run before starting
• Binary outcome: Results are either "significant" or "not" — no nuance
• Common misinterpretation: P-value is NOT the probability the result is real

When to Use Frequentist

• Your organization requires traditional statistical methods
• You need results for academic publication
• You're comfortable with hypothesis testing concepts
• You can commit to a fixed sample size upfront