A/B Testing

A/B testing, also called split testing, randomly assigns visitors to a control (A) and a variant (B), then compares a pre-chosen metric such as signups or purchases. Statistical analysis indicates whether any difference is likely real or just chance.

Why it matters

Opinions about design and copy are often wrong. Experiments replace guesses with evidence and measure causal impact, not just correlation. They need enough traffic to detect meaningful differences.

Example

A (control): 4,000 visitors, 200 signups -> 5.0%
B (variant): 4,000 visitors, 240 signups -> 6.0%

Whether that 1-point gap is statistically significant depends on sample size and the test used; a significance calculator or statistical test answers it.

Common mistakes

• Stopping early the moment the result looks good ("peeking"), which inflates false positives.
• Changing many things at once, so you cannot tell what worked.
• Ignoring side effects, such as a variant that raises clicks but slows page load.

Related

• Data analytics
• Data science
• Dashboard
• Latency

This page is general information written by our editorial team and updated when documentation changes. Servers and providers differ, so test changes safely first. See our editorial policy and disclaimer.