A/B test significance calculator

How this calculator works

It runs a two-proportion z-test: the conversion rates of both variants are compared against the hypothesis that they're really the same and the observed gap is luck. The p-value is the probability of seeing a gap at least this large if the variants truly performed identically. By convention, p < 0.05 (95% confidence) counts as significant — that's a norm, not a law of nature.

Honest caveats

• Don't peek and stop early. Checking daily and stopping the moment the calculator turns green inflates false positives. Decide your sample size up front, then evaluate once.
• One primary metric. Testing ten metrics at once means one will look significant by chance.
• Small samples lie. Below a few hundred conversions per variant, big-looking lifts are usually noise.
• This is a fixed-horizon test on independent visitors; if your setup differs (sequential testing, session-level metrics), the p-value is approximate.

If you run experiments on your site, Tracely's experiments dashboard tracks variant exposure and conversions from the same first-party events as the rest of your analytics — no separate testing pixel.