To calculate statistical significance for an A/B test, divide the difference between the two conversion rates by its standard error to get a z score, convert that z score into a p-value, and compare the p-value with the significance level you set before launch. If the p-value is below that level, usually 0.05, the result is statistically significant.
The calculation is only valid when you fixed the sample size in advance and read the result once. This guide is part of the A/B testing hub and covers the formula, a hypothetical worked example, sample size, peeking and tool readouts.
What do you need to calculate statistical significance for an A/B test?
To calculate statistical significance for a conversion rate test you need four numbers: visitors and conversions in the control, and visitors and conversions in the variant. You also need two decisions made before launch: the significance level you will accept, usually 0.05, and the sample size per variant you will wait for.
Statistical significance is the conclusion that an observed difference would be unlikely if the change had no real effect. The no-effect scenario is called the null hypothesis. The significance level (alpha) is the false positive rate you accept: at 0.05, about one in 20 tests of a change that does nothing will still look like a winner.
The p-value is the probability of seeing a difference at least as large as the one you observed if the null hypothesis were true. The American Statistical Association statement of March 7, 2016 stresses what it is not: "P-values do not measure the probability that the studied hypothesis is true," and "a p-value, or statistical significance, does not measure the size of an effect or the importance of a result."
For conversion rates, the standard method is the two-proportion z test, which compares two shares of "successes" from independent groups. The NIST/SEMATECH e-Handbook uses it for reasonably large samples and recommends Fisher's exact test for small ones.
How do you calculate statistical significance step by step?
You calculate statistical significance step by step by computing each conversion rate, pooling them into one combined rate, using that pooled rate to find the standard error of the difference, dividing the difference by the standard error to get a z score, and converting the z score into a two-sided p-value that you compare with alpha.
The formula below follows the pooled two-proportion test in Penn State's STAT 415 lesson, where the pooled estimate is the share of successes in both samples combined. The standard error is the spread you would expect in the difference from chance alone, and a z score is the number of standard errors that separate the two rates.
- Compute both conversion rates. p1 = control conversions / control visitors, and p2 = variant conversions / variant visitors.
- Compute the pooled rate. p = (control conversions + variant conversions) / (control visitors + variant visitors). This is the best estimate of the rate if the variant changed nothing.
- Compute the standard error. SE = √(p × (1 - p) × (1/n1 + 1/n2)), where n1 and n2 are the visitors per arm.
- Compute the z score. z = (p2 - p1) / SE.
- Convert z to a two-sided p-value. In Google Sheets or Excel use
=2*(1-NORM.S.DIST(ABS(z),TRUE)). A z score beyond 1.96 in either direction means p is below 0.05. - Compare and report. If p is below your alpha, call the result significant, and report the confidence interval of the difference next to it.
Use a two-sided test unless you decided before launch that you only care about one direction. Optimizely, for example, says it uses "two-tailed tests because they are required for the false discovery rate control in Stats Engine".
Is a 15% lift statistically significant? A worked example
A 15% relative lift is not automatically statistically significant. In the hypothetical example below, a variant converting at 3.45% against a 3.0% control gives a p-value of about 0.07 with 10,000 visitors per arm, which misses the 0.05 threshold, and about 0.005 with 24,000 visitors per arm, which passes it.
The numbers are a hypothetical illustration, not data from a real test. The control gets 10,000 visitors and 300 conversions (3.0%), and the variant gets 10,000 visitors and 345 conversions (3.45%).
- Pooled rate: 645 / 20,000 = 3.225%.
- Standard error: √(0.03225 × 0.96775 × (1/10,000 + 1/10,000)) = 0.0025, or 0.25 percentage points.
- z score: (0.0345 - 0.0300) / 0.0025 = 1.80.
- Two-sided p-value: about 0.072, so the result is not significant at 0.05.
- 95% confidence interval of the difference (the range of true differences consistent with the data): about -0.04 to +0.94 percentage points, which includes zero.
The same rates at other sample sizes show how much the verdict depends on volume.
| Visitors per arm | Conversions (control vs variant) | z score | p-value | Significant at 0.05? |
|---|---|---|---|---|
| 2,000 | 60 vs 69 | 0.81 | 0.42 | No |
| 10,000 | 300 vs 345 | 1.80 | 0.072 | No |
| 24,000 | 720 vs 828 | 2.79 | 0.005 | Yes |
A result like the 10,000 visitor case is "not significant", not "no effect": the true difference could be near zero or near a full percentage point, so the test was too small to tell.
How does sample size affect statistical significance?
Sample size affects statistical significance through the standard error, which shrinks as visitors grow. The same difference in conversion rates becomes more significant with more data, so small tests can only detect large lifts, while detecting a small lift on a low baseline rate takes tens of thousands of visitors per variant.
The standard error falls with the square root of the sample size. Quadrupling the visitors per arm halves the standard error and doubles the z score for the same observed rates, which is why a hypothetical 3.0% vs 3.45% test misses by a wide margin at 2,000 visitors per arm (p ≈ 0.42) and passes at 24,000 (p ≈ 0.005).
Evan Miller gives a rule of thumb for sample size: n = 16σ²/δ² per arm, "where δ is the minimum effect you wish to detect and σ² is the sample variance you expect." For a conversion rate, σ² = p × (1 - p). The factor 16 roughly matches 80% statistical power at a 5% two-sided significance level, because 2 × (1.96 + 0.84)² is about 15.7.
For a hypothetical 3.0% baseline and a 0.45 point minimum lift, that gives 16 × 0.03 × 0.97 / 0.0045² ≈ 23,000 visitors per arm. Divide that by your daily traffic per arm to get the run time, round up to full weeks, and check it against Evan Miller's sample size calculator, where you can set power and significance yourself.
Why does checking results early break the significance calculation?
Checking results early breaks the significance calculation because the standard z test assumes you look once, at a sample size fixed in advance. Every extra look is another chance for random noise to cross the threshold, so stopping at the first significant reading produces far more false winners than the 5% the p-value promises.
Peeking is the habit of checking a running fixed-horizon test and stopping as soon as it shows significance. Evan Miller's How Not To Run an A/B Test (2010) shows that when a team checks repeatedly and stops at the first significant result, the real false positive rate for a change that does nothing reaches 26.1%, "more than five times what you probably thought the significance level was."
His fix: "Decide on a sample size in advance and wait until the experiment is over." For teams that must look during a test, he points to sequential designs, which "set up checkpoints in advance," and to Bayesian designs.
Some vendors are more careful about the Bayesian route. GrowthBook's statistics docs say Bayesian results "can suffer from the same 'peeking' problems as frequentist statistics," and that GrowthBook offers sequential testing in its frequentist engine "to mitigate concerns with peeking." Peeking is safe only when the method was built for it.
How do you read the significance output of A/B testing tools?
You read the significance output of an A/B testing tool by first identifying its statistical method. A fixed-horizon p-value is valid only at the planned sample size, a sequential readout is built to be checked as data arrives, and a Bayesian chance to win is a probability that a variant is better, not a p-value.
Most platforms do not run the plain z test. Sequential testing is a method whose error rates stay valid while results are checked repeatedly. The table compares four common tools, based on their own documentation as of October 2026. For price and features, see the guide to conversion rate optimization tools.
| Tool | Method | What the readout means | Safe to check during the test? |
|---|---|---|---|
| Optimizely (Stats Engine) | Sequential testing with false discovery rate control | Significance that should generally rise as evidence builds | Built for it, per Optimizely |
| GrowthBook | Bayesian by default; frequentist with optional sequential testing | "Chance to Win": probability a variation is better | Only with sequential testing enabled |
| Wingify (SmartStats) | Bayesian | Probability of A beating B | Not stated in the SmartStats help article |
| Google Ads experiments | Confidence intervals on metric differences | Blue asterisk when significant; 80% interval by default | Google recommends 4 to 6 weeks first |
Sources for the table: Optimizely says its "Stats Engine uses sequential experimentation, not the fixed-horizon experiments you see on other platforms" (Optimizely docs) and applies "a tiered version of the Benjamini-Hochberg procedure" (false discovery rate control). GrowthBook says teams "typically want to wait until this reaches 95%" (GrowthBook). Wingify calls SmartStats its "Bayesian-powered statistics engine" (Wingify Help). Google Ads notes "80% is the default confidence interval" (Google Ads Help), which is a looser bar than 95%, and its Experiments FAQs recommend running at least 4 to 6 weeks. The Performance Max asset testing guide covers that scorecard in detail.
When can you call an A/B test a winner?
You can call an A/B test a winner when four conditions hold: the test reached the sample size planned before launch, it ran for whole weeks, the p-value is below the alpha chosen up front, and the confidence interval excludes zero with a lower bound still worth shipping. If one is missing, the test is not finished.
The decision table is a rule of thumb for fixed-horizon tests analyzed with the z test.
| Planned sample reached? | p-value vs alpha | Confidence interval of the difference | Call |
|---|---|---|---|
| No | Any | Any | Keep running; do not stop on a significant reading |
| Yes | Below alpha | Lower bound above zero and worth the cost of shipping | Ship the variant |
| Yes | Below alpha | Lower bound above zero but too small to matter | Significant but not worth shipping; keep the control |
| Yes | Above alpha | Wide and includes zero | Inconclusive; the test was too small for this effect |
| Yes | Above alpha | Narrow and centered near zero | No meaningful effect; keep the control and move on |
The rule separates two questions the p-value mixes up: is there an effect, and is it large enough to act on. The conversion rate optimization process shows where tests fit next to research and analytics. Ad platforms use the same logic at campaign level: Google Ads Conversion Lift compares an exposed group with a holdout group.
Common mistakes when calculating statistical significance
The common mistakes when calculating statistical significance come from breaking the assumptions of the test rather than from bad arithmetic. Stopping early, testing many metrics, mixing units and treating p-values as effect sizes all produce confident numbers that do not mean what they appear to mean.
- Stopping when the dashboard turns green. A fixed-horizon p-value is valid once, at the planned sample size. Fix the sample and the end date before launch, or use a tool with sequential statistics.
- Changing alpha after seeing the result. Moving from 0.05 to 0.10 because the test landed at 0.07 turns the threshold into a rationalization. Pick the level in the test brief.
- Testing many metrics and variants without correction. Every extra comparison adds a chance of a false winner. Name one primary metric, and use false discovery rate control or a stricter alpha for several variants.
- Using sessions when users are randomized. The z test assumes independent observations. Count visitors and conversions in the same unit you randomized on.
- Reading the p-value as the chance the variant is better. The ASA statement is explicit that it is not. Report the confidence interval to show the likely size of the effect.
- Ignoring small samples. With only a handful of conversions per arm, the normal approximation is weak. The NIST handbook names Fisher's exact test as "an excellent choice for small samples."
