Control (A)
Variant (B)
95% is the industry default (5% false-positive risk).
Method: two-proportion two-tailed Z-test — the standard for comparing two conversion rates. The p-value comes from the normal distribution, computed in your browser.
The verdict, in plain EnglishSignificance is a gate, not proof a change is worth shipping.
Enter visitors and conversions for both A and B to see whether the difference between them is real or just random variation.
Statistical ≠ practical significance. A significant result means the difference is real; it does not mean it is big enough to be worth shipping, or that the test ran long enough to be trusted. Always check the relative uplift and your conversion count before you act on a winner.
All calculations run in your browser — nothing you type is sent anywhere or stored. Results are statistical estimates; judge a significant winner against its real-world uplift before you ship it.
Run a quick A/B test significance check, then plan your next test on the Sample Size tab. Enter the visitors and conversions for your control and your variant, pick a confidence level, and the calculator returns each conversion rate, the uplift, the z-score, the p-value and a plain-English verdict on whether your test has a real winner. It is built for a simple budget question: did your $500 experiment actually win, or was it just luck? Free, no sign-up, runs entirely in your browser.
Is My A/B Test Significant?
Your A/B test is statistically significant when the difference between your two conversion rates is larger than random chance can comfortably explain. In practice that means the p-value the calculator returns is below your threshold — below 0.05 at a 95% confidence level, the industry standard. When that happens, the verdict reads Significant and tells you which variant won. When the p-value is above the threshold, the result is Not significant yet: there may be a difference in the numbers, but you do not yet have enough evidence to trust it, and you should keep the test running. If both variants convert at exactly the same rate, the result is Inconclusive — there is simply nothing to detect. The one trap to avoid: a significant result tells you a difference is real, not that it is big enough to be worth shipping. Always read the relative uplift alongside the verdict.
How Statistical Significance Is Calculated
Behind the verdict is a single, well-established piece of statistics. Here is what each part means, in plain language.
The Two-Proportion Z-Test
This calculator uses a two-proportion z-test, the standard method for comparing two conversion rate figures. It works in four steps. First it finds each conversion rate — conversions divided by visitors for the control and for the variant. Then it pools both groups into one combined rate and uses that to estimate the standard error, which is how much random wobble to expect when the two groups are really the same. The z-score is the gap between the two conversion rates divided by that standard error, so it measures how many standard errors apart your variants sit. Finally the z-score is turned into a p-value. We run a two-tailed test by default because, before a test starts, you do not know whether B will beat A or lose to it — a two-tailed test honestly accounts for movement in either direction, where a one-tailed test quietly inflates your apparent significance. The starting point of any test is a hypothesis that the change will move the needle; the z-test is how you decide whether the data backs it up.
What Is a P-Value?
The p-value answers one precise question: if the two variants were truly identical, how likely is it that you would see a gap at least this large purely by chance? A p-value of 0.05 means a one-in-twenty chance — uncommon enough that we stop calling it noise. A p-value of 0.30 means there is a 30% chance the gap is random, which is no evidence at all. The smaller the p-value, the stronger the case that the difference is real. It is easy to misread, so be clear on what it is not: the p-value is not the probability that B is better, and it is not the chance your result is wrong. It is only the probability of your data assuming there is no real difference — a gate that keeps you from acting on a false positive.
What Does 95% Confidence Level Mean?
A 95% confidence level means you are willing to accept a 5% chance of a false positive — declaring a winner when there is really no difference. That is the default this calculator ships with, and the one most marketing tests should use. Drop to 90% for low-stakes, easily reversed changes where reaching a verdict quickly on smaller traffic matters more than certainty. Raise it to 99% for high-stakes, hard-to-undo changes — a pricing or checkout change — where a 1% false-positive risk is worth the much larger sample it demands. Whatever you choose, choose it before the test runs. Lowering your confidence level after seeing the data, just to push a borderline result over the line, defeats the entire purpose of the threshold — and is the single most common way an experiment fools the person running it.
Sample Size — How Many Visitors Do You Need?
The honest answer to “how many visitors do I need” is: it depends, and you should work it out before you launch. Starting a test with no idea how long it must run is how underpowered tests produce confident-looking but misleading results. The Sample Size tab on the calculator above turns four inputs into the exact number of visitors each variant needs.
What Is Minimum Detectable Effect (MDE)?
The minimum detectable effect is the smallest improvement you want the test to be able to spot, expressed as a relative lift. An MDE of 10% on a 5% baseline conversion rate means you want to reliably detect a move to 5.5%. MDE is the biggest lever on sample size: the smaller the effect you insist on catching, the more visitors you need, because tiny differences are hard to separate from random noise. A 10% MDE is a sensible default for most small-business tests — chasing a 2% lift can require a sample so large that the test would run for a year. Pick the smallest lift that would actually change a decision, not the smallest lift imaginable.
What Is Statistical Power?
Statistical power is the flip side of confidence. Where confidence guards against false positives, power guards against false negatives — missing a real effect that is genuinely there. A power of 80%, the industry default, means that if the improvement you set as your MDE truly exists, the test has an 80% chance of detecting it. Raising power to 90% makes the test more sensitive but, like a tighter confidence level, requires a larger sample. For most tests, the standard pairing — 95% confidence and 80% power — is the right balance of rigour and practicality.
How to Use the Sample Size Calculator
- Enter your baseline conversion rate. Your current rate for the page or flow you are testing — say 5%.
- Set your minimum detectable effect. The smallest relative lift worth detecting — say 10%.
- Leave confidence and power at their defaults (95% and 80%) unless you have a specific reason to change them.
- Add your daily visitors (optional). If you enter how many visitors land across both variants per day, the calculator estimates how many days the test must run to reach a valid sample. This turns an abstract sample-size number into a practical answer for a site doing 50 to 500 visitors a day.
How to Read Your Results
The verdict panel is built to stop you acting on the wrong signal. Here is what to do with each outcome.
Result: Significant — What to Do Next
A significant result means the difference is real — but “real” and “worth shipping” are not the same thing, and confusing them is expensive. Statistical significance tells you the effect exists; practical significance asks whether the effect is large enough to matter. A 0.3% relative uplift can be statistically significant on a huge sample and still be too small to justify the engineering or the risk of the change. So before you ship a winner, read the relative uplift next to the verdict and ask whether that lift, applied to your real traffic, moves a number you care about. A win on conversion rate also does not automatically mean a win on profit. If a test lifts sign-ups but those visitors cost more to acquire, the gain can evaporate — run the marketing ROI formula before you scale spend, and estimate the campaign cost of more traffic with our advertising metrics calculator. When a winning variant becomes your new default, track it as part of your key metrics for small businesses rather than admiring it once and forgetting it.
Result: Not Significant Yet — What to Do Next
Not significant does not mean B failed; it usually means you do not have enough data yet. The right response is almost always patience: keep the test running until you hit the sample size the Sample Size tab calculated, then judge it. Resist the urge to peek at the result every day and stop the moment it looks good — that is how false positives slip through. If you have already reached your planned sample and the result is still not significant, that is a genuine answer: the change you tested probably is not moving conversions, and you should try a bolder variation rather than a slightly different shade of button. Before you commit a bigger budget to driving traffic to a test, it is also worth confirming your tracking is trustworthy. If you have not set one up yet, our guide to free A/B testing tools covers how to run experiments without paying for an enterprise platform.
Common Mistakes That Break A/B Tests
Most A/B tests fail not because the maths is wrong but because of how they are run. These are the mistakes that quietly invalidate a result:
- Peeking and stopping early. Checking the result repeatedly and stopping the instant it crosses significance dramatically inflates your false-positive rate. A test that “wins” on day two often regresses to no difference by the time it reaches its planned sample. Decide your sample size up front and wait for it.
- Too few conversions per variant. A test with a handful of conversions is underpowered: the verdict can flip with a few more sign-ups in either direction. As a rule of thumb, treat anything under roughly 100 conversions per variant as preliminary — the calculator flags this for you.
- Using a one-tailed test to look more confident. A one-tailed test reports significance sooner, but only because it ignores the possibility that your change made things worse. Stick with the two-tailed default unless you have a strict, pre-registered reason not to.
- Changing the confidence level after seeing the data. Picking 90% because 95% did not quite cross the line is moving the goalposts. Set your threshold before the test, then live with it.
- Calling a tiny lift a win. A statistically significant 0.2% lift is real but rarely worth the cost of shipping. Read the relative uplift, not just the verdict.
Frequently Asked Questions
What is statistical significance in an A/B test?
Statistical significance is the test that tells you whether the difference between your two variations is a real effect or just random luck. In any A/B test the conversion rate of A and B will almost never be identical — even two copies of the same page will differ slightly because visitors arrive randomly. Significance asks a sharper question: if there were truly no difference between A and B, how likely is it that you would see a gap this large purely by chance? That likelihood is the p-value. When the p-value drops below your threshold — usually 5%, which is the same as a 95% confidence level — the result is called statistically significant, meaning the difference is unlikely to be noise. It does not prove B is better by a useful amount, and it does not mean the test is finished if you have only a handful of conversions. It simply means the gap you measured is bigger than random variation can comfortably explain.
How do you calculate A/B test significance?
This calculator uses a two-proportion two-tailed Z-test, the standard method for comparing two conversion rates. First it works out each conversion rate: conversions divided by visitors for A and for B. Then it pools the two groups into a single combined rate and uses that to estimate how much random variation to expect — the standard error. The z-score is the gap between the two conversion rates divided by that standard error, so it measures how many standard errors apart your variants are. Finally the z-score is converted into a two-tailed p-value using the normal distribution. A two-tailed test is used because before a test runs you do not know whether B will beat A or lose to it, and a two-tailed test honestly accounts for both directions. The result is significant when the p-value falls below one minus your chosen confidence level — below 0.05 for 95% confidence. You never need to do this by hand: enter four numbers and the calculator returns the z-score, the p-value and a verdict.
What confidence level should I use for an A/B test?
For most marketing and conversion tests, 95% is the right confidence level and the industry standard. A 95% confidence level means you accept a 5% chance of a false positive — declaring a winner when there is actually no real difference. That balance keeps you from chasing noise without demanding an impractically large sample. Use 90% confidence when the decision is low-stakes and easily reversible, such as a minor copy tweak where acting on a slightly weaker signal costs you almost nothing; you accept a 10% false-positive risk in exchange for reaching a verdict faster on smaller traffic. Reserve 99% confidence for high-stakes, hard-to-undo changes — a pricing change, a checkout redesign, anything that would be expensive to roll back — where a 1% false-positive risk is worth the much larger sample it requires. Whatever level you pick, choose it before the test starts, not after you have seen the data, or you defeat the purpose of the threshold.
How many visitors do I need for a valid A/B test?
There is no universal number — it depends entirely on your current conversion rate and the size of the improvement you want to detect. The lower your baseline conversion rate and the smaller the lift you are hunting for, the more visitors each variant needs, because subtle differences are harder to separate from random noise. As a rough orientation, detecting a 10% relative lift on a 5% baseline conversion rate at 95% confidence and 80% power needs on the order of tens of thousands of visitors per variant; a bigger expected lift or a higher baseline brings that down sharply. Rather than guess, switch to the Sample Size tab on this page, enter your baseline conversion rate, the minimum detectable effect you care about, and your confidence and power, and it returns the exact visitors required per variant and in total. Plan that number before you launch — starting a test without knowing how long it must run is the most common way underpowered tests produce misleading results.
What does a p-value of 0.05 mean in an A/B test?
A p-value of 0.05 means there is a 5% probability of seeing a difference at least as large as the one you measured if A and B were actually identical. Put plainly: if nothing real were going on, a gap this big would happen by chance roughly one time in twenty. That is exactly the threshold a 95% confidence level draws, which is why p < 0.05 is the conventional line for calling a result statistically significant. A common misreading is to think the p-value is the probability that B is better, or the probability the result is wrong — it is neither. It is the probability of your data under the assumption of no difference, nothing more. A smaller p-value means stronger evidence against the idea that the two variants perform the same; a p-value of 0.01 is stronger evidence than 0.05, and 0.5 is essentially no evidence at all. Significance is a gate, not a measure of how big or how valuable the effect is.
Is my A/B test result significant if p = 0.06?
At a 95% confidence level, no — a p-value of 0.06 is just above the 0.05 threshold, so the result is not statistically significant and you should not declare a winner yet. But 0.06 is genuinely a borderline case, and it is worth understanding rather than dismissing. The same result would be significant at a 90% confidence level, where the threshold is 0.10, so if the change is low-stakes and easily reversible, 90% confidence may be a reasonable bar. More often, a p-value of 0.06 simply means you are close and need a little more data: the trend is promising but your sample is not yet large enough to rule out chance. The right move is usually to keep the test running until you reach the sample size your Sample Size tab calculated, rather than stopping the moment the number looks tempting. What you must not do is quietly lower your confidence level after seeing the data purely to cross the line — that is the classic way a test fools you.