A/B test significance calculator
Find out whether the gap between two versions is real or chance. Free, no signup.
Version A (control)
Version B (variant)
What is an A/B test significance calculator?
An A/B test shows two versions of a page, popup or offer to different visitors and compares how many of each group buy or sign up. A significance calculator tells you whether the gap between the two is large enough to trust, or small enough to be chance.
Small tests fool people. If version B gets 10 more orders than version A, it may just be luck, especially with a few hundred visitors. The calculator runs a chi-squared test on your numbers and gives a confidence level. At 95% or more, it names a winner.
The calculator runs in your browser. We don't store the numbers you enter, and you don't need an account.
How to use the A/B test calculator
Enter version A.
The visitors who saw your current version and how many of them converted.
Enter version B.
The same two numbers for the new version.
Click Check significance.
You get both conversion rates, the uplift of B over A, the confidence level and, at 95% or more, the winning version.
Use the same goal for both versions, such as orders or email signups, and the same dates.
How is A/B test significance calculated?
The calculator compares the conversions you got in each version with what you would expect if both versions converted at the same rate. The bigger the gap, the larger the chi-squared value and the smaller the p-value. Confidence is 1 minus the p-value.
| Version A | Version B | |
|---|---|---|
| Visitors | 5,000 | 5,000 |
| Conversions | 150 | 185 |
| Conversion rate | 3.00% | 3.70% |
Uplift is +23.33%, but confidence is 94.82%, just short of 95%, so the calculator reports "Promising, but not significant yet." With 10,000 visitors per version and rates of 3.00% and 3.60%, confidence reaches 98.25% and version B wins. A large uplift on a small sample can still be chance.
How to run an A/B test you can trust
Decide the sample size before you start.
Checking the result every day and stopping as soon as it crosses 95% raises the chance of a false winner. Evan Miller's guide How not to run an A/B test explains why.
Run full weeks.
Weekday and weekend visitors behave differently. A test that covers whole weeks compares like with like.
Change one thing.
If B has a new headline and a new offer, you won't know which one made the difference.
Split visitors at random, and keep each visitor on the same version for the whole test.
Check the result here before you keep the change.
What to A/B test in an online store
Popup headline
Two headlines for the same offer. See which one gets more signups.
Discount or free shipping
15% off against free shipping. Let your customers decide which offer works better.
Product page layout
Image order, where the price sits, or the color of the buy button.
Button text
"Buy now" against "Add to cart."
For ideas from real stores, see our email popup examples.
Test your popups with numbers
Run two popups with different offers, then check the result here before you keep one. PopupSnap's popups run on Salla and Zid stores. On another platform? Pick it and we'll show you what you can install today.
Frequently asked questions
It means the gap between your two versions is unlikely to be chance alone. At 95% confidence, if both versions really converted at the same rate, you would see a gap this large less than 5% of the time. It doesn't tell you the size of the real gain. The uplift you measured is only an estimate.
95% is the usual standard, and it is the level at which this calculator names a winner. Between 90% and 95%, it shows the result as promising: keep the test running rather than call it. If a wrong decision is cheap and easy to undo, some teams accept 90%. Pick the level before you start.
Because the gap is small compared with your sample. With 1,000 visitors per version, 3.0% against 3.7% gives only 61.56% confidence. The same rates with 5,000 visitors each reach 94.82%. More visitors make a real gap easier to tell apart from chance, so keep the test running until you reach the size you planned.
It depends on your conversion rate and how small a gain you want to detect: the smaller the gain, the more visitors you need. This calculator checks significance after the test; it doesn't work out sample size. Set your sample size before you start, with a sample size calculator, and don't stop early.
No. If you check every day and stop the first time it crosses 95%, false winners become much more likely. Decide on the number of visitors in advance, run whole weeks, and read the result once the test reaches that number. Evan Miller's article "How not to run an A/B test" explains the problem with examples.
A chi-squared test on the visitors and conversions of the two versions, with one degree of freedom. For two conversion rates this gives the same p-value as a two-sided two-proportion z-test. The calculator shows the chi-squared value and the p-value under the result so you can check the numbers.
No. The calculation runs in your browser, so your test results never leave your device. You don't need an account, and nothing is saved when you close the page. To keep a record, note the visitors, conversions and confidence for each test along with the dates it ran.