An A/B test shows two versions of a page to randomly split visitors and compares how many of each convert. It gives a trustworthy answer only when you decide the sample size in advance, change one meaningful thing, and wait until that sample is reached. For a page converting at 3%, detecting a 20% relative improvement needs roughly 13,000 visitors per version by a common rule of thumb, which is more than many small business pages receive in months. Below that level of traffic, usability tests, fixes for obvious problems and bigger changes measured over time will teach you more.
Key takeaways
- Work out the sample size before you start, from your current conversion rate and the smallest improvement worth detecting. Do not stop when the dashboard first shows a winner: Evan Miller shows that peeking can push the real false-positive rate to 26.1% when you thought it was 5%.
- Run for at least one to two full weeks even with enough traffic, as Nielsen Norman Group recommends, so weekdays and weekends are both represented.
- Check that the split is what you configured. Microsoft's experimentation team found a sample ratio mismatch in about 6% of its A/B tests and advises not to trust such results until the cause is found.
- With low traffic, test bigger differences, use qualitative research to choose them, or fix what is clearly broken without a test.
- Google Optimize ended on 30 September 2023; Google points to third-party tools such as AB Tasty, Optimizely and VWO that integrate with GA4.
What is an A/B test, and what can it tell you?
You show version A (the current page) to one randomly chosen half of visitors and version B (one change) to the other half, then compare a single pre-agreed metric such as form submissions per visitor. Because the only systematic difference between the groups is the change, a large enough difference in results can be attributed to it.
Nielsen Norman Group sums up the limit well: A/B testing is good at showing how behaviour changes but "will not provide any insights into why these changes occur". It also notes that only about one in seven A/B tests produces a winner, which is why the hypothesis matters more than the tool. The "why" comes from talking to customers, watching people use the page and reading what they ask your sales team.
How much traffic does an A/B test need?
Evan Miller, who publishes a sample size calculator, gives a rule of thumb for each version of the page:
n = 16 × σ² / δ², where δ is the smallest absolute change you want to detect and, for a conversion rate p, σ² = p × (1 − p).
Worked examples from that formula (our arithmetic, rounded):
| Current conversion rate | Improvement you want to detect | δ (absolute) | Visitors per version | Total visitors |
|---|---|---|---|---|
| 3% | 20% relative (3.0% to 3.6%) | 0.006 | about 12,900 | about 25,900 |
| 3% | 50% relative (3.0% to 4.5%) | 0.015 | about 2,100 | about 4,100 |
| 10% | 20% relative (10% to 12%) | 0.02 | about 3,600 | about 7,200 |
Divide the total by the weekly visitors to the page and you have the test length. A page with 1,000 visitors a week and a 3% conversion rate needs about half a year to detect a 20% lift, during which seasonality, campaigns and site changes will interfere. The same page could detect a 50% lift in about a month. That is the practical lesson for smaller sites: test changes big enough to move the number a lot, or do not run a formal test.
Use a proper calculator for real planning, and note that testing three or more versions at once needs more traffic again.
How do you design a test you can trust?
- Start from evidence. Pick the change from something you observed: form abandonment on a field, a question customers keep asking, a mismatch between ad and page. Our conversion review checklist is one way to find candidates.
- Write the hypothesis down: "Because we saw X, changing Y will increase Z." One primary metric, chosen before the test.
- Check that the metric is measured correctly. If the conversion event fires twice or not at all, the test will measure the bug. Our GA4 event plan covers the setup and how to verify it in DebugView.
- Fix the sample size and the end date in advance. Run for whole weeks; Nielsen Norman Group recommends at least one to two weeks even when traffic allows a shorter test.
- Do not peek and stop. Evan Miller's analysis shows that stopping as soon as a result looks significant inflates false positives: in his example, a 5% nominal threshold became a 26.1% real false-positive rate. If you need to look early, use a tool that supports sequential or Bayesian analysis designed for it.
- Check the split. If you configured 50/50 and got 53/47 on a large sample, something is dropping visitors from one version (a slow script, a redirect, a bot filter). Microsoft's researchers report sample ratio mismatches in about 6% of their tests and advise: "Don't trust the results of A/B tests with an SRM until you diagnose the root cause."
- Decide on business value, not only significance. A statistically significant 2% improvement on a page that produces ten enquiries a month may not be worth the maintenance of the variant.
Will an A/B test hurt your search rankings?
Not if you follow Google's testing guidance, last updated in December 2025:
- No cloaking. "Don't show one set of URLs to Googlebot, and a different set to humans."
- Canonical on variant URLs. If version B lives on its own URL, add
rel="canonical"pointing to the original; Google recommends this rather thannoindex. - Temporary redirects. If you redirect visitors to the variant, "use a 302 (temporary) redirect, not a 301 (permanent) redirect."
- End it promptly. Remove the test code once you have your answer. Google warns that a test left running unnecessarily long may be interpreted "as an attempt to deceive search engines".
Which tool should you use now that Google Optimize is gone?
Google ended Optimize and Optimize 360 on 30 September 2023. Its stated direction is third-party A/B testing tools that integrate with Google Analytics, naming AB Tasty, Optimizely and VWO, and its APIs are open to others. Several website platforms and landing-page builders also include basic split testing. When choosing, check:
- Whether variants are applied without a visible flash of the original page, which can itself change behaviour and layout stability.
- Whether it respects your consent setup: a test that only counts visitors who accept cookies measures a different population from your whole audience.
- Whether it sends the experiment and variant to GA4, so you can analyse results against your existing key events.
- What it adds to page weight, and the cost at your traffic level.
What should you do when traffic is too low to test?
Most small and mid-size business sites are in this position for most pages. Better options:
- Usability testing. Jakob Nielsen's long-standing finding is that a study with five users finds about 85% of usability problems in his model, with the caveat that different user groups need their own participants. Watching five customers try to request a quote on their own phones costs little and shows the "why".
- Fix what is clearly broken. A form that fails on one browser, a button below the fold on phones, a headline that does not match the ad. These do not need a test; they need doing. Our landing page checklist lists the usual suspects.
- Make bigger changes. A new offer, a different page structure or a shorter form can move conversion enough to see in your normal reporting.
- Compare periods carefully. A before-and-after comparison is weaker than a test because other things change too. Compare equal periods, note campaigns and seasonality, and treat the result as a direction rather than proof.
- Test where the traffic is. Paid campaigns often send more visitors to one page than your whole site gets organically; test there first.
Impinge Solutions plans and runs tests, usability checks and the measurement behind them as part of its conversion rate optimisation service. Run our free website audit to fix the clear problems before you buy a testing tool.
FAQ
How long should an A/B test run?
Until the sample size you calculated in advance is reached, and for at least one to two full weeks, as Nielsen Norman Group recommends, so that different days of the week are represented. Decide the length before starting from your traffic, current conversion rate and the smallest improvement worth detecting, and do not stop early because the result looks good.
How many visitors do I need for an A/B test?
It depends on your conversion rate and the size of the change you want to detect. Using Evan Miller's rule of thumb, a page converting at 3% needs about 12,900 visitors per version to detect a 20% relative improvement, but only about 2,100 per version for a 50% improvement. Use a sample size calculator with your own numbers.
Is it a problem if I check the results every day?
Looking is fine; stopping because of what you see is the problem. Evan Miller shows that repeatedly checking and stopping at the first significant result can turn a nominal 5% false-positive rate into 26.1%. Fix the sample size in advance, or use a tool built for sequential or Bayesian analysis.
Can A/B testing hurt my SEO?
Not if you follow Google's guidance: show Googlebot the same content as users (no cloaking), put a canonical tag on variant URLs pointing to the original, use 302 rather than 301 redirects for variants, and remove the test once it is finished.
What replaced Google Optimize?
Nothing from Google directly. Optimize ended on 30 September 2023, and Google now points to third-party A/B testing tools that integrate with Google Analytics, naming AB Tasty, Optimizely and VWO, with its APIs available to other tools.
Sources
- Evan Miller, "How Not To Run an A/B Test" (peeking; sample size rule of thumb): https://www.evanmiller.org/how-not-to-run-an-ab-test.html
- Nielsen Norman Group, Tim Neusesser, "A/B Testing 101" (30 August 2024): https://www.nngroup.com/articles/ab-testing/
- Microsoft Research, Fabijan et al., "Diagnosing Sample Ratio Mismatch in A/B Testing" (14 September 2020): https://www.microsoft.com/en-us/research/articles/diagnosing-sample-ratio-mismatch-in-a-b-testing/
- Google Search Central, "Minimize A/B testing impact in Google Search" (last updated 10 December 2025): https://developers.google.com/search/docs/crawling-indexing/website-testing
- Google Analytics Help, "Sunset of Google Optimize": https://support.google.com/analytics/answer/12979939
- Nielsen Norman Group, Jakob Nielsen, "Why You Only Need to Test with 5 Users" (18 March 2000): https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/