What is A/B testing, and when is it worth doing?

A/B testing means running two versions of the same web page (version A, the one you have now, and version B, with one change) side by side with real visitors, split at random, and measuring which one brings in more orders or enquiries. It is worth doing when your site has enough visitors and enough conversions for the result to be trustworthy. When it doesn't, your money is better spent fixing the obvious problems first.

You don't need to run tests yourself. But it pays to understand them well enough to tell when an A/B testing proposal is worth the money, and when it is just a service package that sounds sophisticated.

What A/B testing is, in plain English

Imagine you're printing flyers for a coffee shop and can't decide between two offers: "20% off your first coffee" or "Buy one, get a pastry free". Instead of guessing, you print both, hand them out alternately to passers-by during the same week, and count how many people each flyer brings through the door.

A/B testing on a website does exactly that, except software does the work:

• Visitors are split at random: half see version A, half see version B. Nobody chooses, so the two groups are alike in every way that matters.
• Both versions run at the same time, so things like holidays, the weather or an ad campaign affect both sides equally.
• One main goal is measured: for example clicks on "Book now", completed orders, or contact forms sent.

The change being tested can be a headline, an image, how a price is shown, the number of fields in a form, where a button sits, or a whole new page layout. The core idea: decisions are made by what real customers do, not by whoever argues loudest in the meeting.

Why it pays

The biggest reason sounds backwards: most ideas that "will definitely work" don't.

• In a Harvard Business Review article, Ron Kohavi (who led experimentation at Microsoft) and Professor Stefan Thomke (Harvard Business School) report that at Google and Bing, only about 10% to 20% of experiments generate positive results. Across Microsoft as a whole, about a third of ideas prove effective, a third are neutral, and a third make results worse.
• The same article tells the story of a small change to how Bing displayed ad headlines, left at "low priority" for more than six months. When it was finally tested, it increased revenue by 12%, which came to more than $100 million a year in the United States alone.

Both numbers say the same thing: if you change your website on gut feeling, roughly one change in three may be quietly costing you customers, and you would never know. A/B testing lets you keep the winners and throw out the losers before the losers get expensive.

The second benefit is compounding. A single winning test rarely makes revenue jump. Many small improvements stacked over months do. That is what I saw first-hand running the experimentation program for Neighborly: more than 150 A/B tests, a 35% higher booking rate, and $2.8M in added revenue. That came from a long, disciplined program, not from one lucky test.

When A/B testing isn't worth it yet

This is the part that sellers of these services rarely spell out: A/B testing needs far more traffic than most people expect.

What does "statistically significant" mean?

Flip a coin 10 times and get 6 heads, and you wouldn't conclude the coin is biased. That could easily be chance. Flip it 10,000 times and get 6,000 heads, and you can be fairly sure it really is biased.

Websites work the same way. Version B gets 12 orders and version A gets 9: that gap may well be luck. "Statistically significant" simply means the difference is big enough, with enough data behind it, that luck is an unlikely explanation. Getting there takes enough visitors, and above all enough conversions.

How many visitors do you need?

A concrete example, worked out with the standard method used by tools such as Evan Miller's sample size calculator (95% confidence and 80% power, the usual industry settings):

• Your page currently converts 2 visitors in every 100.
• You want to detect an improvement of 20% (from 2% to 2.4%).
• You need roughly 21,000 visitors per version, so about 42,000 visitors to that one page.

If the page gets 10,000 visits a month, one test takes around four months. To detect a smaller improvement, say 10%, the number grows almost fourfold. For many small and mid-sized business sites, the arithmetic gives a clear answer straight away: not yet.

Stopping a test early is a trap

A very common mistake: checking the results every day and stopping as soon as version B looks like it's "winning". Evan Miller's analysis shows that this "keep looking and stop when it wins" habit can push the rate of false conclusions from the intended 5% to over 26% in the worst case. You think you've found a winner; you've actually just found noise.

Signs you should do something else first

• The page you want to test gets fewer than a few hundred orders or enquiries a month.
• You aren't sure your tracking is right. If GA4 is counting wrong, every test is meaningless. See common GA4 and GTM setup mistakes.
• The site still has obvious problems: it's slow, breaks on phones, or the form doesn't send. Just fix those; they don't need a test.

A low-traffic site can still improve its conversion rate, only with different tools: watching session recordings, talking to customers, and applying proven principles. The article on A/B testing for low-traffic sites goes deeper into those options.

How to judge an A/B testing proposal

A serious proposal includes the following. If anything is missing, ask:

• A hypothesis with a reason: not "let's try a different button colour", but "visitors drop off at step 2 because the form is too long; cutting it to 3 fields should increase submissions".
• One main metric tied to revenue: orders, bookings, enquiries. Not random clicks.
• Sample size and duration worked out before the test starts, and kept, with no stopping halfway because "it looks like it's winning".
• Tracking checked before launch: both versions are recorded correctly.
• Full weeks, so both weekday and weekend behaviour are included.
• Losing tests reported too: an honest program has plenty of tests that don't win. If every test "wins", be suspicious.
• The tool and its cost spelled out: the most popular free option, Google Optimize, shut down on September 30, 2023, so most options today are paid tools.

Questions to ask your web team

• "How many visitors and conversions does the page we want to test get each month? At that level, how long would one test need to run?"
• "What is the smallest improvement this test can reliably detect?"
• "Will you work out the sample size up front and commit to not stopping the test early?"
• "How will we know the tracking is accurate before the test begins?"
• "If our site doesn't have enough traffic to test, what would you do instead?"

The last question matters most. A team worth hiring will tell you "don't test yet" when the numbers don't support it, rather than selling you a testing package on too little traffic.

Next step

In short: A/B testing lets real customers decide which version sells better, and it pays off when your site has enough traffic and your tracking is accurate. If it doesn't yet, the right move is to fix measurement and the obvious problems first.

If you want to know whether your site is ready to test, or you need a properly run experimentation program, see the A/B testing & CRO service, or get in touch and I'll look at your numbers with you.

---

Sources:

• The Surprising Power of Online Experiments — Ron Kohavi & Stefan Thomke, Harvard Business Review (2017)
• Sample Size Calculator — Evan Miller
• How Not To Run an A/B Test — Evan Miller
• Google Optimize will sunset this year — Search Engine Land