Your Site Is Too Small for A/B Testing: The 40,000-Visitor Math Nobody Puts on the Pricing Page

The most repeated advice in conversion optimization is: "Don't guess. A/B test it."

It sounds rigorous. Two versions, split the traffic, see which one wins. The tools are a script tag away. Case studies about "we changed the button color and revenue jumped" are everywhere.

Here is what the people selling testing tools rarely say out loud: for most small business websites, a classic A/B test is a coin toss with a subscription fee. Not because the method is wrong. The method is sound. The problem is that you don't have enough visitors for it to give you an answer you can trust.

And an untrustworthy answer is worse than no answer. With no result, you know you don't know. With a fake "win", you confidently ship a change that does nothing, then tell the whole team you "optimized with data".

The math below shows why.

The math the testing tools don't put on their homepage

Take a very ordinary scenario:

• A product or service page converts at 2% (2 out of every 100 visitors buy or submit the form).
• You want to know whether a new version lifts that rate by 20%, from 2% to 2.4%. A 20% relative lift is already a meaningful improvement. Most small tweaks don't get there.
• You use the industry-standard settings: 5% significance level, 80% statistical power.

Put those numbers into Evan Miller's sample size calculator, a long-standing reference among experimentation practitioners, and the answer is about 19,800 visitors per variation.

Two variations means almost 40,000 visitors to the page you're testing. Not 40,000 to the whole site. 40,000 to that one page.

Now put that next to the page's real traffic:

Computed with the formula behind Evan Miller's calculator: 2% baseline, 50/50 split, 5% significance, 80% power.

If you want to catch a 10% lift, which is what most "tweak" tests are really chasing, you need about 78,000 visitors per variation, or more than 150,000 for a single test.

Be honest about your own numbers: how many monthly visitors does your most important service page get? For a lot of small business sites, it's a few hundred to a few thousand. At that level, a properly powered A/B test runs for more than a year. In a year, seasons change, ad campaigns change, competitors change. The final result measures all of that, not just your variation.

How a fake "win" is born

Here is the scenario I see most often. A business owner runs a test for two weeks. Each variation gets 150 visitors. The original gets 3 orders, the new version gets 6.

The dashboard says: the new version is up 100%. Anyone would want to ship it.

But run the standard significance test on those numbers (a two-proportion z-test, which is what most tools use under the hood) and the p-value is about 0.31. In plain terms: if the two versions performed exactly the same, you'd still see a gap that large about 31% of the time from chance alone. A three-order gap can come from one repeat customer, one day with a promo running, or the owner's friend placing an order.

There's a subtler trap too: checking the results every day and stopping when you see a winner. Evan Miller showed in his classic essay How Not To Run an A/B Test that repeated significance testing always increases the false-positive rate. In his example, for a change that does nothing at all, stopping the first time the test looks significant pushes the false-win rate to 26.1% instead of the 5% you think you're getting. His advice is blunt: decide on a sample size in advance and wait until the experiment is over.

Add one more uncomfortable fact. Even at the best experimentation organizations in the world, most ideas don't win. Microsoft has reported that when ideas are put through controlled experiments, less than a third move the metrics they were designed to improve, and in a heavily optimized product like Bing the number is lower (Bing Search Blog, 2013).

Put the three together: most ideas don't win, small samples swing wildly, and stopping early amplifies the swings. The result is that on a low-traffic site, many of the "wins" you see are probably just noise.

Fake results, real costs

"What's the harm in testing anyway?" There is harm, in four places:

1. Building the variations. Every variant needs design, development, QA on mobile and desktop, and tracking. Those are real hours, whether you do them in-house or pay someone.
2. The wait. Months spent waiting on a test that never reaches its sample size are months the obvious problems on your site keep turning customers away.
3. Shipping a useless change, or a harmful one. A fake result doesn't just call a win where there's nothing. It can crown a version that is actually slightly worse, and you lock that worse version into your site.
4. Learning the wrong lesson. This is the most expensive one. The team now believes "red buttons sell better than green", and carries that belief into the next three projects. A wrong conclusion stamped "validated by data" is very hard to undo.

I ran 150+ tests for Neighborly. That's exactly why I'm writing this.

Let me be clear so you don't read this as anti-testing. I ran the experimentation program for Neighborly, a large US home-services brand: more than 150 A/B tests, a 35% lift in booking rate, and $2.8M in added revenue (see the work). A/B testing is one of the tools I trust most.

But I have to admit a professional reflex that was wrong: after that many tests, my first thought on looking at any website was "we should test this". For a small site, that reflex is wrong.

The Neighborly program worked because it had three things small sites usually lack: high traffic, so each test reached its sample size in a reasonable time; measurement that had been checked carefully, so the numbers could be trusted; and enough tests, so small lifts could compound into a large result. That 35% didn't come from one magic test. It came from a program.

Ron Kohavi, who led experimentation at Microsoft and Airbnb, named the same three conditions in a conversation about experimentation: working in a domain where changes are easy, such as websites; having enough users; and having trustworthy results. The write-up of that conversation puts it this way: once you have tens of thousands of users, you can start experimenting at scale.

Copying a big brand's testing playbook onto a site with a few thousand monthly visitors is like handing a marathoner's training plan to someone who just started running. The plan isn't wrong. It just isn't for you yet.

Low traffic? Do these 5 things, in this order

Skipping classic A/B tests doesn't mean going back to guessing. It means using the right tool for the amount of data you actually have. This is the order I recommend.

1. Fix the obvious problems first, no test required

The form breaks on iPhone. The contact button is hidden on small screens. The page takes 6 seconds to load on mobile data. There's no price anywhere. None of these needs an A/B test. They need fixing. Nobody A/B tests whether a shop with a jammed front door sells less.

Speed is the clearest example. The Milliseconds Make Millions study, run by Deloitte and commissioned by Google across 37 European and US brands, found that a 0.1-second improvement in mobile site speed lifted retail conversions by 8.4% and average order value by 9.2%. You don't need 40,000 visitors to decide to make your pages faster.

And before you trust any conversion number at all, make sure it's measured correctly. If GA4 is double-counting orders or counting button clicks instead of real form submissions, every before-and-after comparison is meaningless. The 7 most common GA4 and GTM setup mistakes is the place to start.

2. Do qualitative research: watch real people use the site

With low traffic, you don't have the numbers to know which version is better. But you can absolutely find out why people leave, and that question is often worth more.

• Session recordings and heatmaps. Microsoft Clarity offers session recordings and heatmaps and, per Microsoft's own FAQ, is free forever with no traffic limits. Watch 20–30 sessions of people who landed on your service page and didn't submit the form. You'll see them scroll hunting for a price, tap an image that isn't a link, or abandon at the phone-number field.
• Interview or observe 5 people. Nielsen Norman Group, one of the most respected names in user-experience research, found that a study with 5 users uncovers about 85% of usability problems, and recommends several small rounds of 5 over one large study. Ask five real customers (not staff) to book through your site while you sit quietly and watch. It will be the most valuable afternoon of the quarter.
• Ask the people who bought. "What almost stopped you from getting in touch?" One question, sent after the deal closes. The answers that keep repeating are your fix list.

3. If you change something, change something big

Look back at the table: to detect a 50% lift instead of 20%, the sample size drops from about 19,800 to about 3,300 visitors per variation. The bigger the effect, the fewer people you need to see it.

So on a low-traffic site, don't test button colors. Change things that could plausibly make a big difference: how the offer is presented, showing a price versus not, cutting a form from 9 fields to 3, replacing a generic homepage with a message about the customer's actual problem. If a change is so small it takes 150,000 visitors to measure, then for your business right now it isn't worth measuring.

4. Compare before and after, with discipline

When you can't split traffic, the practical option is comparing the period before a change with the period after. This is weaker than an A/B test, and you should know exactly where it's weak:

• Seasonality: comparing the month before a major holiday with the month after is comparing two different markets. Compare the same period year over year, or at least the same number of weeks and the same days of the week.
• Traffic sources: if you started a new ad campaign the week after, the conversion rate changed because the visitors changed, not the page. Compare each source separately.
• One change at a time: change the headline, the form and the pricing at once, and even if conversions go up you won't know why.
• Log the launch date: an annotation in GA4 or a shared doc. Three months from now nobody will remember.

Treat a before-and-after result as a directional signal, not proof. If the signal is clear and matches what you saw in your qualitative research, keep the change. If it's murky, don't tell anyone it's been "proven".

5. Test only where the traffic is, and measure something closer

If you do want to run a real A/B test, pick the single highest-traffic page you have, usually the homepage, a paid-traffic landing page or the main category page. Don't spread tests across ten pages with a few hundred visits each.

You can also measure a metric that sits earlier in the customer journey. The click-through rate on a "Book now" button is usually much higher than the completed-booking rate. With a 10% baseline instead of 2%, detecting a 20% lift needs about 3,600 visitors per variation instead of 19,800. The trade-off is real: more clicks don't guarantee more customers. Keep an eye on the final metric too, so the winning version isn't just pushing more people into a form they then abandon.

When A/B testing starts to pay off

A/B testing isn't something to abandon. It's something to wait for. You're ready when you can say yes to all five of these:

• A sample-size calculator says the page reaches its sample in about 4–6 weeks at the lift you genuinely expect.
• Conversion tracking has been reconciled against real orders or real form submissions, not just "GA4 shows numbers".
• Your hypothesis comes from research (recordings, interviews, data), not from a "10 conversion hacks" listicle.
• You commit to fixing the sample size up front and not stopping early, even when week one looks great.
• You plan for a series of tests, because the value is in the program, not in any single test.

Missing any of these? Go back to the five steps above. They aren't a stopgap. They're the foundation every serious testing program needs anyway.

If the concept is new to you, What is A/B testing? explains it from the ground up, without jargon.

The real cost of following the crowd

Back to "don't guess, A/B test it." That's true, but only for people who have the data. For a low-traffic site, the honest version is: don't guess. Observe, fix what's already obvious, and save A/B testing for when the numbers are big enough to answer.

The business that follows the crowd spends months, a tool subscription and the hours to build variations, and gets back a "win" nobody can confidently tell apart from luck. The business that works in the right order spends the same months fixing the broken form, speeding up the pages and sharpening the offer. Those changes don't need statistics to prove them. Your customers prove them for you.

If you want to know whether your site has enough traffic to test yet, or what to fix first, take a look at A/B testing and conversion optimization. I start with the sample-size math, not with a tool. Or if you'd rather send your numbers straight over, get in touch here.

---

Sources:

• Evan Miller — Sample Size Calculator (sample sizes in this post were computed with this calculator's own formula: 5% significance, 80% power, two-sided)
• Evan Miller — How Not To Run an A/B Test
• Bing Search Blog — Large Scale Experimentation at Bing (08/2013)
• Wingify — 1,000 Experiments Club: A Conversation With Ronny Kohavi
• Deloitte — Milliseconds Make Millions
• Microsoft Learn — Clarity: Frequently asked questions
• Nielsen Norman Group — Why You Only Need to Test with 5 Users