If you run a business with a few hundred visitors a week instead of a few hundred thousand, most advice about A/B testing was not written for you. It assumes you have enough traffic that running a test costs almost nothing. You do not have that luxury. Every visitor you send to a test variant is a visitor who might have converted on the version you already know works. That is the budget, and it is worth thinking about on purpose instead of by accident.
This is not a post about statistics for their own sake. It is about deciding, before you start a test, how much of your limited traffic and time you are willing to spend finding out something, and when that spend is not worth it.
The real cost of an experiment isn’t just money
When people say “test budget,” they usually picture ad spend or tool subscriptions. For a small operator, the bigger cost is almost always opportunity cost: the conversions, signups, or sales you give up by splitting traffic between a known page and an unproven one, for as long as the test runs.
Say you get 300 checkout visits a week and your current page converts at some rate you trust because you have watched it for months. Running a 50/50 test means half of those 300 visits go to a variant you have no track record on. If the variant is worse, you have paid for that information with real orders, not just with time. If your margins are thin, that is a real cost, even though it never shows up on an invoice.
This is why “budget” for a small-sample experiment means two things at once: how much traffic you are willing to divert, and how long you are willing to let the test run before you make a call either way.
Why small samples change the math
A/B testing works by comparing outcomes between two groups and asking whether the difference you see is big enough that it probably was not just noise. The fewer people in each group, the more noise there is, so a small difference in conversion rate can easily be random variation rather than a real effect.
This is not a flaw in the method. It is math. The number of visitors you need to detect a given improvement grows quickly as the improvement gets smaller. Detecting a large, obvious effect (a redesign that either clearly helps or clearly hurts) takes far fewer visitors than detecting a small one (a button color change that might move conversion by half a percentage point). With low traffic, you simply cannot reliably detect small effects in a reasonable amount of time. That is the constraint your budget has to work around, not fight.
Start with the smallest effect you’d actually act on
Before you decide how much traffic or time to spend, decide what size of result would actually change your behavior. This is usually called the minimum detectable effect, and it is the single most useful number to set before running a test with a small audience.
If you would only bother keeping a new page design when it beats the old one by a wide margin, you can test with fewer visitors, because you are not trying to detect a subtle difference, only a big one. If you care about small gains, you need more traffic and more patience to see them clearly. Most solo operators, whether they realize it or not, only care about the big wins anyway. Setting that threshold explicitly, in plain terms like “I want to see at least a meaningfully different result before I trust it,” keeps you from chasing precision your traffic cannot actually support.
Decide how long you’re willing to wait, not just how much you’re willing to spend
With a small user base, sample size accumulates over weeks, not hours. That means your real budget constraint is often time, not money. Ask yourself directly: how many weeks am I willing to run this before I make a decision either way, win, lose, or inconclusive?
Pick that window before you start, and hold to it. It is tempting to check results every day and stop the moment the numbers look good, but that habit inflates your odds of mistaking noise for a real effect, especially with a small sample where day-to-day swings are larger. Deciding on a fixed window in advance, and resisting the urge to call the test early because a few good days happened to line up, is one of the simplest ways to protect a small budget from being wasted on a false signal.
Two budgets: a money budget and a traffic budget
It helps to separate the two kinds of budget you are actually setting.
The traffic budget is what share of your visitors or customers you are willing to route into the test instead of your known-good version. For a low-traffic business, an even 50/50 split can feel expensive, so some operators send a smaller share, like a quarter or a third, to the new variant, accepting that the test will take longer to reach a usable sample in exchange for limiting the downside if the variant underperforms.
The money budget is anything you spend to make the test possible: ad spend to bring in enough visitors to test with, a tool subscription, or paid traffic used specifically to reach a sample size faster than organic traffic would. If you are considering buying traffic just to speed up a test, be honest about whether that traffic behaves like your normal customers. Paid clicks and organic visitors do not always convert the same way, and a test run mostly on rented traffic may not tell you much about how your real audience will behave.
Neither budget has to be large. The point is naming both before you start, so a test that is technically still “running” after six weeks does not quietly keep consuming traffic and attention with no decision point in sight.
When a formal test isn’t worth running
Sometimes the honest answer, given the traffic you have, is that a formal test is not worth the visitors it would cost. If reaching a sample size that could actually distinguish a real effect from noise would take you four months, and the change you are testing is low stakes (a headline tweak, a minor layout shift), it is often more sensible to make the change based on judgment, watch the trend over the following weeks, and move on. You are not going to get statistical confidence out of traffic you do not have, and pretending otherwise just burns time you could spend on changes with more obvious impact.
Save formal tests, with a stated traffic budget and a fixed time window, for changes where being wrong is expensive: pricing, the core checkout flow, or anything that touches how customers pay you. Those are the places where the cost of a wrong guess is high enough to justify the cost of finding out properly, even on a small audience.
What this can’t tell you
None of this turns a small sample into a large one. A budget for an experiment tells you how much you are willing to spend finding out something, not how certain the answer will be. Even a well-run test on a limited audience can produce a result that looks clear and still be wrong, and a result that looks murky can still be pointing at something real that a larger sample would confirm. Setting a budget protects your traffic and your time from being spent without limit. It does not replace judgment about whether the result you got is one you should actually trust.
If you want more explainers like this on running a data-informed one-person business, on metrics like churn and CLV, and on the analysis methods that actually hold up at small scale, you can find the rest of the collection here.
Get new guides and videos first — join the Telegram channel.