I run my own numbers. No analyst, no data team, no statistician on retainer. So when I want to test a new checkout page or a different email subject line, the question “how many people do I need to see this before I trust the result” falls on me. For a long time I answered it badly, by just running the test until I got bored or until one version “felt” ahead. That’s a guess with a spreadsheet attached.
This is the version of sample sizing I actually use. It won’t make you a statistician, but it will get you past the guessing stage.
Why this question matters more than it seems
A sample size is just the number of visitors, customers, or transactions you need to look at before you can trust a difference between two things. If you show 40 people two versions of a landing page and 22 convert on one and 18 on the other, that’s not evidence of anything. It’s close enough that it could easily flip the other way if you ran it again. Small samples are noisy. The fewer people in your test, the more random luck decides the outcome instead of the actual difference between your two versions.
The uncomfortable part is that this noise doesn’t announce itself. A small sample will still hand you a clean-looking percentage difference. It just won’t tell you whether that difference is real or whether you got unlucky with the coin flip. Sizing your sample ahead of time is how you stop fooling yourself.
The three inputs every sample size calculation needs
You don’t need to know the math behind a sample size calculator to use one well, but you do need to feed it the right three things. Get any one of these wrong and the number it spits out is meaningless.
Your baseline rate. This is what you’re already getting: current conversion rate, current open rate, current checkout completion rate. Pull this from your own data, not an industry benchmark, because your traffic and your audience behave differently from someone else’s. If you don’t know your baseline, that’s step zero, not something to skip.
The minimum effect you actually care about. This is the part most people skip and it’s the one that matters most. Ask yourself: what’s the smallest improvement that would be worth acting on? If your current checkout conversion is 3% and you’d only bother changing your checkout flow for a jump to 3.5% or better, that’s your minimum detectable effect. Smaller differences than that either aren’t worth the engineering time to implement, or they’re so small your test would need an enormous sample to ever confirm them. Chasing a 0.1 percentage point difference on a website that gets a few hundred visitors a day is not a fight you can win.
Your confidence level and power. These sound academic but the idea is simple. Confidence level is how sure you want to be that a result you see isn’t just noise, usually set at 95%. Power is how likely you are to actually catch the effect if it’s real, usually set at 80%. You don’t need to memorize why those specific numbers are standard. Just know that raising either one means you need a bigger sample, and lowering either one means you’re accepting more risk of a false read in exchange for a faster test.
Plug those three numbers into any free A/B test sample size calculator (there are several solid ones online, and most tools like Optimizely or VWO have one built in) and it will hand you a number: how many visitors per variant you need before you look at the result.
A worked example
Say you run a small subscription box business and your current email has a 2% click rate. You want to test a new subject line and you’ve decided that anything less than a jump to 2.5% isn’t worth writing new subject lines every week for. You plug in a 2% baseline, a target of 2.5%, and standard 95% confidence and 80% power settings. The calculator will tell you that you need a few thousand opens per variant, not a few hundred, before that specific difference would show up reliably.
If your list is only 800 people, you now know something useful even before you send anything: you cannot reliably detect a jump from 2% to 2.5% with a list that size. You have two honest options. Either you widen the effect you’re testing for (only chase changes that would move the click rate to something like 4%, a bigger swing that a smaller sample can actually detect), or you accept that on a list this size, you’re making a judgment call informed by the data rather than a statistically confirmed one. Both are legitimate. Pretending you ran a “real” test when your sample was too small to detect the effect you were hoping for is the part to avoid.
What skipping this step actually costs you
I used to run tests until a difference “looked” big enough, then stop and declare a winner. The problem is that with small samples, a swing to a clear-looking winner and then a swing back is completely normal, it’s the sample never having been big enough to settle in the first place. I’ve killed page variants that were probably fine and kept ones that weren’t actually better, because I stopped watching too early.
Sizing the sample before you start solves this in a boring but effective way: you decide upfront how long you’re going to run the test and you don’t look at the result to decide when to stop. That discipline is worth more than any calculator, because the calculator only works if you actually respect the number it gives you.
When your business is just too small for a proper test
This is the honest limitation. A lot of solo operators and small shops don’t have the traffic volume to detect small effects in any reasonable timeframe. If your site gets 50 visitors a day, a calculator asking for a sample of several thousand per variant is telling you the truth: that test isn’t going to finish this quarter, maybe not this year.
In that situation the right move isn’t to fake a test on data too thin to support it. It’s to either widen what you’re testing (bigger, bolder changes produce bigger effects, which need smaller samples to detect), stretch your test window and be patient, or accept that some decisions at your scale will be made on judgment, qualitative feedback, and your own read of the business rather than a formal split test. That’s not a failure. It’s matching the tool to the size of the operation.
What a sample size calculation can’t do for you
Getting your sample size right protects you from one specific kind of mistake: reading noise as signal. It does not protect you from a bad test design, a biased audience split, running two tests on your site at the same time so they interfere with each other, or measuring the wrong metric entirely. A well-sized test on the wrong question is still the wrong question, just with more confidence attached to the wrong answer.
It also doesn’t tell you what to do with a result that turns out statistically real but tiny in business terms. A confirmed 0.3 percentage point lift might be true and still not worth the cost of maintaining two versions of a page. Sample size math tells you whether you can trust a number. It doesn’t tell you whether that number matters to your business, and it never replaces your own judgment about what’s worth acting on.
If you want more of this kind of plain-English breakdown of the metrics and methods that actually run a small business, from A/B testing to cohort analysis to figuring out which numbers are worth watching in the first place, you can find the rest of what I’ve written over at the home page.
Get new guides and videos first — join the Telegram channel.