If you run a one-person business, you don’t get the sample sizes a data team at a big company gets. You might test two email subject lines against a list of 40 people, or compare upgrade rates between two customer segments where one segment only has 15 people in it. At some point you’ll be tempted to reach for a chi squared test to see if the difference you’re looking at is real or just noise. That’s a reasonable instinct. It’s also where a lot of people get burned, because the chi squared test has an assumption baked into it that small datasets routinely violate.
This isn’t a reason to avoid the test. It’s a reason to know what it’s actually doing under the hood, so you know when to trust it and when to swap in something more reliable.
What a chi squared test is actually checking
A chi squared test of independence asks one question: are two categorical variables related, or are they behaving like they’re independent of each other? Categorical means the data falls into buckets, not a continuous number. “Opened the email” versus “didn’t open it.” “On the annual plan” versus “on the monthly plan.” “Churned” versus “stayed.”
You lay the counts out in a table (statisticians call it a contingency table), with one variable across the rows and the other across the columns. If the two variables were truly unrelated, you’d expect the counts in each cell to follow the overall proportions in the data. The test compares what you actually observed against what you’d expect if there were no relationship at all, and asks how far apart those two pictures are.
This is different from a goodness of fit chi squared test, which compares one set of observed counts against a theoretical distribution, like checking whether your traffic sources this month match the split you’d expect from history. Same underlying math, different question. Most small business use cases, like comparing an A/B test outcome or checking whether churn differs across two customer groups, are independence tests, so that’s what I’m focused on here.
The mechanics in plain terms
For each cell in your table, you calculate an expected count based on the row and column totals. Then you compute the squared difference between the observed count and the expected count, divide by the expected count, and add that up across every cell. That sum is your chi squared statistic. The bigger it is, the further your actual data is from what independence would predict.
To decide if that statistic is “big enough” to call meaningful, you compare it against a chi squared distribution with the appropriate degrees of freedom. That comparison is where the trouble starts, because the chi squared distribution is a continuous approximation. Your actual data, especially with small counts, is discrete and lumpy. The approximation works well when your numbers are large enough to smooth out. It works badly when they’re not.
Why small samples break the test
The chi squared statistic is only reliably approximated by the chi squared distribution when the expected counts in your table are large enough. When expected counts get small, particularly under 5, the approximation gets shaky and the p-value it produces stops being trustworthy. In practice this usually means the uncorrected test overstates how significant your result is. You can end up looking at a p-value under 0.05 that would tell a very different story if calculated correctly.
This matters because a solo operator’s data is frequently exactly this kind of small. A test group of 10 or 15 people isn’t unusual when you’re segmenting an email list or comparing two cohorts of early customers. If you run a standard chi squared test on numbers like that without checking the expected cell counts first, you’re at real risk of convincing yourself a difference is real when it’s within the range you’d expect from chance alone.
What counts as “small”
The common rule of thumb, going back to statistician William Cochran, is that every expected cell count in a 2×2 table should be at least 5. For larger tables, the guideline loosens slightly: no more than 20% of cells should have an expected count under 5, and none should be under 1. Note that this is about the expected count, not the observed count, so you have to actually do the calculation rather than eyeball your raw numbers.
It’s easy to have a decent-sized total sample and still fail this check. If you’re comparing three customer segments across five outcome categories, that’s a 3×5 table with 15 cells, and even a sample of 100 people can leave several of those cells with an expected count under 5.
A hypothetical walkthrough
Here’s a made-up example to show the mechanics, not a real result from any business. Say you test a trial-extension offer on 20 people, split into two variants of 10 each, and track who upgrades.
Variant A: 2 upgrades out of 10.
Variant B: 5 upgrades out of 10.
The expected count under “no difference” for each cell, based on the row and column totals, works out to 3.5 or 6.5 depending on the cell. Every one of those falls below the usual threshold of 5.
Run the standard chi squared calculation on this table and you get a statistic of roughly 1.98, which lands at a p-value around 0.16. Not significant at the usual 0.05 cutoff, but close enough that someone might be tempted to say “almost there, let’s watch this.”
Apply Yates’ continuity correction, a small adjustment designed for 2×2 tables that subtracts 0.5 from each difference before squaring it to better match the discrete nature of the data, and the statistic drops to about 0.88, pushing the p-value out to roughly 0.35. Run Fisher’s exact test, which calculates the exact probability of your table (and every table more extreme) directly from the hypergeometric distribution instead of approximating with a curve, and you land at a p-value close to 0.35 as well.
Same data, three different answers, and the uncorrected chi squared test is the outlier. That gap is the whole reason this matters. If you’d stopped at the plain chi squared number, you’d have walked away with a more optimistic read than the data actually supports.
Fisher’s exact test: the standard fix
For 2×2 tables, which cover most simple A/B comparisons, Fisher’s exact test is the standard substitute when your expected counts are small. Instead of approximating with a distribution, it computes the exact probability of the observed table, and anything more extreme, given your fixed row and column totals. It doesn’t rely on a large-sample assumption at all, which is exactly what you need when your sample isn’t large.
Most spreadsheet tools don’t have Fisher’s exact test built in the way they have a chi squared function, but it’s a standard option in R, Python’s SciPy, and most statistics packages. For anything bigger than a 2×2 table, the equivalent is harder to compute by hand, but the same software will usually offer a Fisher’s exact or simulation-based version for larger tables too. Another practical option, if your table has categories that are individually thin, is to combine categories so the expected counts clear the threshold. Merging “trial” and “free” into one “non-paying” bucket, for instance, can turn an unreliable table into a usable one, as long as the merge still makes sense for the question you’re asking.
What none of this tells you
Getting the right test doesn’t turn a small sample into a strong one. Small datasets have low statistical power, which means a real difference can easily fail to reach significance simply because you don’t have enough data to detect it. A non-significant result on 20 people isn’t proof that two variants perform the same. It might just mean you asked a question your sample size can’t answer yet.
It’s also worth saying plainly: a p-value, however it’s calculated, tells you about the odds of seeing your pattern by chance. It doesn’t tell you why the pattern exists, whether it will hold up next month, or whether the difference is big enough to matter for your business. A statistically significant jump from 2 upgrades to 5 upgrades out of 10 is still only 3 people. Whether that’s meaningful depends on your judgment about the business, not on the test.
A short checklist before you run one
Before you trust a chi squared result on a small dataset, run through this:
- Calculate your expected cell counts, not just your observed ones, and check whether any fall under 5.
- If you’re working with a 2×2 table and counts are small, default to Fisher’s exact test instead.
- If you have more categories than you need, consider combining thin ones before testing, as long as the grouping still makes sense.
- Don’t test a dozen small segments after the fact looking for one that comes back significant. Testing many small slices raises your odds of a false positive by chance alone.
- Treat “not significant” as “not enough evidence yet,” not as “confirmed no difference,” especially with small samples.
None of this makes the chi squared test useless for a small operation. It just means the test needs more care exactly when your data is thinnest, which is usually when you can least afford to be wrong about what it’s telling you.
If you want more of this kind of plain-English walkthrough on the methods and metrics that actually show up in running a small business, you can find the rest of the site here.
Get new guides and videos first — join the Telegram channel.