I wrote off a modem model on the strength of two dead units.
Two, out of a batch of eight. Both refused to register on the network, both the same model I had been buying happily for months. So I decided the supplier had slipped me a bad revision, ordered a different model at roughly twenty percent more per unit, and felt decisive about it.
The two dead ones were plugged into the same powered hub. One bank of ports on that hub had gone. The modems were fine. I still have them, and they still work.
A hub costs me about a hundred and fifty dollars and carries around thirty ports, so the actual fault was a fraction of one cheap component. What it bought me was a more expensive modem on every order after that, and a few months of quietly distrusting a supplier who had done nothing wrong. Two data points did that.
There is no row count that means enough
Search this question and you get numbers thrown at you. A thousand rows. Thirty per group. Three hundred sessions. A full month.
Every one of those is a rule of thumb that escaped from the situation it was invented for. None of them knows what you are trying to decide, so none of them can tell you when to stop.
Enough is a property of two things: the size of the difference you are hoping to see, and the decision waiting on the other side of it. The number of rows you happen to have is the last thing in that chain, not the first.
Enough depends on how big the difference is
Say forty people a month reach your signup form and twelve of them finish. That is your baseline.
If a change pushes that from twelve to eighteen, you will see it. The shape of it turns up inside a couple of months, because six extra people out of forty is a movement large enough to survive the ordinary week to week wobble.
If a change pushes twelve to thirteen, you will never see it. Not this year, not next year. One person is inside the range that your number bounces around anyway, and no amount of patience at forty a month separates it from luck.
Both changes might be real. Only one is visible at your volume. So the useful question before you go digging is roughly how big the thing you are hunting is, and how many rows you happen to own comes a distant second. Large differences are cheap to detect. Small ones are ruinously expensive, and most solo businesses cannot afford to look for them at all.
That sounds like bad news. It is actually a filter. If the effect you are chasing is small, you can skip the analysis entirely and spend the afternoon on something with a bigger swing in it.
Volume will not fix a sample that leans
Here is the failure that costs more than thin data, because it does not feel like a failure.
Everything above assumes your data is a fair picture of the thing you care about. When it is not, more of it does not help you. More of it makes you more confident about the wrong picture.
My own favourite example sits inside my analytics. Any javascript based analytics tool cannot see visitors whose blocker ate the script. Fine, except my audience is people buying proxies and automation tooling, which is to say people who are unusually likely to run a blocker. The visitors invisible to my analytics are disproportionately the ones most likely to buy. The tool is bent in one specific direction, and it is bent hardest exactly where I care most.
If I ten times my traffic, that lean does not wash out. Random noise averages away with volume. A bias does not, because every error points the same way, so stacking more of them just makes the crooked picture smoother and more convincing.
Which gives you the rule that matters. A large sample of the wrong people is more dangerous than a small sample of the right ones. A small sample feels thin, and feeling thin keeps you sceptical. A large one feels authoritative, and that feeling is what switches your judgement off.
Five conversations with people who did not buy will teach you more than fifty thousand pageviews from a tool that cannot see them.
The calendar is lying to you before you start
Then there are the dumb traps, which catch far more people than the subtle ones.
Singapore has a lot of public holidays. Chinese New Year, Hari Raya, Deepavali, National Day, Christmas. When one lands inside the seven days you are staring at, you are holding six working days and calling it a week. Set that against a clean week and you get a drop of fifteen or twenty percent with no cause behind it beyond people having the day off. I have gone a genuinely embarrassing distance into investigating one of those before the penny dropped.
The partial month version is worse because it happens every single month. You look on the nineteenth, this month is well down on last month, and your stomach drops. Of course it is down. You are comparing nineteen days against thirty one. Dashboards fill the current period box as the period goes along, so any look before the month closes shows you a number guaranteed to be losing.
Two finished months are not clean either. February has twenty eight days and January has thirty one, which is a ten percent gap baked into the calendar before your business does anything at all. Some months carry five weekends and some carry four.
There is a version of this specific to subscriptions that took me a while to spot. Monthly renewals dated on the thirty first do not exist in February, so they shuffle, and a small pile of them can slide across a month boundary and make one month look soft and the next look strong. Nothing happened. The calendar just moved some money around.
Compare windows of the same length, or a rate per day, or the same span a year earlier. When you cannot, say out loud that the comparison is dirty before you let yourself conclude anything from it.
Every filter costs you evidence
Segmentation is the other daily one.
A few hundred customers is workable. Then you filter to one country, then one plan, then people who signed up this quarter. Each cut multiplies against the last, and three of them on a modest base leaves you reading four people.
The chart prints a percentage anyway. Seventy five percent, same font, same confident colour as the number built from everybody. It will not mention that the seventy five percent is three out of four, because almost no chart shows its denominator, and that is a design choice rather than an oversight.
So put the raw count next to every percentage in anything you build for yourself. Write n equals four right there in the cell. A surprising share of your questions dissolve before you have to think about them.
Write down what would change your mind
This is the habit that fixed more of my analysis than any statistics ever did.
Before opening the data, write one sentence. If the number comes back above this, I do that. If it comes back below, I do the other thing. Mine last month was roughly: if more than half of this month’s tickets are about setup, I rewrite the setup guide, and if not, I leave it alone until October.
Thirty seconds of work. It has to exist before you see the answer, because afterwards you will quietly slide the threshold to wherever the answer already sits, and you will not even notice doing it.
If you cannot write that sentence, if no result would make you act differently on Monday, then what you are about to do is entertainment. That is allowed. I read numbers for fun too. But name it, because the entertainment version has a habit of producing a decision anyway, and that decision walks away carrying a confidence it never earned.
The useful side effect is that the sentence converts an unanswerable question into an answerable one. How much data is enough becomes how much data do I need to separate above from below, and that has a shape you can reason about, because you have finally named the size of the gap.
When you honestly do not have enough
You will often write the sentence and then find your data cannot answer it. At small volume that is the normal outcome.
Run it longer. Time is the only data you can buy for nothing, and most tests get killed at two weeks out of boredom rather than because they finished. A decision that can wait six weeks should wait six weeks. Mine could have. My modem supplier switch could have waited a fortnight and cost me a single hub port instead.
Or widen the question. You cannot tell which of two headlines wins. You can probably tell whether anyone scrolls past the first screen. Drop to a coarser question the data can actually answer, and answer that one.
Or make the call on judgement and label it as one. Write in your notes: decided on taste, no evidence, revisit in March. This matters more than it sounds. Decisions harden inside a business, and six months from now somebody, probably you, will find that change, assume there was a reason underneath it, and start building on top of it. An unmarked guess becomes a fact the moment everyone has forgotten it was a guess.
What I am not going to pretend
I do not run significance tests on my own operations. At my volume the maths would mostly confirm what I already suspect, which is that I do not have the data to be sure, and I would rather spend those hours making changes large enough that a microscope never enters into it.
Plenty of people will argue with that, and if you run something with real volume they are right and I am not talking to you. But if you are one person reading your own numbers on a Sunday evening, the thing holding you back is almost never your statistical technique. It is that you looked at nineteen days, in a month with a holiday in it, filtered down to one segment, having never decided what would change your mind. Every metric and method behind that sentence is written up in plain English at dataresearchanalysiscollection.com, including the ones I got wrong first.
Get new guides and videos first — join the Telegram channel.