The dashboard that always has good news
Open any analytics dashboard long enough and you will find something that looks like a win. Conversion rate is up 4% on mobile. Tuesday signups from the newsletter are outperforming every other day. The cohort that joined in March has noticeably better retention than the one from April. Each of these looks like a finding. Most of the time, it is not one.
This is the multiple comparisons problem, and it is one of the easiest ways a solo operator can talk himself into a decision that is not actually supported by the data. I have done it. I have pulled a “win” out of a dashboard, built a plan around it, and later realized the win was never there. It was just noise that happened to land in a shape that looked like a pattern.
Why looking twenty times finds something
Statistical significance, in plain terms, is a threshold for how surprised you should be by a result if there was actually nothing going on. A common threshold is a 5% chance of seeing a result that strong purely by random luck, even when nothing real is happening. That 5% is not zero. If you run one clean comparison, a 5% false alarm rate is a reasonable risk to accept.
The problem is that nobody runs one comparison. You run an A/B test, and then you check it by device. Then by traffic source. Then by day of week. Then by new versus returning visitor. Then by every cohort you have. Each of those is a separate roll of the dice, and each roll carries its own 5% chance of a false alarm even if your change did absolutely nothing.
Roll that die twenty times and the odds of at least one false alarm stop being small. It is not that any single check is dishonest. It is that when you check enough things, something will clear the bar by chance alone, and it will look exactly like a real result because that is what a false alarm looks like when it happens: indistinguishable from a true one, in isolation.
A concrete example: the segmented A/B test
Say you run an A/B test on a landing page for six weeks. Headline A versus headline B. The overall result comes back flat, no meaningful difference in conversion rate. Disappointing, but clean.
Then you start slicing. Mobile versus desktop. Organic versus paid. Weekday versus weekend. First-time visitor versus returning. Each slice is another comparison. If you check eight or ten slices, it becomes fairly likely that one of them shows a difference big enough to look “significant,” purely from the sample splitting into smaller, noisier pieces and one of them landing favorably by chance.
The trap is that this is exactly the moment your brain wants to declare victory. “Headline B actually works, but only for mobile users coming from paid traffic.” That is a specific, satisfying, story-shaped finding. It is also the kind of finding multiple comparisons manufactures constantly, because the more specific and narrow the slice, the smaller the sample, and the more room there is for noise to look like signal.
None of this means segment-level differences are never real. It means that finding one after scanning many slices is weak evidence on its own, and needs to be treated that way until it survives a second look.
A concrete example: cohort scanning
Cohort analysis has the same trap built in. If you look at monthly signup cohorts and track their 90-day retention, you will naturally see some variation between cohorts. Some months will retain a bit better, some a bit worse. That is expected even if nothing about your product or onboarding changed at all, just from the normal churn in who signs up and when.
If you scan twelve months of cohorts looking for “the one that worked,” you will almost always find one that looks meaningfully better than the rest. Then the natural next step is to ask what was different about that cohort, and you will always be able to find something, because you can always find a difference between any two groups of people if you look hard enough. Maybe that cohort came in right after you tweaked your onboarding email. Maybe it was a seasonal effect. Maybe it is just noise. Scanning after the fact cannot tell you which.
What a correction actually does, and does not do
Statisticians have formal ways to account for this, like Bonferroni corrections or false discovery rate controls. In plain terms, these methods raise the bar for what counts as significant when you are running many comparisons at once, so your overall false alarm rate across all those comparisons stays close to what you intended, instead of quietly compounding with every extra slice you check.
That is useful, and worth knowing exists if you are running formal experiments. But it is not a fix that makes multiple comparisons safe to do casually. A correction only protects you if you decide in advance how many comparisons you are running and apply it to all of them together. If you slice your data twenty ways, notice one interesting result, and then go back and calculate a correction just for that one comparison, you have not fixed anything. You have to commit to the full set of comparisons before you see the results, which is exactly the discipline that is hardest to keep when you are the only person looking at your own dashboard and free to check whatever you want, whenever you want.
What actually helps a one-person operation
You do not need formal statistical corrections to protect yourself from this. A few habits do most of the work.
Decide what you are testing before you look at the data. If you are running an A/B test, write down beforehand which metric matters and whether you plan to check any subgroups. If a subgroup result shows up that you did not plan to check, treat it as an idea to test again, not as a conclusion.
Be suspicious of narrow, specific wins. “Conversion is up overall” is more trustworthy than “conversion is up for mobile users who arrived on a Tuesday from a specific ad.” The narrower the slice, the smaller the sample behind it and the easier it is for chance to produce a number that looks big.
Replicate before you act. If a segment or cohort looks like it is behaving differently, the cheapest test is time. Does the pattern hold up over the next few weeks, or was it a one-off blip. A result that only ever showed up once, in one look, is not something to build a business decision on.
Hold something back. If you have enough data, split it. Look for a pattern in half of it, then check whether that same pattern shows up in the other half before you trust it. This is a rough version of what statisticians call a holdout set, and it does not require any special tooling, just the discipline to not peek at everything at once.
Care more about the size of the effect than whether it crossed a line. A conversion lift that is technically significant but tiny in absolute terms rarely justifies rebuilding a funnel around it. A meaningful, repeated effect is worth more attention than a narrow one that barely cleared a threshold once.
The honest limit here
None of this tells you whether a specific pattern in your own data is real or noise. It only tells you to be more skeptical the more looking you did to find it, and it will not save you from a genuinely misleading result if you only check something once and get unlucky. This is a way of thinking about risk, not a guarantee, and it is not a substitute for actually understanding your business well enough to know whether a result makes sense in the first place. Numbers do not replace judgment. They inform it, if you are careful about how you look at them.
If you run your own numbers and want more of this kind of plain-English look at the metrics and methods behind small business decisions, you can find the rest of the collection here.
Get new guides and videos first — join the Telegram channel.