What the base rate fallacy actually is
The base rate fallacy is what happens when you judge how likely something is based on a specific piece of evidence, while ignoring how common that thing was to begin with. The “base rate” is just that starting frequency: how often something happens in general, before you look at any particular case.
Here’s the classic version. Say a test for a rare disease is 99% accurate. Someone tests positive. What are the odds they actually have the disease? Most people’s gut answer is “99%.” The real answer depends heavily on how rare the disease is. If only 1 in 10,000 people have it, then even a 99% accurate test will flag a lot of healthy people as false positives, because there are so many more healthy people to begin with. Run the numbers and a positive result might mean less than a 1% chance of actually having the disease, even though the test is “99% accurate.”
That gap between what a test result feels like it’s telling you and what it’s actually telling you is the base rate fallacy. It shows up constantly once you run a business on numbers, because most of the alerts, models, and metrics you look at day to day are exactly this kind of test.
Why this matters if you run your own numbers
I don’t have a data science team. When I look at churn flags, fraud alerts, or a funnel report, I’m the one deciding whether to act on it. The base rate fallacy is dangerous precisely in that setup, because the instinct is always to trust the specific alert in front of you and forget to ask “how common is this event across my whole base, and how does that change what this alert actually means.”
Three places this bites solo operators the hardest: churn prediction, fraud or spam detection, and reading A/B test results on a small conversion rate.
Churn prediction: the alert is not the truth
Say you build (or buy) a churn model that flags “high risk of cancelling next month” and it’s right 90% of the time when it says someone is going to churn. Sounds solid. But if only 3% of your customers churn in a given month, look at what happens across, say, 1,000 customers. Roughly 30 will actually churn. Your model catches most of those, but it’s also going to falsely flag a chunk of the other 970 who were never leaving. Depending on the model’s false positive rate, the number of people wrongly flagged as “high risk” can end up close to, or even higher than, the number of genuinely at-risk customers.
The practical effect: you get a list of “high churn risk” accounts, and you’d be wrong to treat every name on it as someone about to walk out the door. A meaningful share of that list is healthy customers who just look risky by whatever pattern the model keys on. If you use that list to justify a discount, a win-back email, or a founder phone call, you’re spending time and goodwill on people who were never leaving, and possibly annoying them by implying they were about to churn when they weren’t.
This doesn’t mean churn models are useless. It means the base rate (how rare churn actually is in your base) has to be part of how you read the output, not an afterthought. A flag is a prioritization signal, not a verdict.
Fraud and spam filters: same math, different label
Payment fraud, signup spam, and referral abuse all have the same shape. Genuine bad actors are a small fraction of total traffic. So even a filter with a low false positive rate will, in absolute terms, catch a lot of real customers in its net, simply because real customers vastly outnumber fraudsters.
If you’ve ever had a customer complain that their card got declined or their account got flagged for “suspicious activity” when they did nothing wrong, this is usually why. It’s not that the filter is broken. It’s that the base rate of fraud is low enough that even a well-tuned filter produces a meaningful number of false positives relative to the small number of true positives it catches. Knowing this changes how you set the filter: tightening it to catch one more fraudster might cost you several legitimate customers, and that trade only makes sense if you’ve actually looked at your fraud rate rather than assumed it.
A/B tests on a small conversion rate
This one is subtler. If your baseline conversion rate is low, say a landing page converts at 1-2%, then a “test” that ran for a few days on modest traffic is working with a small base rate of the event you actually care about (a conversion). Small numbers of successes swing around a lot by chance alone. A variant that looks like it’s “beating” the original by 30% might just be noise, because the base rate of conversion was never high enough for that sample size to say much.
The base rate fallacy here isn’t about a diagnostic test, it’s about mistaking a small, noisy sample for a confident readout. The fix isn’t a formula so much as a habit: before trusting a lift number, ask how many actual conversions (not visitors, conversions) the test was built on. A lift built on 12 conversions per arm is a different claim than one built on 400.
How to correct for it without a stats degree
You don’t need Bayes’ theorem memorized to protect yourself here. A few habits do most of the work:
Ask “how common is this to begin with?” before trusting any specific alert. If churn, fraud, or conversion events are rare in your base, expect any imperfect signal to produce more false alarms than feels intuitive.
Look for the raw counts, not just the rate. “90% accurate” tells you almost nothing on its own. “Out of everyone flagged this month, how many actually turned out to be true” tells you what you need to know. If your tools don’t show you that, that’s worth asking for or calculating by hand from a sample.
Treat flags as a shortlist, not a verdict. A churn flag, fraud flag, or “winning” variant is a reason to look closer, not a fact to act on immediately, especially when the underlying event is rare.
Be skeptical of exciting-sounding case studies and one-off outcomes, your own or anyone else’s. A single anecdote or standout result tells you almost nothing about the base rate it came from. That’s true of a customer testimonial and it’s true of your own best week ever.
None of this makes churn models, fraud filters, or A/B tests useless. They’re still better than guessing. But a metric or a flag is only as trustworthy as your understanding of what it’s actually measuring against, and the base rate is usually the missing piece. I don’t treat any single number as the whole picture, and I’d be doing you a disservice if I told you a model or a test result was a substitute for actually looking at what’s going on with your customers.
If you want more of this kind of plain-English breakdown of the metrics and methods behind running a data-driven small business, without the hype or invented benchmarks, you can find the rest of what I’ve written here.
Get new guides and videos first — join the Telegram channel.