You test a new checkout page against your old one, watch the overall conversion climb, and ship the winner. Then, weeks later, you split that same test by where the traffic came from, and your stomach drops. The new page converted worse on your email visitors and worse on your ad visitors too. Worse in every single group, yet somehow better in total.
That is not a spreadsheet error and you are not losing your mind. It is a genuine statistical twist called Simpson’s paradox, and it is one of the most important traps a solo operator can learn to spot. I am not a statistician, I run my own small businesses, and I have been ambushed by this more than once: I shipped the wrong page, killed the better performing ad, and praised the worse month, all because I trusted one pooled figure and never asked what was hiding underneath it.
What Simpson’s paradox actually is
Simpson’s paradox is when a pattern that shows up inside every separate group of your data reverses, or simply vanishes, the moment you pool those groups into one combined number.
The new page wins in the total but loses in each segment. A treatment helps every kind of patient yet looks useless overall. The direction of the story depends entirely on whether you look at the whole or the parts, and the two are allowed to point flatly opposite ways.
A checkout test that lied
Let me make it concrete. Treat every figure here as illustration, not a benchmark to copy. Imagine your old checkout page had been grinding away for months, shown mostly to cold traffic from paid ads. Then you launched the new page during a big email push, so almost everyone who saw it was one of your warm, ready to buy subscribers.
| Traffic source | Old page | New page | Segment winner |
|---|---|---|---|
| Email (warm) | 22 in 100 | 20 in 100 | Old |
| Ads (cold) | 3 in 100 | 2 in 100 | Old |
| Overall (pooled) | lower | higher | New |
The old page was better with warm buyers and better with cold ones. And yet, because the new page was shown almost entirely to that high converting email crowd, its overall conversion looked dramatically higher. You cheerfully crowned the loser.
Why it happens: the lurking third variable
This feels like magic the first time you meet it, but it is just arithmetic with something hidden inside. Two things have to be true at once.
There must be a lurking third variable that genuinely affects your outcome, and your groups have to be lopsided in how much of that variable each one carries. In the checkout example, the third variable is the traffic source, or really the buying intent behind it. It drives the outcome hard, because warm email visitors convert far better than cold ad clicks no matter which page they land on. And it was split unevenly across the two pages.
Any time a variable both moves the needle and sits lopsided across the groups you are comparing, it can bend your totals until they lie. Statisticians call that variable a confounder, and it is the engine humming underneath the whole paradox.
Here is the intuition to keep: a combined rate is really a weighted average of the separate group rates, and the weights are simply how many observations landed in each group. When the weights match across the things you are comparing, the total behaves. When the weights are skewed, one group can dominate the pooled figure so completely that it drowns out the truth living in the parts.
The famous real cases
This is not a made up curiosity for textbooks. The most famous example is a university admissions study from decades ago, where the school overall appeared to admit men at a higher rate than women, which understandably looked alarming. But when researchers broke it down department by department, most departments actually favoured women slightly. The catch was that women had applied in far greater numbers to the most competitive departments, the ones that admitted very few of anybody. The total told one story, the parts told the opposite, and the parts were the honest ones.
The same reversal turns up in medicine, where a treatment can look worse overall and yet prove better for both the mild cases and the severe cases taken separately, simply because it was given more often to the sicker patients who fare worse no matter what. I will not put invented figures on these, because the point does not need them. The shape is the whole lesson: a clean, repeatable pattern where pooling the groups quietly flips the very conclusion you were about to act on.
Why it matters for your dashboards
Your dashboards aggregate almost everything by default. That single headline conversion rate, that one overall churn number, that average star rating across all your products, every one of them is a pooled figure that could be hiding, or even inverting, a story living at the segment level.
The moment you make a decision straight off a total, you are trusting that no confounder is quietly steering it. Simpson’s paradox is the reason that trust is sometimes badly misplaced.
How to catch it
The habit is easy to say: always look at your important comparisons both pooled and split, then reconcile the two. Before you trust any total, break it down by the obvious suspects, the traffic source, the device, the customer type, the cohort, the time period, and see whether the segment story still agrees with the headline. When the split flatly disagrees with the whole, treat that as a flashing warning light, not a rounding quirk.
In practice this is less work than it sounds, and a plain pivot table does almost all of it. Put the outcome you care about down one side, break it out by your suspected confounder along the other, and read each group’s rate sitting next to the pooled rate at the bottom. If that bottom line points one way while every row above it points the other, you have caught a live reversal and you cannot trust the total.
Which view is right, the total or the parts?
Here is the part most people get wrong: the segments are not automatically the truth and the total automatically the lie. Which view you should trust depends on whether the variable you split by is a genuine cause of the outcome, a real confounder, or just an irrelevant slice. When the split is on something that truly drives the result and sits unevenly across your groups, the segments hold the honest answer. When you slice on pure noise, you can manufacture a fake reversal out of nothing.
And you genuinely can torture your data this way, so be careful. Chop your numbers into ever smaller buckets and random wobble alone will eventually hand you a reversal in some tiny corner, a subgroup of eleven people where the pattern happens to flip. That is not insight, it is noise wearing a costume. Segment on the two or three variables you have a real reason to believe matter, and resist the urge to keep dicing until the data finally tells you what you were hoping to hear.
Mix shift, the everyday cousin
Even when you get no full reversal, the milder cousin of this effect will visit you constantly, and it is worth naming: mix shift. Your overall average order value can fall in a month when not a single product changed its price, simply because your cheaper products made up a bigger share of the sales this time. The total moved, but no segment did, only the blend between them.
So whenever a headline number surprises you, ask the two part question: did the groups themselves change, or did only the mix between them change? It is the same idea that makes correlation so easy to mistake for causation, and a close relative of how an average quietly stops being typical the moment your data is skewed.
The honest limit
Simpson’s paradox does not tell you which view is correct. It only warns you that the total and the parts can disagree, and that aggregation on its own can mislead. It will not name your confounder for you, it will not prove which split is the real cause, and it will not make your decision.
All it really buys you is the reflex to stop trusting a headline number until you have honestly tried to break it, which is worth far more than a confident total that reversed the whole time. Every metric and method like this one is explained in plain English at dataresearchanalysiscollection.com, so you can read it again slowly with your own dashboard open. No hype, no promises about your results, just the idea explained until you can start breaking your own totals apart.
Get new guides and videos first — join the Telegram channel.