Comparing Two Groups That Were Never Comparable

I once spent an afternoon convinced that customers who used a discount code churned less than customers who paid full price. The spreadsheet said so. I was ready to build a whole retention strategy around handing out more coupons.

Then I asked myself a simple question: who actually redeems a coupon code in the first place? People who were already searching for a reason to sign up, who read the fine print, who were more engaged before they ever hit the checkout page. The coupon didn’t make them stickier. They were already stickier, and the coupon was just something that happened to them along the way. Comparing “coupon users” to “everyone else” wasn’t comparing two similar groups that got different treatment. It was comparing two groups that were never comparable to start with.

That’s selection bias, and if you run your business on your own numbers, it’s probably already living in three or four reports you trust.

What selection bias actually is

Selection bias shows up whenever the way a group ended up in your dataset is connected to the outcome you’re measuring. It’s not a data entry error. It’s not a bug in your dashboard. The numbers are all correct. The problem is that the two groups you’re lining up side by side were sorted by something other than random chance, and that something also affects the number you care about.

Here’s the plain-English version: if group membership and outcome share a common cause, comparing the groups tells you about that common cause, not about whatever you think you’re testing.

In the coupon example, “engagement level” was the common cause. It pushed people toward using the coupon and it also pushed them toward staying subscribed. The coupon itself might have done nothing.

Where it hides in the metrics you already track

Churn cohorts. A classic move is pulling your churned customers and your retained customers and comparing their behavior in the weeks before the split, like login frequency or feature usage. The problem is that low usage might be the reason they churned, or it might just be a symptom of the same underlying thing that made them cancel, like a mismatch between what they needed and what you sell. If you only ever looked at existing customers to build your ideal customer profile, you’ve also silently excluded everyone who bounced off your pricing page or trial before becoming a customer at all. Your “typical customer” picture is built entirely from people who already survived a filter.

A/B tests with self-selected participation. If your test only counts people who actually completed the flow, and one variant is more confusing or has more friction, you’ve quietly filtered out different people in each arm. Say variant B has an extra form field. Some visitors who would have converted anyway just don’t bother filling it in and drop off before they’re ever counted as a completed session in either group. Now you’re comparing “everyone who saw A” to “the more patient subset who saw B.” That’s not a fair test of the form field. It’s a comparison of two differently filtered audiences.

Cohort comparisons across acquisition channels. Comparing customers who came from paid ads to customers who came from organic search or referral is comparing groups that were never on equal footing. Paid traffic often includes people who clicked on impulse. Referral traffic comes pre-vetted by a friend’s recommendation. If referral customers have a longer average lifetime, that could be the channel doing something magic, or it could just be that referred customers already had a reason to trust you before they landed on your site. Same output metric, completely different starting point.

Before and after a price change. If you raise your price and watch churn for the next month, and it doesn’t move much, it’s tempting to conclude the price increase was safe. But the customers still around when you raised the price are, by definition, the ones who hadn’t already left for other reasons. You’re watching a group that already survived one round of natural attrition, not a fresh, representative slice of everyone who would ever consider your product at the new price.

Survey and review data. People who leave reviews or answer your NPS survey are not a random sample of your customer base. They tend to be either unusually happy or unusually annoyed. If you use survey results to represent “what customers think,” you’re really hearing from whoever felt strongly enough to respond, which is a different group than whoever’s quietly using or quietly leaving your product without saying a word.

Why this matters more for a one-person business

A big company with a data team might run a proper randomized experiment to check whether a pattern holds up. Most solo operators don’t have the traffic volume, the time, or frankly the need to build that kind of infrastructure for every decision. That’s fine. But it means you’re leaning more heavily on comparisons between naturally occurring groups, like “customers who upgraded” versus “customers who didn’t,” and those comparisons are exactly where selection bias lives.

The fix isn’t to stop looking at your data. It’s to ask one extra question before you act on a comparison: how did each group end up in its bucket, and could that sorting process be connected to the outcome I’m measuring?

A few questions that catch it early

Before you draw a conclusion from two groups, walk through these:

  • Did people choose to be in one group or the other, or were they assigned? Chosen groups almost always carry some hidden reason for the choice.
  • Could the thing that put someone in group A also independently affect the outcome I’m measuring? If yes, you likely have a confound, not a clean comparison.
  • Am I only looking at people who made it through some earlier filter, like completing checkout, replying to a survey, or staying a customer long enough to show up in this quarter’s cohort? If so, what happened to everyone who didn’t make it through?
  • If I ran this exact comparison on a random split instead of the natural grouping, would I expect the same difference? If your gut says no, that’s a signal the groups were never equivalent.

None of this means every non-random comparison is useless. It means treat it as a hypothesis, not a verdict. “Coupon users seem to churn less, worth digging into why” is a reasonable note to keep in your back pocket. “Coupons reduce churn, so we should hand them out more” is a leap the data didn’t actually support.

What actually helps

Where you can, randomize. Even a rough 50/50 split on something like which onboarding email a new signup gets is far more trustworthy than comparing whoever happened to open an email against whoever didn’t, because the open itself is already a selection filter.

Where you can’t randomize, at least look for what’s different about the two groups besides the thing you’re testing. If paid and referral customers also differ by average order size, company size, or the month they signed up, you’re not looking at a clean two-variable comparison anymore, and any single number you report should come with that caveat attached.

And when you’re not sure, say so. A metric that comes with “this comparison has a likely confound, treat it as a lead not a conclusion” is more useful to future you than a confident number built on a shaky comparison.

Selection bias doesn’t announce itself. The spreadsheet looks clean, the numbers add up, and the story it tells is usually the one you were hoping to hear. That’s exactly why it’s worth pausing on the comparisons that matter most before you build a strategy on top of them.

If you want more explainers like this on the metrics and methods behind running a business on your own numbers, you can find the rest of them on the home page.

Get new guides and videos first — join the Telegram channel.