The bug that hides inside a working test
Here’s a specific way an A/B test can go wrong that has nothing to do with your headline, your button color, or your pricing. The test runs fine. Data comes in. You get a result. And the result is garbage, because the two groups you were comparing were never actually equal to begin with.
This is called a sample ratio mismatch, usually shortened to SRM. It means the split between your test groups came out meaningfully different from what you set up. You told the system to send 50% of visitors to variant A and 50% to variant B. Instead you got something like 47/53, or worse, 42/58. Nobody changed the settings. The traffic just didn’t land where it was supposed to.
If you run split tests for your own business, this is worth understanding even if you’ve never heard the term, because it’s one of the few things that can make a “winning” variant look like a winner for reasons that have nothing to do with the variant itself.
Why an uneven split is a real problem, not just a cosmetic one
A/B testing works because randomization is doing a specific job: making sure the two groups are alike in every way except the one thing you’re testing. Same mix of new and returning visitors, same mix of devices, same mix of traffic sources, same time-of-day pattern. When the split is genuinely random, any difference in outcomes between the groups is something you can attribute to the variant, within the limits of your sample size.
An uneven split doesn’t just mean one group has fewer people in it. It means something in the assignment process is systematically letting some visitors into one bucket and not the other, and that “something” is very often correlated with the exact traits that affect the outcome you’re measuring. If the assignment mechanism drops a chunk of slow-loading mobile sessions before they get tagged into variant B, you haven’t just lost some sample size in B. You’ve made B’s remaining visitors skew toward faster connections, certain browsers, or certain devices. Your conversion rate difference at the end might be measuring device mix, not the thing you changed.
That’s the trap. The test still produces a number. The dashboard still shows a “winner.” Nothing about the output looks broken. The damage is invisible unless you specifically check for it.
How assignment actually breaks in practice
I’ve seen a handful of ways this happens on small setups, and none of them require anyone to make an obvious mistake:
Redirect-based tests lose people asymmetrically. If variant B lives on a different URL and visitors get redirected there, some percentage will bail out during the redirect, close the tab, or get slowed down by a script that only loads on that path. Variant A, served directly with no redirect, doesn’t lose that slice of people. Now A’s sample is padded with visitors who would have bounced from B anyway had they experienced the same friction.
The tracking script fires before the assignment does, or after. If your analytics tag records a pageview before the split test script has finished bucketing the visitor, you can end up logging visits that never got properly counted into either arm, or that got double-counted. Timing mismatches between two separate tools are a common source of this.
One variant is heavier and triggers more bounces or ad blocker interference. A variant with more scripts, more images, or a third-party embed is more likely to get partially blocked, especially by ad blockers or privacy extensions. If that blocking also prevents the assignment cookie from being set correctly, those visitors either don’t get counted or get miscounted.
Bot and crawler traffic doesn’t split evenly. Bots often hit whichever variant is linked first, cached, or easiest to crawl, rather than getting randomly assigned like a human visitor would. If your filtering doesn’t catch them, they pad one side of the split.
Caching serves a stale bucket assignment. If a CDN or browser cache serves an old version of a page, or a cached response includes a stale cookie, users can get funneled into a group that doesn’t match your intended random assignment ratio.
None of these require broken code in the sense of an error message. They’re all quiet failures that a normal QA pass on the variant itself won’t catch, because the variant works. It’s the assignment layer that’s off.
Checking for it without a statistics degree
You don’t need to become a statistician to catch this. The basic idea is simple: compare the number of visitors you actually got in each group against the number you’d expect given your intended split, and ask whether the gap is bigger than plain randomness would explain.
If you set up a 50/50 test and got exactly 50/50 every single day, that would actually be a little suspicious, because pure randomness naturally produces small day-to-day wobble. The question isn’t “is it exactly even.” It’s “is the imbalance small enough to be normal noise, or large enough that something structural is pulling traffic to one side.”
The formal tool for this is a chi-square test, which takes your observed counts and your expected counts and tells you how likely it is that a gap that size happened by chance. You don’t need to run this by hand. Most experimentation platforms worth using will flag an SRM automatically, or you can drop your counts into any free chi-square calculator and get a p-value back. A very small p-value, the kind under about 0.01, is the standard trigger point people use to say “this split test has an assignment problem.” That threshold is intentionally stricter than the usual 0.05 you might use for the test result itself, because you want to be conservative about declaring the underlying test broken.
The practical version for a solo operator: pull your daily visitor counts by variant, watch the running total, and if you’re consistently seeing a split like 45/55 instead of your intended 50/50 across a decent sample, stop and dig in before you trust anything else the test tells you. A gap that shows up on day one and shrinks as more data comes in is probably noise. A gap that persists or grows is not.
What to do once you find one
The instinct is to want to fix the numbers after the fact, maybe by reweighting the results to account for the imbalance. Don’t. If the imbalance came from a mechanism that also correlates with the outcome, like slower connections dropping out of one variant, there’s no clean way to reweight your way back to a valid comparison. You don’t know what you don’t know about who got excluded and why.
The right move is to pause the test and go find the mechanical cause. Check whether both variants load with the same number of scripts and third-party calls. Check your redirect setup if one variant lives on a separate URL. Check whether your analytics tag and your testing tool are firing in a consistent order on both variants. Check whether bot traffic is being filtered the same way for both. Check your caching rules, especially if you’re on a CDN that might serve a stale version of one variant to some visitors.
Once you find and fix the cause, restart the test with a fresh assignment window rather than trying to patch the existing data. It costs you time, but a test built on bad assignment isn’t just weaker evidence, it can actively point you toward the wrong decision, which is worse than no test at all.
Make the check a habit, not a one-off
The cheapest version of this discipline is just building the split ratio check into whatever cadence you already use to look at a live test. If you’re already glancing at conversion numbers every few days, glance at the raw visitor counts per group at the same time and eyeball whether they’re roughly where they should be. Catching an assignment problem on day three costs you a few days of test time. Catching it after you’ve already shipped the “winning” variant to everyone costs you a lot more, and you may never know it happened at all.
If you want more explainers like this on the methods behind the metrics small businesses actually run on, you can find the rest of them on the Data Research Analysis Collection home page.
Get new guides and videos first — join the Telegram channel.