Why this matters if you run your own numbers
If you’ve ever run an A/B test on a checkout page, a subject line, or a pricing page, you’ve probably seen the phrase “statistically significant” attached to a p value. Most tools report it as a green light: under 0.05, ship it. Over 0.05, don’t.
I use p values in my own testing, and I think they’re useful. I also think most of the explanations out there make them sound more certain than they are. A p value is a narrow, specific number that answers a narrow, specific question. It does not tell you your idea worked. It does not tell you how much money you’ll make. It does not even tell you the effect is real in the way most people assume. Knowing what it actually measures, and what it stays silent on, is what keeps a solo operator from making a bad call with a straight face.
What a p value actually is
Say you’re testing two versions of a landing page headline, A and B. You run the test, B ends up converting higher than A, and you get a p value of 0.03 for that difference.
Here’s the actual definition: the p value is the probability of seeing a difference at least as large as the one you observed, if there were truly no difference between A and B and everything you saw was just noise from who happened to land on your page that week.
That’s it. It’s not the probability that B is better. It’s not the probability that your result is a fluke. It’s a statement about how surprising your data would be under the assumption that nothing is actually different between the two versions, which statisticians call the “null hypothesis.”
A small p value means: if A and B were truly identical, a gap this big would be unusual to see by chance alone. It doesn’t mean B is definitely better. It means the pattern in this sample is hard to explain by chance alone, assuming your test was set up cleanly, which is a assumption worth checking, not a given.
It does not tell you how big the effect is
This is the one that trips up small business owners the most, because it’s the one they actually care about.
A p value of 0.001 does not mean a huge effect. A p value of 0.04 does not mean a small one. The p value is about confidence in the existence of a difference, not the size of that difference. You can get a very small p value from a very small, practically irrelevant lift if you have a lot of data. And you can get a shaky p value from a genuinely large lift if you only had a handful of conversions to work with.
If B converts at 3.1% versus A’s 3.0%, and you ran the test long enough on high traffic, that tiny gap can produce a “significant” p value. It’s real in the statistical sense. It’s also not worth rebuilding your page over. Before you act on a result, look at the actual size of the lift in dollars or conversions, not just whether the p value cleared the bar. The p value tells you whether to trust that a gap exists. It doesn’t tell you whether the gap is worth caring about.
It does not tell you the reason behind the number
A significant p value tells you something moved. It does not tell you why.
Say your email open rate jumps and the p value says the change is unlikely to be chance. Was it the new subject line? Or did you send at a different time of day, to a slightly different segment, during a week when a competitor had an outage? A/B tests only isolate the variable you controlled for if everything else was actually held equal. In a one-person operation, “held equal” is harder than it sounds. You’re often running tests in real market conditions, not a lab, and other things move at the same time your test is running.
The p value can’t detect a confound. It just measures whether the pattern in the data is unlikely under “nothing changed.” If something else changed too, the p value has no way to know and no way to warn you.
It does not fix a bad test setup
A p value is only as trustworthy as the test that produced it. A few ways I’ve seen this go wrong, in my own work and in tools I’ve used:
Peeking early. If you check your test results daily and stop the moment you see a “significant” result, you’re not running the test the statistics assume you’re running. Checking repeatedly and stopping at the first green light inflates your odds of a false positive well past whatever threshold you think you’re using. If you’re going to check early, use a testing method built for that, and treat an early peek as informational, not a stopping signal.
Small sample sizes. Ten conversions on A and thirteen on B can produce a p value, but it’s built on almost nothing. A single big order or a single referral spike can swing a small sample enough to change the whole picture.
Testing too many things at once. If you test five headlines against a control, you’d expect roughly one of them to look “significant” at the 0.05 threshold purely by chance, even if none of them actually did anything. This is sometimes called the multiple comparisons problem. The more variants or metrics you test in one pass, the more you need to correct for it, or the more skeptical you should be of any single result.
0.05 is a convention, not a law
The 0.05 threshold gets treated like a line in the sand between “true” and “false.” It isn’t. It’s a convention that became standard in academic research decades ago, and it carries no special meaning for your business. A result at p = 0.06 isn’t meaningfully less real than one at p = 0.04. Both are estimates of how surprising your data would be under the null hypothesis, and both come with real uncertainty attached.
For a small operation, what actually matters is the decision on the other side of the number: how much does it cost to be wrong, and how much does it cost to keep testing longer to be more sure? A pricing change you can’t easily reverse deserves more caution than a subject line you can change again next week.
What I actually do with a p value
I treat a p value as one input, not a verdict. Before I act on a test result I ask: how big is the actual lift, in real numbers I care about, not just percent. Did I let the test run long enough, and did I avoid stopping the moment I liked what I saw. Is there anything else that changed at the same time that could explain the result. And is the decision easy to reverse if I’m wrong.
A p value is good at one job: telling you whether a pattern in your data is more than you’d expect from noise, given your assumptions hold. It was never built to tell you the size of the win, the reason behind it, or whether your test was clean. That’s still on you to check.
If you want more explainers like this, written from the position of someone actually running the tests on his own businesses, you can find the rest on the home page.
Get new guides and videos first — join the Telegram channel.