The problem with only having one business
A/B testing works because you split traffic in half at the same time and let everything else stay equal. Same week, same season, same ad spend, same news cycle. The only difference between your two groups is the thing you changed. That is what makes the comparison clean.
Most of the changes a solo operator makes cannot be split tested that way. You cannot A/B test a new pricing page against your old one if you only have one pricing page and switching it over is a one-way door. You cannot split test raising your prices, because a customer who sees the old price and a customer who sees the new price at the same time is going to notice, and now you have a different problem. You cannot split test switching your entire onboarding email sequence, or moving to a new payment processor, or changing your refund policy. Some changes are structural. You flip a switch and everyone is on the new thing.
So you are left with a before and after comparison. You look at the metric before the change, you look at it after, and you try to figure out if the change did anything. This is weaker evidence than a proper test. It is also the only evidence you are going to get for a lot of real decisions, so it is worth learning how to do it without fooling yourself.
Why before and after is a weaker kind of evidence
An A/B test controls for time. A before and after comparison does not. Everything that happens to your business also happens to your metric during the after period, not just your change. If your MRR grows in the month after you raised prices, you do not automatically know the price increase caused the growth. Maybe you also picked up a backlink that sent new traffic. Maybe a competitor shut down. Maybe it is just a seasonally strong month, the way December is stronger than February for a lot of consumer products regardless of anything you do.
This is the core weakness, and no amount of clever analysis erases it. What you can do is reduce how much it fools you.
Give yourself a real baseline, not a guess
Before you make the change, write down what the metric was actually doing on its own, not what you assume it was doing. Pull the trend for the prior few months, not just the single number from last week. A metric that was already climbing 5% a month before your change is a different starting point than a metric that was flat.
This matters because a lot of solo operators eyeball “last month vs this month” and call that their baseline. If your churn was 4% in month one, 5% in month two, and 6% in month three, and then you ship a change and month four comes in at 6.5%, you did not cause a spike. You were already on a rising line and the line kept rising, maybe a little slower than it would have. Without the trend, that same 6.5% looks alarming in isolation.
Use a comparable period, not just “before”
Pick your after window to match your before window as closely as you can. If you are comparing conversion rate, compare the same number of days, ideally covering the same days of the week, since weekday and weekend traffic often converts differently. If your business has a seasonal pattern, like a tax prep tool that gets busy in March or a gift business that spikes in December, compare against the same point in a prior cycle if you have that data, not just the four weeks right before the change.
If you genuinely have no seasonal history to lean on, be honest with yourself that your after window is contaminated by whatever normal week-to-week noise your business has, and size your confidence accordingly. A three day comparison after a pricing change tells you almost nothing. A six week comparison tells you more, though still not as much as a real test would.
Watch for the other things that moved at the same time
Before you credit your change, list what else happened in the after period. Did you also post more on your channel that month? Did a piece of content start ranking? Did you run any outreach or backlink work in that window? Did a payment processor outage or a site slowdown touch the numbers on one side and not the other? This is not about being paranoid, it is about writing down the confounders so you can weigh them instead of ignoring them.
This is the same discipline attribution work forces on you when you are trying to figure out which channel actually drove a signup. The honest answer is often “some mix, and I cannot cleanly separate it,” and that is a more useful answer than a false single cause.
Segment the comparison instead of trusting the top line
One thing you can do that a raw before and after cannot do on its own is break the after period into cohorts and see if the story holds up inside the data, not just across the whole average. If you changed your onboarding flow, look at the cohort of users who signed up after the change and follow their retention curve on its own terms, rather than blending them into last month’s aggregate churn number. A cohort view will not fix the lack of a control group, but it stops a big swing in one part of your business from hiding inside a flat looking overall number, and it stops a good week from a different part of the business from covering for a change that actually made things worse.
Funnel data works the same way. If you changed your checkout page, do not just watch overall conversion rate. Watch each step of the funnel separately. If sign ups to cart stayed the same but cart to paid dropped, that tells you where the change actually bit, in a way the single top line number cannot.
What to do when the result is ambiguous
Most before and after comparisons for a small operation land in a gray zone. The metric moved, but not by enough, or not clearly enough, to say the change caused it with any confidence. This is the normal outcome, not a failure of your analysis. The honest move is to say so, rather than pick a story that flatters the change you already made.
If the metric moved a lot and moved in the direction you wanted, and you cannot find an obvious confounder that explains it, that is reasonable evidence, not proof. Treat it as a hypothesis that got a little stronger, and look for it to hold up over the following month too. If it fades, the original move might have been noise, or a short-lived reaction from existing customers rather than a lasting shift.
If you can eventually run a real A/B test on part of the change, even a smaller piece of it, that is worth doing to firm up what the before and after comparison could only suggest. Not every change allows for that, but some do. A new headline on a landing page can often be tested even when a full pricing overhaul cannot.
The honest limits
A before and after comparison cannot rule out coincidence. It cannot separate your change from every other thing happening to your business in the same window. It cannot tell you what would have happened if you had not made the change, because that version of events never existed. What it can do, if you build a real baseline, match your comparison period, track the confounders, and segment the data instead of trusting one blended number, is give you a reasonable read on whether a change is worth keeping, reversing, or watching longer. That is a lower bar than a controlled test, and it is still worth clearing carefully rather than skipping.
If you want more on how to build baselines, cohorts, and funnels you can actually trust for decisions like this, you can find the rest of the breakdowns on the site home page.
Get new guides and videos first — join the Telegram channel.