Metrics that only work at scale

Five percent monthly churn is the figure most articles will tell you is the edge of trouble. At forty two customers, five percent is two people, and one of mine had emailed in January to say he was closing his shop. I still spent a Sunday afternoon writing a careful theory about my onboarding to account for the movement.

That is the whole problem in one paragraph. The metric did exactly what it was built to do. I was the one asking it a question it had no way of answering.

Every metric has a resolution

A percentage can only move in steps the size of one unit of its denominator. With forty customers, one cancellation is two and a half percentage points, so those are the only steps on offer.

Customers Smallest possible move in monthly churn Same single cancellation, annualised
40 2.5 points about 26 percent
400 0.25 points about 3 percent
4,000 0.025 points about 0.3 percent

Read the top row twice. One person deciding to stop paying you moves your implied annual churn by twenty six points, and nothing else about the business has to change for that to happen.

At four thousand customers the identical event is a rounding error. Which is precisely why churn is one of the best early warnings a larger company owns: when that number moves, something moved it. The metric is fine. It is the denominator underneath you that decides whether it can speak.

Churn has about four possible readings

At forty customers your monthly churn rate can be zero, two and a half, five, or seven and a half percent. There is nothing in between, because half a person cannot cancel.

Now hold that up against the benchmarks. Five percent a month is considered healthy for a cheap consumer subscription. Eight percent is considered a fire. At your size the distance between healthy and fire is one customer, who may have moved country for reasons that have nothing to do with you.

So the month to month change carries no information about your business. It tells you which week somebody got around to clicking cancel.

Here is the position I will defend: a founder with forty customers reading a churn dashboard every week is doing astrology with extra steps. Most weeks the reading is zero, which feels like health. Then one week it is enormous, which feels like collapse. Both readings came from the calendar.

Lifetime value divides by the noise

The standard formula is average monthly revenue per customer divided by monthly churn. Look at what that asks for. It takes the jumpiest number you own and puts it in the denominator.

Say your product is nineteen dollars a month. Nineteen divided by two and a half percent gives a lifetime value of seven hundred and sixty dollars. Nineteen divided by five percent gives three hundred and eighty.

One cancellation, and the figure halves.

Underneath the arithmetic there is a worse problem. Lifetime value needs a lifetime, and if your oldest customer signed up fourteen months ago, you have not observed one. You have observed fourteen months and then extrapolated forever. That is a projection wearing a measurement’s clothes, and it turns up with a dollar sign and two decimal places attached, and then gets spent on the most expensive decision in the business: how much you are willing to pay to acquire someone.

I have watched an advertising budget get set from a lifetime value built on eleven months of history and four cancellations. On the dashboard it looked identical to one built from a hundred thousand customers, because nothing in any analytics tool tells you how many rows went into a figure.

Cohort tables at forty customers

A cohort table is one of the genuinely good ideas in analytics. Group everyone who signed up in the same month, follow that group forward, and you can see whether June’s customers behave differently from March’s.

Then you build one with forty people. January has five. February has four. March has nine, because you got mentioned somewhere.

A five person cohort can only report retention in fifths. A hundred percent, eighty, sixty, forty, twenty, zero. One person moving house is a twenty point drop, and with the colour shading turned on it renders as a cliff.

You will explain the cliff. That is the dangerous part, not the wobble itself. You will remember that you changed the pricing page that month and it will feel like a finding.

There is a separate trap where a cohort table appears to improve because of who is left in it rather than anything you did, and that deserves its own read. This one is cruder. A grid of forty people looks like analysis because it is shaped like analysis.

Conversion rate on small traffic

Sixty people reach your checkout in a week and four of them buy. That is 6.7 percent. The next week six buy out of sixty two, which is 9.7 percent, and your dashboard will render the difference as a forty five percent improvement in green with an arrow on it.

Two people.

What makes this one expensive is that a conversion swing always arrives with a candidate cause already attached, because you always changed something last week. The number hands you a story and you take it, and by Friday you believe something about your checkout that no evidence supports.

Why a misleading number beats no number, in the worst way

A blank cell sends you off to find out. A filled one lets you decide.

That asymmetry is the actual damage. Imprecision you can discount, because a number you know is rough gets treated as rough. Confidence you act on. And these figures arrive in the same font, the same colour, with the same trend arrow as a number computed from four thousand customers. Nothing on the screen distinguishes them.

The distinction worth keeping straight

None of these are vanity metrics. A follower count is meaningless at forty customers and equally meaningless at four million, because it never changes a decision at any size. Churn, lifetime value, cohorts and conversion rate are real metrics that earn their keep later. They are being asked to do work their sample cannot support yet.

This is also narrower than the general question of how much data you need before a number means anything, which has no fixed answer and depends entirely on the size of the gap you are hunting. This one does come with a list, and the list is short.

What I read instead at this size

Counts rather than rates. Write down “two people cancelled”, not “five percent monthly churn”. The sentence is shorter, it is truer, and it does the one thing a percentage refuses to do, which is make you ask who. A percentage is a way of avoiding your customers.

Longer windows. A quarter holds more cancellations than a month, so the figure has slightly more shape, and a rolling twelve months opened twice a year has more again. You lose nothing by waiting, because there was never an action you were going to take on the March number.

Then the part that actually works, which is not a metric at all. At forty customers the entire denominator fits on one screen. You can email every person who cancelled this quarter, and this quarter that is four emails. Perhaps two reply. Two replies with reasons in them beat any rate you can compute, because a rate never tells you why. It is a smoke alarm at best, and at this size it goes off when somebody makes toast.

We build the dashboard instead because the dashboard is comfortable. Asking somebody why they stopped paying you is an unpleasant thing to do on a Wednesday morning, and a chart has never once said anything unkind to me.

And keep a short list of things that cannot be divided by anything. Money in the bank at month end. How many people pay you. How many new ones arrived. No small denominator can inflate a count.

Where the line sits

I cannot tell you where these numbers stop being noise. Somewhere between forty customers and a few hundred they turn readable, and I have no defensible threshold to hand over. I have seen a hundred asserted, and two hundred, and a thousand, in the same confident tone, and none of them arrived with a reason.

So keep computing all four. Fix each definition once and leave it alone, and let the history accumulate, because in three years you will want the early rows and there is no going back to create them. Keep the instrument and ignore the dial.

I still compute my own churn every month, and I still feel something when it goes up. Twice this year I have caught myself halfway into explaining a move that was one person leaving for reasons unrelated to me. Understanding the trap did not stop me falling into it. What stopped me was deleting the chart and replacing it with a line that reads how many people cancelled, followed by their names, because it turns out to be very hard to theorise about onboarding while looking at a name and already knowing the man closed his shop. The rest of the metrics I keep, and the ones I stopped opening, are written up in plain English at dataresearchanalysiscollection.com.

Get new guides and videos first — join the Telegram channel.