Two hundred and twelve on the Monday. Two hundred and fifty one on the Tuesday.
I spent parts of six weeks on that. Duplicate accounts. A payment retry reactivating a batch of lapsed subscriptions. The signup log, day by day, hunting for a source I had missed.
The answer was that eleven days earlier I had changed the query behind that number so free trials counted as active, because I was working out my support load and a trial user emails you exactly like a customer does. It shipped on a Tuesday. The chart stepped on the Tuesday.
Nothing had gone wrong. That is the part worth sitting with.
Definitions drift, they do not break
A metric definition never changes in one dramatic moment. It moves a little at a time, and every move has a good reason attached to it on the day you make it.
You take your own test accounts out of the customer list, because seeing yourself in there is irritating. You start counting trials as active for one specific question. You fix a tracking bug that had been dropping roughly one signup in twelve since February, which means every month before the fix is now understated and every month after it is not. You switch from counting orders to counting paying customers because two people order weekly and were quietly inflating the line. Your analytics tool changes what it considers a session in a release note you did not open.
Five changes. Four of them I would make again without hesitating.
The damage comes from all five of them landing in one line on one chart, because a chart is a comparison. Every point is read against the points to its left. Change what a point means and you have drawn a line between two different rulers.
It still slopes. It still looks like a trend. Some of that slope is your business and some of it is your own edit history, and nothing on the screen tells the two apart.
This is drift over time, not two tools disagreeing
Worth separating these, because they get treated as the same complaint.
Two tools disagreeing is the processor saying 38 sales and analytics saying 41 for the same week. Two numbers, both visible, sitting in two browser tabs. The problem announces itself. It is irritating enough that you go and deal with it, usually the same afternoon.
Drift inside one tool over time gives you one number. One series, one label, one line that looks continuous all the way back. A single number has nothing to argue with.
That asymmetry is the whole reason this one costs you six weeks instead of one afternoon.
The small business version is worse
Large companies have this problem and suffer less from it, for reasons that have nothing to do with talent.
A definition change there goes through a ticket, and the ticket has a date on it whether anyone wanted one or not. Somebody other than the author reads it. Most importantly the person who reads the chart is usually not the person who edited the query, so when the number moves, somebody who genuinely does not know the answer asks why out loud.
You have none of that. You are the author and the reviewer and the reader, asking yourself a question you already believe you know the answer to. There is also nobody to look silly in front of, which sounds like an advantage for about a week.
An undocumented definition makes your history a rumour
Here is the position I will actually defend.
If you cannot write in one sentence what a number counts, and name the date that sentence started being true, your historical chart is a rumour. Not false. A rumour: probably roughly right, sourced from somewhere you cannot check, and uncomfortable to stand behind if anyone presses.
And you will stand behind it. That is the awkward part. Growth figures go into investor updates. Retention numbers get quoted to partners on calls. You will read them off a chart whose left half and right half count different populations, with a completely straight face, because the chart looked fine.
Go and find the drift you have already caused
Before you set anything up, do the retrospective version. It takes about an hour.
Open your most important number at the widest date range you have and look for steps. Real growth is a slope. An edit is a step: a jump between two adjacent points that then holds at the new level. Once you spot one, date it, and go and find what you were doing that week. Your deploy history. The version history on the spreadsheet. The product update email your analytics vendor sent that you archived unread.
Doing this I found two things I had genuinely forgotten. One was the trial change. The other was a filter I had set on a saved report to exclude one country during a spam wave, left in place for nine months.
Then look for the harder shape, which is a ramp rather than a step. I started tagging orders with an acquisition channel in March, and coverage climbed over about six weeks as I backfilled the older ones by hand in the evenings. On the chart, direct traffic declined steadily across those six weeks and paid climbed to meet it. My marketing had done none of that. The line was describing how fast I was typing in the evenings, and a ramp convinces far more easily than a step, because it has the shape everyone is trained to trust.
The page, and the field everyone leaves off
One page per metric. Plain text, three things:
- what is counted
- what is excluded
- the date the current wording took effect
Mine is half a page covering six metrics. The third line is the one people skip and it does most of the work, because a definition without a date only describes today. It says nothing about the eighteen months behind it, which is where all your comparisons live.
When the definition changes, add a line rather than editing the old one. Active means a paid subscription in good standing. Underneath: from 4 March, includes free trials. Two dated lines, and the page has become a history you can read down instead of a snapshot.
The urge to overwrite is strong, because the new wording is better and the old one has been retired. The old wording is still the only honest reading of the period it covered, and that period is most of your chart.
Keep both running for a season
When you do change a definition, run the old one alongside the new one for a while. A quarter is what I use. Ninety days is plenty.
For that window you have both numbers on the same days, which means you can measure the gap rather than estimate it. Mine came out at 19 percent, steady across two months. That single figure is what makes the boundary crossable: take any number from last year, scale it, and you have something honest enough to reason with. Skip the overlap and everything before the change is denominated in a currency you no longer have a rate for.
The overlap tells you something else for free. If both lines move together, the change was cosmetic. If they pull apart, your two definitions disagree about what is happening in the business, and that disagreement is a real finding rather than an accounting artefact.
The annotation is what actually saves you
Now the habit that matters more than the page.
Write the change on the chart, at the date, where the step is.
A definitions file only helps you if you open the definitions file. At ten at night in six months, staring at a line that has done something strange, you will not open it. You will have forgotten it exists. What you will look at is the chart, because the chart is what made you suspicious.
So the note has to be there. A vertical line at 4 March with four words on it: trials now counted active. That is the entire intervention, and it is the difference between six weeks and six seconds.
Most dashboard tools have annotations or event markers buried in a menu somewhere. If yours has nothing, put the date in the chart title, or keep a short dated list beside your definitions. I annotate things that are not definition changes too: the week the payment provider had an outage, the day the price went up, the fortnight the signup form was broken in one browser and nobody told me.
What I did wrong once I found it
My first instinct on finding the 19 percent step was to make it go away, so I backfilled. Eighteen months of history recalculated under the new definition, one smooth line, no jump. Ten minutes of work and it felt like tidying.
What it did was orphan every number I had ever said out loud. A growth figure from a January email to a partner no longer existed anywhere in my own data. A churn rate I had quoted on a call became a different churn rate. Anyone going back through those would have found me contradicting myself half a dozen times with no explanation that did not sound like an excuse.
I still backfill. I keep the original series next to it now, labelled, and I never do it to a number that has already left the building.
Do not turn this into a process
The natural response is to build something. Resist it. No change request form, no monthly definitions meeting with yourself. Governance advice for one person businesses fails by producing a system too elaborate to survive week three.
One text file, and the habit of adding a line to it on the day you change something. An afternoon to set up, a minute per change afterwards. If even that is too much, do the annotations and skip the file. The mark on the chart is worth more than the page anyway.
The limit
Annotations only work if you read them, and I have scrolled past my own note at least twice.
I also document about half of what I should. Six metrics have a dated definition. The rest live inside queries, and the query is the definition, which sounds fine until you notice it is a claim I cannot check without reading two hundred lines of it.
None of this makes a number correct either. A carefully dated definition of a badly chosen metric buys you eighteen months of consistent, well documented nonsense. What it buys is narrower than that: the ability to compare this Tuesday to last Tuesday and mean it. More plain English writing on the numbers a one person business runs on is at dataresearchanalysiscollection.com.
Get new guides and videos first — join the Telegram channel.