Setting an alert that does not cry wolf

What crying wolf actually costs you

I set up my first alert years ago on a whim. Churn goes above 5%, send me a text. Within two weeks I’d muted it. My churn number bounced around anywhere from 2% to 6% week to week just because I had a small subscriber base, so the alert fired constantly and told me nothing I didn’t already know from glancing at the dashboard. Once I muted it, I missed a real churn spike three months later because I wasn’t looking at the number at all anymore.

That’s the whole problem with a badly set metric alert threshold. It’s not that alerts are a bad idea. It’s that a threshold set without thinking about how the metric actually behaves either fires too often and trains you to ignore it, or sits so loose that it never fires until the damage is done. Either way, you’ve built something that feels like safety and isn’t.

What a metric alert threshold is doing under the hood

Strip away the dashboard and an alert is just a rule: if this number crosses that line, notify someone. The line is the threshold. Everything about whether the alert is useful comes down to how that line was chosen and what counts as “crossing” it.

Most people pick the line the same way I did with churn: they guess a number that sounds bad, type it in, and move on. That works fine for metrics that barely move, like your business registration status. It falls apart for anything with natural variance, which is most of the metrics you actually care about: daily conversion rate, weekly signups, MRR growth, cart abandonment.

The most common way thresholds go wrong

The mistake is treating the threshold as if it should predict the future. It shouldn’t. It should be tuned to what “normal” looks like for your specific business, at its current size, so that a crossing genuinely means something changed.

Say you run a small SaaS tool with forty active customers. Your conversion rate from trial to paid moves between 8% and 14% depending on the week, just from small-sample noise. If you set an alert at “conversion below 12%,” you’ll get paged most weeks for nothing. If you set it at “conversion below 5%,” you’ll never get paged until you’re already in real trouble, because your normal range never gets anywhere near 5% until something has clearly broken.

Neither number is wrong on its face. Both are wrong because they weren’t chosen by looking at how the metric actually moves for your business.

Look at the variance before you pick a number

Before setting any metric alert threshold, pull up several weeks or months of the actual number and look at the spread. Not the average, the spread. What’s the lowest it’s been on an ordinary week? What’s the highest? That range is your noise floor, and your threshold needs to sit outside of it, not inside it.

For a metric like CLV, which usually moves slowly because it’s built from a lot of underlying purchase data, the noise floor is narrow, so you can set a tighter threshold. For something like daily website conversion on a low-traffic funnel, the noise floor is wide because you’re often looking at ratios built on small counts, and small counts swing hard. A funnel with twenty visitors a day can jump from 5% to 15% conversion just because one extra person bought something. That’s not a real signal. That’s arithmetic.

Percentage change beats a fixed number, most of the time

A second issue: a fixed threshold ages badly as your business grows. “Alert if MRR drops by $500” made sense when your MRR was $3,000. It’s meaningless noise once you’re at $30,000, where a $500 wobble happens most weeks from normal billing timing, failed cards, and mid-cycle cancellations.

Where you can, tie the threshold to a percentage of a trailing average instead of a flat number. “Alert if MRR drops more than 8% below the trailing four-week average” scales with the business automatically. You still have to pick that 8%, and you still have to check it against your own variance, but at least it won’t quietly become useless as you grow.

This matters just as much for churn and conversion, both of which are already percentages but still benefit from being compared to a trailing average rather than a single static cutoff, since a trailing average absorbs some of the week-to-week noise for you.

Require the pattern to hold before you react

One more thing cuts down false alarms more than almost anything else: don’t alert on a single data point. Require the crossing to hold for more than one measurement before it fires.

If your daily conversion rate dips below your threshold on a Tuesday, that could be one slow day. If it’s still below threshold on Wednesday and Thursday too, that’s a pattern worth interrupting your day for. Most alerting tools let you set this as a condition, something like “threshold breached for three consecutive periods,” rather than firing the instant one number blips.

This is the single change that turned my own alerts from something I dreaded into something I trusted. It doesn’t eliminate false positives. It just means the false positives that do slip through usually resolve themselves before they generate a notification, because a genuine one-off blip rarely repeats three times in a row.

Two tiers beat one

Not every crossing deserves the same response. I run two tiers on the metrics I actually watch closely: a warning threshold that just logs itself somewhere I’ll see when I check in, and a critical threshold that interrupts me directly, by text or a loud notification.

Warning is “conversion is running below its usual range, worth a look this week.” Critical is “conversion has fallen off a cliff and stayed there, look now.” Splitting it this way means the noisy, more-sensitive threshold doesn’t cost you anything, because it just sits there as context for when you do check the metric, never demanding your attention. Only the rarer, tighter threshold gets to interrupt you.

Write down why, then revisit it

Whatever number you land on, write down why you picked it. Not for anyone else, for future you. “Set churn warning at 4.5% because trailing 8-week range has been 2-3.8%, and this gives room above the noise floor” is a sentence that lets you sanity-check the threshold again in six months. Without it, you’ll forget whether the number was ever grounded in anything, and you’ll either leave a stale threshold in place long after your business has outgrown it, or you’ll second-guess a perfectly good one and loosen it right before it would have caught something real.

Revisit thresholds on a schedule, not just when they annoy you. A business that doubles its customer base will usually see its metrics settle into a narrower relative range, since larger samples smooth out noise. That’s usually the moment to tighten a threshold that used to need to be loose.

What automation can and can’t do for you

If you’re using an automated tool or a bot to watch these numbers and ping you, it’s worth being honest about what it’s actually doing. It’s checking a rule you gave it against a number it’s pulling from somewhere. It doesn’t know that your Tuesday numbers always run low because that’s when your biggest customer’s invoice renews, and it doesn’t know that a churn spike last month was one customer’s card expiring rather than a trend. It will tell you a line was crossed. It will not tell you why, and it can’t tell you whether the crossing matters. That judgment call is still yours, every time.

A metric alert threshold, even a well-tuned one, is a prompt to go look at the number more closely, not a verdict on what’s happening in your business. Treat it that way and it earns your trust back. Treat it like an oracle and you’ll either drown in false alarms or get blindsided by the one that mattered.

If you want more of this kind of plain-English breakdown of the metrics and methods that actually run a small business, you can find the rest of what we’ve written over on the home page.

Get new guides and videos first — join the Telegram channel.