Watching Ten Users Is Not A Dataset

Watching someone use your product feels like truth

I remember the first time I turned on a session recording tool for one of my sites. I watched maybe fifteen sessions in a row, one after another, and by the end I was convinced I knew exactly what was wrong with the signup flow. Two people had scrolled past the button. One person had rage-clicked on a link that didn’t go anywhere. I went and “fixed” it that afternoon.

Nothing changed. Conversion stayed flat for the next few weeks. What I’d actually done was watch a handful of specific people do specific things, then treat those specific things as if they were universal. That’s the trap. Session recordings are one of the most convincing tools in analytics precisely because you’re not looking at a chart, you’re watching a real person move their mouse and hesitate and click. It doesn’t feel like a sample. It feels like evidence. But ten people, or even fifty, is not a dataset in the sense that matters for decisions. It’s a set of stories.

What a session recording actually shows you

A session recording tool, whether it’s something like Hotjar, FullStory, or Microsoft Clarity, captures the events happening in one visitor’s browser: mouse movement, scrolling, clicks, form field focus, page changes. It stitches those events into a replay you can watch back. That’s genuinely useful. It shows you things a funnel report never will, like someone hovering over a price for eight seconds before leaving, or clicking on an image that isn’t actually a link.

What it doesn’t do is tell you how representative that person is. The recording has no opinion on whether the visitor was a returning customer, a first-time browser on mobile data, someone testing your site on a slow connection, or a bot that slipped past the filter. You bring all the context. And when you’re watching a recording, your brain fills in a story: “this person got confused here” or “this person almost bought and then didn’t.” Sometimes that story is right. Often it’s your own assumptions wearing the visitor’s face.

Why small numbers feel so convincing

There’s a reason watching ten sessions feels more persuasive than looking at a conversion rate of 2.3%. A percentage is abstract. A person scrolling past your call-to-action three times is concrete, and concrete things stick in memory far better than abstract ones. This isn’t a flaw specific to session recordings, it’s just how attention works. But it means the tool is quietly working against your judgment at the exact moment you’re trying to be objective.

The other issue is that when you go looking for a recording to explain a problem you already suspect exists, you’ll usually find one. If you believe your checkout page is confusing, you can watch twenty sessions and find at least two or three that look like confusion, because some fraction of any set of humans will hesitate, backtrack, or click the wrong thing regardless of your design. That’s not proof your theory is right. It’s confirmation bias with video evidence attached, which is a more convincing package than confirmation bias usually comes in.

The selection bias baked into the tool itself

Most recording tools sample. They don’t record every visitor, both for storage cost and because most plans cap the number of sessions captured per month. Which sessions get recorded, and which of those you personally choose to open and watch, are both filtered in ways you don’t fully control. If you’re skimming a session list, you’re more likely to click into ones with obviously long durations, or ones flagged with rage clicks, or ones from a device type you’re currently worried about. That’s a reasonable way to triage your time, but it means the sessions you actually watch are not a random slice of your traffic. They’re the ones that already looked interesting, which usually means they were already unusual.

This matters more for a small business than it does for a company with millions of monthly visitors, because at low traffic volumes the whole recorded pool might only be a few hundred sessions to begin with. Watching ten or twenty of those isn’t a sample of your users, it’s closer to a sample of a sample of a sample.

What a handful of recordings can actually tell you

None of this means session recordings are useless, and I still watch them regularly. What they’re good for is generating hypotheses, not confirming them. If you watch a stretch of recordings and notice three separate people clicking on your logo expecting it to reset a filter, or hesitating on the same form field before abandoning, that’s worth writing down as a question to check against your numbers. It’s not worth acting on by itself.

They’re also good for catching things your analytics can’t describe at all, like a broken button that only fails on one browser, or a form field that’s technically visible but overlapped by another element on certain screen widths. A funnel report will tell you conversion dropped on mobile Safari last week. It won’t tell you why. A recording might show you in thirty seconds. That combination, quantitative data pointing at where to look and qualitative recordings showing you what’s happening there, is where session recordings earn their keep.

Pairing recordings with something you can count

The fix for the “ten users isn’t a dataset” problem isn’t to stop watching recordings, it’s to never let a recording be the last step. If you notice a pattern in a handful of sessions, go check whether it shows up in the numbers that actually have enough volume to mean something: your funnel drop-off at that step, your bounce rate on that page, your support tickets mentioning that flow. If the pattern you saw in five recordings also shows up as a dip in a funnel report covering a few thousand sessions, now you’re onto something worth fixing. If it doesn’t show up anywhere else, you probably just watched five people being five different kinds of confused, which happens on every website that has ever existed.

I’ve started treating recordings the way I’d treat a conversation with a single customer. If one customer tells me the pricing page is confusing, I take it seriously enough to ask around, but I don’t rebuild the page around one opinion. The same discipline applies here. A recording is a customer conversation you happen to be watching instead of having.

Being honest about what you saw

The most useful habit I’ve built around this is just being precise in how I describe what I watched to myself, and to anyone else on the team. “I watched a session where a user missed the CTA” is a true, small, specific statement. “Users are missing the CTA” is a claim that needs a lot more than one recording behind it. It’s easy to let the first sentence quietly turn into the second one in your own head, especially after you’ve watched three or four recordings in a row that all felt similar.

None of this is about distrusting the tool. It’s about knowing what kind of evidence it’s giving you. A session recording is testimony, not a survey. It’s worth listening to, and worth writing down, but it earns the right to change something you built only after it’s been checked against numbers large enough to actually mean something.

If you’re trying to build a habit of checking your gut against your numbers before you make a call, that’s most of what this site is for.

Data Research Analysis Collection

Get new guides and videos first — join the Telegram channel.