The pile of text nobody reads
Every survey I’ve ever run has the same shape. Ten multiple choice questions with clean bar charts, and then one open text box: “anything else you’d like to add?” or “why did you cancel?” That box is where the useful stuff lives, and it’s also the box I used to skip past because reading three hundred free text answers by hand takes an evening I don’t have.
That’s the actual problem AI text classification solves for a solo operator. Not “understanding customer sentiment” in some abstract sense. Just: I have a spreadsheet column full of sentences, and I need to know what buckets they fall into without reading every row myself.
What “letting a model categorise” actually means
In plain terms, you’re handing a language model a piece of text and a list of categories, and asking it to pick the one that fits, or tell you none of them do. That’s it. There are two common setups.
The first is closed categorisation: you already know your buckets. For a churn survey that might be “too expensive,” “missing a feature,” “switched to a competitor,” “no longer needed,” and “other.” You feed the model each response plus that list, and it returns a label. This works the same way whether you’re doing it through a spreadsheet plugin, a script calling an API, or pasting responses into a chat interface a few dozen at a time.
The second is open discovery: you don’t have categories yet, or you don’t trust the ones you’d guess. Here the model (or a clustering step that runs on the text’s embeddings, which are just numerical representations of meaning) groups similar responses together first, and then you name the clusters after looking at what landed in each one. This is closer to how a researcher would do open coding by hand, except the first pass of grouping is automated.
Most tools blend the two: you give a starting list of categories plus an “other” catch-all, and periodically review what’s piling up in “other” to decide if it deserves its own bucket.
A concrete example: the cancellation survey
Say you run a small subscription tool and you ask people why they’re cancelling. Over a few months you collect a stack of answers like:
“Just didn’t use it enough to justify the cost.”
“Found something that does the same thing for less.”
“Company restructured and cut the budget.”
“It’s fine, just don’t need it right now.”
A model can sort these into categories like price sensitivity, competitor switch, budget or organisational change, and low usage, faster than you’d get through the first ten by hand. What it’s actually doing is pattern matching against the meaning of the category label you gave it and the meaning of the sentence, using everything it learned about how language works during training. It’s not looking anything up about your specific business. It doesn’t know your product. It’s working purely off the words in front of it and the category names you supplied.
That last point matters, because it’s also where the mistakes come from.
Where it goes wrong
“It’s fine, just don’t need it right now” is a real example of an answer that’s genuinely ambiguous. Is that low usage? Is it a soft no on value? A human reading it in isolation could argue it either way, and so can a model. You’ll see responses land in a category that’s defensible but not what you’d have picked, and you’ll see it flip-flop on borderline cases if you run the same batch twice with slightly different wording in your prompt.
Sarcasm and context are the other common failure points. “Oh sure, because I definitely need another subscription eating my card” reads as enthusiasm if you’re only pattern matching on words like “need” and “subscription,” but any person would catch the sarcasm immediately. Models catch a lot of this now, more than they used to, but not all of it, and you won’t know which responses tripped it up unless you check.
There’s also a quieter failure mode: category drift. If your category list is vague, “product issue” versus “feature request” versus “missing feature,” the model will apply them inconsistently, not because it’s bad at the task but because you gave it overlapping definitions. That’s not a model problem. That’s a you-wrote-fuzzy-categories problem, and it shows up whether a human or a model is doing the sorting.
None of this makes the tool useless. It makes it a tool that needs a human checking its work, same as any other measurement in this business.
The workflow that actually holds up
I don’t trust a classification pass I haven’t spot checked, and I’d suggest you don’t either. A workflow that works:
Read a sample yourself first. Before you write category definitions, read fifty or so responses by hand. This is how you find out your customers describe “too expensive” five different ways, and it’s how you catch the categories you wouldn’t have thought of from the multiple choice questions alone.
Write tight category definitions, not just labels. “Price” as a category name is vague. “Price: says the cost isn’t justified relative to how much they used it” is something a model, or a new hire, can apply consistently.
Keep an “other” bucket and actually look at it. If ten percent of responses land in “other,” treat that as information rather than a failure of the classifier. Either you’re missing a category or the responses genuinely don’t fit a pattern.
Spot check a sample of the machine’s labels against your own read. Pull twenty or thirty labelled responses and grade them yourself. If you disagree with the label more than a handful of times, your category definitions probably need sharpening before you trust the output at scale.
Treat the output as a first pass, not a verdict. The categorised counts tell you where to look closer, not what decision to make. If “competitor switch” jumps as a category one month, that’s a prompt to go read those specific responses in full, not a green light to assume you know why people are leaving.
What this buys you and what it doesn’t
What it actually buys you is speed and consistency on a task that’s tedious enough that most solo operators skip it entirely. A stack of two hundred open text answers that would sit unread in a spreadsheet becomes a rough breakdown you can act on in an afternoon instead of never.
What it doesn’t buy you is certainty. It’s not measuring anything more real than a human coder would measure by hand, it’s just doing the sorting faster and occasionally getting the edge cases wrong in ways a careful human wouldn’t. Treat the category counts the way you’d treat any other survey number: as a signal that tells you where to dig, not as the finding itself.
If you’re weighing whether this is worth setting up for your own business, the honest answer depends on how much open text feedback you’re actually sitting on and whether it’s currently going unread. If it’s a handful of responses a month, reading them yourself is still faster than building a pipeline. If it’s hundreds, a categorisation pass, checked against your own judgment, is a reasonable way to get through it.
For more on running your business on the numbers you actually have instead of the ones you wish you had, head back to the homepage.
Get new guides and videos first — join the Telegram channel.