Khalil A. Cassimally is an audience consultant and coach who helps organisations align teams around change, user needs and AI. The user needs audit is a prerequisite for putting the model to work, yet it often stalls. Tagging hundreds of pieces by hand is slow, and the people doing it work from instinct they’ve rarely put into words. Drawing on hundreds of headlines he tagged himself and then with AI, he shares a seven-step process to:
- Build a tagging rubric: an LLM drafts it, you tag a sample of about 100 headlines, and the LLM turns your disagreements into written rules you approve
- Tag your whole archive with an LLM instead of hand-tagging a sample
- Check 20 headlines at random to surface the rules you haven’t written yet
- Set what you publish against whichever performance measures your newsroom values
In his tests, the LLM agreed with his own blind tags around nine times in ten.
> Plenty of newsrooms use the language of user needs but never audit against it. Tagging 75 pieces per need by hand is slow, and taggers work from instincts they've never written down > Let an LLM take the labour. Tag a calibration sample of about 100 headlines yourself, with notes on the hard calls, then have the LLM tag your whole archive against a written rubric > The bigger win is alignment. Ask the LLM to turn its disagreements with you into rules, such as "a research finding is UNDERSTAND, however flatly the headline states it", then approve each one as a newsroom > In Khalil A. Cassimally's tests, the LLM agreed with his blind tags around nine times in ten. Re-check 20 random rows after every rubric change: agreement under about 14 out of 20 means a rule is missing
The user needs model asks one question of everything a newsroom publishes: what is this piece for, for the person reading it? Answer it consistently, and we can see which needs we serve well and which we neglect, and commission accordingly.
In my experience, plenty of newsrooms use the language of user needs without ever having audited against it, and others did it once, on a sample, and never went back. There are two reasons for that, and AI can help with both.
Why the audit matters
The audit matters, first, because it gives us a baseline. It shows how our content serves each need today, so there’s something to measure any change against later.
It also shows us the gaps. Some needs perform far better on whatever measure we care about than others do but they’re rarely the needs most of our content serves. That mismatch, laid out in one chart, is what I often find persuades editors and leadership to act. It tells us which needs to prioritise, and which to produce less of.
Plus, the tagging itself is how a team learns user needs. In my experience, discussing whether a headline is there to update or to explain teaches the needs better than a training session does. Done together, the audit can leave a newsroom with a shared sense of what each need looks like in its own content.
> You should also sign up for Khalil’s user needs course: implement user needs in your newsroom
The two challenges that block an audit
Two things tend to stall the audit. The first is the sheer labour. Tagging means someone reads each piece, decides which need it serves, and logs that tag alongside the piece’s details and its performance figures. To get a reliable picture, I recommend tagging 75 pieces of content per number of user need. For a newsroom working with eight user needs, that’s 600 pieces. Each one is a judgement call that takes time. It piles up.
The second challenge is alignment: getting everyone who tags to apply each need the same way. If one person does all the tagging, the tags stay consistent, but the work is slow. If a small team shares it, everyone needs training, plus about a month to absorb the definitions.
Both challenges have the same root: experienced taggers work from instinct. Dmitry Shishkin, who championed the model at the BBC World Service, pointed out that it feels familiar to journalists from the start since the user needs correspond to story angles they already use. I’d add that the same familiarity means the rules people tag by rarely get written down. An unwritten rule can’t be checked or taught, and a newsroom can’t explicitly agree on it.
How AI helps
AI can ease both the labour and the alignment problem.
The labour is the obvious one: an LLM can tag an entire archive of headlines in a fraction of the time it would take a person. It also changes the maths. The 75 pieces per user need is the size of the audit. Once the LLM does the tagging, it can take on your whole archive rather than a sample. What you still tag by hand shrinks to a calibration sample of about 100 headlines, used to build and check the rubric the LLM works from.
The alignment help is, I think, the more valuable. When an LLM’s tags differ from yours, you can ask it why, and ask it to turn the differences into rules. It takes the scattered notes you made while tagging, and the calls you made without explaining them, and writes them up as general rules anyone can apply. Once they’re written down, a newsroom can discuss them and agree on them.
Doing this sharpened my own definitions of each user need. Two examples, from the definitions I’d been teaching with to the rules the process produced:
KNOW.
Before: give me the who-what-when-where; let me know the latest.
After: the headline states an event in the world and stops there. A research finding isn’t an event, however flatly the headline states it, so it’s UNDERSTAND. A poll of what people think this week is a fact about the world, so it’s KNOW.
DO.
Before: take action; find useful information for my day-to-day.
After: the reader has to be the one acting, and the headline has to instruct them. Advice aimed at governments is UNDERSTAND, and so is a question about the reader’s life that weighs the options rather than telling them what to do.
When I ran the rubric on headlines from a publisher, the LLM agreed with my blind tags around nine times in ten: 91% on Claude and 89% on Gemini.
More publishers are looking into using AI this way. Le Nouvel Obs, for one, is exploring AI to classify content though it’s “still very much in the R&D phase”.
Here’s the process I’d recommend after testing it on 275 headlines from two publishers, all of which I also tagged by hand.
Using AI to audit your content: a seven-step process
Each step exists because the next one depends on it.
The first step gives you a draft rubric, so you aren’t tagging against bare definitions. But a rubric drafted by an AI alone holds its judgement, not yours, so the second step captures yours: you tag a sample and note how you decided on the hard ones. The third step is where the alignment value comes from. The LLM compares its tags with yours and turns the differences into written rules. The fourth step makes sure every one of those rules is your newsroom’s decision before it’s applied to hundreds of pieces. The fifth applies the rubric to your whole archive, which is where the labour saving comes from. The sixth checks a random sample of the results, mainly to find the rules you still haven’t written, which go back through step four. The seventh turns the tags into the chart.
You only tag by hand in steps two and six: just over 100 headlines to build and check the rubric. The LLM does the rest.
1. Draft a rubric with AI, or start from mine
All you need coming into this step is your archive and the user needs definitions you already work with, however rudimentary at this stage.
A rubric sets out, in writing, how to decide the headlines that sit on the line between two needs. Most archives have them: a research finding stated like news (KNOW or UNDERSTAND?), an event headline with an explanation added after the dash (KNOW or UNDERSTAND?), a feature with an emotive headline that’s really an explainer (FEEL or UNDERSTAND?). The rubric is where you settle those decisions.
Give an LLM a few dozen headlines from your own archive, so the rules fit your content, and ask it for a rubric with three things: a definition of each category, the order to test them in, and a decision rule for each kind of borderline headline, with a worked example for each rule.
If you’d rather not start from scratch, the rubric I ended up with is included in full below. You can paste it as a prompt.
Either way, you now have a draft rubric. The next step captures your own judgement to set against it.
2. Tag a sample yourself, then have the LLM tag it too
The draft rubric holds the LLM’s judgement. This step captures yours, which is what turns a generic rubric into your newsroom’s.
Export your archive from your CMS or analytics tool, if you haven’t already: headlines, sections, dates and the performance metrics you care about (you’ll need those in step seven). From that export, draw around 100 headlines, spread across topics and formats. This is your calibration sample. It’s small enough to tag by hand, and it’s separate from the audit itself, which the LLM will run on the whole archive.
Tag the sample yourself first, without looking at any LLM tags. Wherever the rubric doesn’t settle a headline, write a short note on how you decided. The notes matter as much as the tags, because they’re what the LLM will turn into rules. Then give the LLM the draft rubric and the same 100 headlines, and have it tag them too.
Ideally, several colleagues tag the sample on their own as well. Where they disagree with each other, you’ve found your alignment problem at a size you can work with.
You now have two sets of tags for the same headlines. The next step turns the differences between them into rules.
3. Ask AI to turn your disagreements into rules
With your tags, your notes and the LLM’s tags side by side, this step turns scattered judgement into rules people can see and discuss. Give the LLM all three. Ask it to group the disagreements by cause, and to draft one general rule for each group, with a worked headline that shows the reasoning. Ask it as well to flag anywhere your tags contradict your own notes.
Expect a few kinds of rules. Some will generalise reasoning you’d already written in your notes, one headline at a time, into a rule for any headline. Others will write down calls you made consistently but never explained. The second kind can be the more useful, because it’s knowledge that hadn’t previously been put into words.
What comes out is a list of proposed rules. The next step decides which of them go into the rubric.
4. Decide on every rule
The proposed rules are still the LLM’s drafts, and this step makes them your newsroom’s. Every one needs a human decision: accept it or reject it.
Keep in mind that not every pattern deserves a rule. Some disagreements are genuine house calls, better left for a team to talk through. And even when a rule is written down, people can argue with it, and I’d expect those arguments to be how colleagues come to apply the needs the same way.
The rules you accept go into the rubric, which is now ready to apply at scale.
5. Tag your archive
With a rubric that holds your newsroom’s calls, this step applies it to everything you want to audit. This is where the labour goes. Paste the rubric into the prompt, then give the LLM the headlines from your export (and sections, if you have them). Because the LLM is doing the tagging, you can go well beyond 75 pieces per need if you want to – perhaps to the whole archive. Ask for one tag per headline, a confidence level, and a short reason naming which test in the rubric decided it. The reasons make the next step much quicker.
You don’t need to add many worked examples in the prompt. In my tests, adding them moved agreement by a few points at most, because the rubric already holds the detail.
Also: which LLM you use matters less than you might expect. Claude and Gemini were a few points apart in my tests.
Every piece now has a tag. The next step checks how far to trust them.
6. Check 20 at random, to surface the rules you haven’t written yet
The archive is tagged, but however good the rubric, there will be calls it doesn’t cover yet. This step finds them. Pull 20 tagged rows at random, tag them yourself without looking, then compare. For each disagreement, ask who’s right: did you slip, did the LLM, or is this a call the rubric doesn’t cover? The third kind becomes a new rule.
If you can, run the same 20 through a second LLM. Where both LLMs agree with each other against you, check your own tag first. Where both LLMs make the same mistake, there’s probably a rule missing. As a rule of thumb, under about 14 out of 20 suggests the rubric is missing a rule.
Tip: pick the rows at random, not by the LLM’s confidence flags. Some LLMs mark nearly every row high confidence, including the ones they get wrong.
Any new rules go back through step four. Once the check comes back clean enough, the tags are ready for the chart.
7. Plot what you publish against how it performs
With tags you trust, this step sets what you publish against how it performs. That’s why the export in step two needs your performance metrics as columns next to each headline: pageviews (which is what I used), time spent, conversions, return visits, or whichever measures your newsroom values. For each need, set its share of what you publish against its share of each measure. Look at the median per piece as well as the share, so one exceptional piece can’t inflate a whole category.
That chart is your baseline. The next audit will show you whether editorial changes have moved things in the right direction.
Keeping it running
The rubric is a living document. Every random check can add a rule, so date and number each version, and you’ll always know which rules a batch of tags was made under. Mine has been through four versions so far.
From there, the natural next move is tagging new content at publication, with the same rubric, rather than in a big audit after the fact. The mix can then sit on a dashboard, and you see it shift month by month. A settled rubric should also help new staff and new desks learn how the newsroom applies the needs.
What the AI and I tagged differently, and what came out of it
When I compared my tags with the LLM’s, the differences fell into three groups, and each produced something useful.
- Rules generalised from my own notes.
- The LLM’s early drafts tagged research findings as KNOW. My notes on those headlines kept giving the same reason for disagreeing, and the LLM turned that reasoning into a rule for any headline: a research finding is UNDERSTAND, however flatly the headline states it.
- On a piece about how a government could protect people from heatwaves, I’d noted that it would only be DO if the action were the reader’s own. That became the rule that DO needs the reader to be the actor.
- And I’d kept asking myself whether a headline’s emotional register was strong enough for FEEL. The LLM swapped my test for a better one: FEEL needs the piece itself to be the experience, like a review or a personal essay.
- Calls I’d made without a rule.
- Some of my tags were consistent, but I’d never said why. I’d tagged “How late is too late for dinner?” UNDERSTAND without a note. The LLM wrote down the rule I’d been applying: DO needs an instruction, not a question about the reader’s life. I’d tagged a poll of public opinion KNOW without comment, and the rule now says a poll of what people think is KNOW, while a study of how the world works is UNDERSTAND.
- My own slips.
- The LLM also held me to my own rules, flagging places where my tags contradicted my notes. I’d tagged a footballer’s sprained ankle FEEL because it was football rather than event news. The LLM had the injury as KNOW, and it was right. The rubric now says sport and celebrity coverage isn’t automatically FEEL.
The table sets the definitions I’d been teaching with against what the rubric now says:
| Category | The definitions I started with | What the rubric now says |
| KNOW (update me, keep me engaged) | Give me the who-what-when-where. Let me know the latest. Keep me up to date with the latest trends. Catch me up with what my tribe is talking about. | The headline states an event in the world and stops there. Research findings aren’t events. A poll of opinion is. Official warnings and service announcements are KNOW. Sport and celebrity news is KNOW like any other event. |
| UNDERSTAND (educate me, give me perspective) | Help me learn something. Teach me about the wider world. Broaden my horizons. Challenge my opinions. | The piece explains, analyses or argues, including every research finding. “How it works” and “why it happens” both count. An explainer stays an explainer whatever adjectives it wears. Advice aimed at institutions is UNDERSTAND. |
| FEEL (inspire me, entertain me) | Inspire me to be the best person I can be. Give me hope. Show me a not-so-serious side of a story. Give me something to smile about. | The piece itself is the experience: a review, a human story, a personal essay, a celebration of a person, something told purely for pleasure. Reviews are FEEL. Arguments about culture are UNDERSTAND. |
| DO (help me, connect me) | Take action. Find useful information for my day-to-day. Help me become part of something bigger than myself. | The reader is the one who acts, and the headline instructs them. A question about the reader’s life that weighs the options is UNDERSTAND. |
What to watch out for
AI doesn’t get this right on its own. The process works because the human and the LLM check each other. A few things are worth watching.
- A new rule can fix some headlines and break others. When I added rules, they fixed a few headlines and pulled others the wrong way. So, re-check a sample every time you change the rubric.
- Don’t defer to the LLM, even when it’s often right. Where two LLMs agreed against me, they were usually right. That makes it tempting to accept every disagreement. Every correction I made went towards the LLMs’ answer, so it’s worth asking each time whether you’ve genuinely changed your mind, or just been outvoted.
The real limitations
Beyond those watch-outs, two limitations are worth being clear about.
- One person’s tags were the ground truth. Every number in this piece measures agreement with me. Yours will measure agreement with whoever tags in step two, which is a good reason for more than one of you to do it.
- Some headlines stay contested. A few serve two needs equally, and a rule may not settle them. Those are for a team to talk through.
Start with 100 of your own headlines
The audit is the foundation of user needs work, and it stalls for two reasons: tagging hundreds of pieces by hand is slow, and the people doing it work from instinct they’ve rarely had to put into words. AI takes on most of the tagging. More usefully, it turns the notes and unexplained calls behind that instinct into written rules, which a newsroom can argue over and agree on. The judgement stays with us, in the tags we make and the rules we approve.
You don’t need a data team or a big project to start. Take the rubric below, draw 100 headlines from your own archive, tag them yourself, and see which rules your newsroom would write differently. That calibration sample is the beginning of a rubric that’s yours. From there, the LLM can tag the whole archive, and you can run the audit again whenever you need to.
***
The rubric in full
The four categories follow the top level of smartocto’s User Needs model 2.0. Paste everything below into your prompt, then adapt the rules to your own newsroom’s calls.
What you are tagging
Every piece of content gets one tag: the need it serves for the person consuming it. Not what it’s about (that’s topic), not what shape it is (that’s format). What it’s for.
Four categories, each covering two of the eight needs in the model:
| Tag | Covers | The reader comes to… | The headline typically… |
| KNOW | update me, keep me engaged | find out what happened, or what everyone’s talking about | states an event in the world: a decision, an announcement, a result, a development, a trending moment |
| UNDERSTAND | educate me, give me perspective | learn how something works, or make sense of what it means | promises an explanation, a background, an analysis, an argument, a verdict |
| FEEL | inspire me, entertain me | be moved, uplifted, amused, delighted, taken somewhere | signals the register: a review, a human story, something funny, curious or beautiful |
| DO | help me, connect me | act, in their own life or with others around a cause | addresses the reader as the one who will act, decide or take part |
The tests
Run them in this order. Stop at the first clear yes.
Does the headline address the reader as the one who will act? Tips, how-to, should-you, what-to-do, here’s-how-to, join, sign up – aimed at the reader’s own life or their part in something collective. If so, DO.
Does the headline state an event in the world, and stop there? Something happened, was decided, announced, won, lost, or is trending, with no promise to explain it. If so, KNOW.
Is the piece itself the experience? A review, a human story, a personal essay, a celebration of a person, something funny or told purely for pleasure. Strip it to a neutral summary and the value disappears. If so, FEEL.
Otherwise the piece explains, analyses or argues. How it works, why it happens, what a study found, the background, what it means, what the evidence says, the case for or against. That’s UNDERSTAND.
Decision rules for the cases that cause arguments
A research finding is UNDERSTAND, however flatly the headline states it. KNOW is for events in the world. A study result is something the reader learns (educate me), not something that happened to them. “Collagen supplements can help your skin and joints, large new study finds”: UNDERSTAND. “Horses really can smell fear, new study claims”: UNDERSTAND. It becomes DO only when the headline addresses what the reader does with it: “How eggs can help you come off Wegovy”.
A poll of opinion is KNOW. A study of the world is UNDERSTAND. “Three out of four Americans say AI firms are not doing enough to prevent disaster, Reuters/Ipsos poll finds”: KNOW, because a poll reports what people currently think, which is a fact about the world this week. “Collagen supplements can help your skin and joints, large new study finds”: UNDERSTAND, because a study reports how the world works, which is something the reader learns.
An event with an appended promise takes the tag of the promise. The event is the peg; the clause after the dash is the need. “Ebola outbreak declared a global health emergency”: KNOW. “…– what you need to know”: UNDERSTAND. “Taiwan Travelogue wins the International Booker – a deftly translated tale of food, love and history”: FEEL. “Keir Starmer has resigned – but what could a new PM actually do differently?”: UNDERSTAND.
“How it works” and “why it happens” are both UNDERSTAND. Mechanism and meaning sit in the same category: educate me and give me perspective. “Why sepsis is becoming harder to treat in Europe” and “Why climate action stalls, despite widespread popular support” are both UNDERSTAND. You don’t have to decide which of the two it is.
An explainer stays an explainer, whatever adjectives it wears. “This is why you only breathe out of one nostril at a time”: UNDERSTAND. “How Nasa’s Artemis II mission rediscovered the majesty and mystery of the Moon”: UNDERSTAND, an emotive frame on an explanatory piece. “Why was an Egyptian mummy stuffed with a fragment of Homer’s Iliad?”: UNDERSTAND, a curiosity explained. FEEL needs the piece to be the experience: a review, a story, a celebration of a person, a personal essay, something told purely for pleasure. “Forget flowers: lovers in 19th-century Ireland exchanged hair”: FEEL. “My unsung hero of science: Carolyn Wood Sherif”: FEEL.
Reviews are FEEL. Arguments about culture are UNDERSTAND. “Normal: this quirky Bob Odenkirk caper isDie Hard meets Fargo”: FEEL. “Horror won big at the Oscars – it’s time the genre was taken seriously”: UNDERSTAND.
A trending peg with an explanation is UNDERSTAND. The trend itself is KNOW. “Kim Kardashian’s new caffeine-free energy drink – here’s what the science says”: UNDERSTAND. “Prescriptions for ivermectin soar after Mel Gibson claims it cured friends of cancer”: KNOW, because the conversation is the story.
“What you need to know” is UNDERSTAND, unless the need is a personal decision. “Osteopenia: loss of bone mineral density affects millions – here’s what you need to know”: UNDERSTAND. “Vitamin D: everything you need to know – from when to take it, to how much you really need”: DO.
DO needs an instruction, not a question about the reader’s life. “Eight ways to sleep well in hot weather”: DO. “How late is too late for dinner? Does eating earlier help with weight loss?”: UNDERSTAND, because it asks and explains rather than telling you what to do. “US or UCITS ETFs: which is better for Singapore investors?”: UNDERSTAND, a comparison the reader may act on, but the piece weighs rather than instructs.
An official warning or a service announcement is KNOW. “Bus passengers face fines of up to S$500 for playing loud audio”: KNOW. It reports that something has been said or made available, even when the reader could act on it. DO needs the headline itself to instruct the reader.
DO also needs the reader to be the actor. Advice aimed at institutions is UNDERSTAND. “Eight ways to sleep well in hot weather”: DO. “Heatwaves are now everyday disasters – here’s how the UK government can better protect people”: UNDERSTAND, because it is a case for what someone else should do. “What your local council can actually do to tackle the climate crisis”: DO, low confidence, because the reader has some agency there, which is the connect-me edge of the category.
Topic never decides. Health can be any of the four. Politics can be any of the four. Sport and celebrity coverage is not automatically FEEL: an injury, a squad change or an award is an event, and tags KNOW like any other. If you find yourself tagging by section, stop.
Tag the promise the headline makes, not the piece you imagine behind it. When you only have the title, tag the title. When you have the standfirst too, let it settle the call, not overrule it.
One tag per piece. Most pieces serve more than one need. Tag the one that, if unmet, would make the reader feel the piece failed. If two are genuinely equal, tag the one higher in the test order (DO, then KNOW, then FEEL, then UNDERSTAND) and mark the row low confidence.
Output
For each row: id, tag, confidence, reason
tag is exactly one of KNOW, UNDERSTAND, FEEL, DO
confidence is high or low. Low means a second reader could reasonably tag it differently. Low-confidence rows are the ones worth checking by hand
reason is one short clause naming which test fired – e.g. “addresses reader’s action”, “event stated”, “review”, “appended promise: explains”, “study finding – educate me”
