Your answer to a better knowledge base might be a much smaller one
The pitch for agentic AI in the service desk is seductive. Point an agent at your knowledge base, let it resolve tickets, watch the deflection rate climb, go on holiday and let the agents make the money. However, that pitch assumes your knowledge base is ready for a machine reader.
Knowledge written for humans and knowledge written for AI are not quite the same artefact:
- An [AI] agent can infer context and work around gaps too, but it does so silently and without the social checks a human has, no colleague to ask, no hesitation before acting on a guess.
- A human filling a gap usually leaves a trace, a note, a question in the ticket, a moment of doubt.
- An [AI] agent filling the same gap just proceeds, confidently, and the guess becomes the record.
That’s the core difference here; not capability, but visibility. A wrong inference from an agent looks identical to a right one, right up until it’s repeated a thousand times, and there’s been real business impact. It’s a bit like those Chrome tabs you keep opening. We all know they are there, and eventually it’s going to cause you a problem.
Good quality knowledge is optional until your first deterministic workflow, then it becomes mandatory.
That’s the line I keep coming back to. Whilst your knowledge base is just a reference for human analysts, gaps and inconsistencies are survivable, people work around them and leave a trail whilst they do it.
“The moment you wire an agent into a workflow that acts without a human in the loop, every gap becomes a guaranteed failure somewhere downstream, and you don’t get to find that out gradually.”
I get that “fix the knowledge base” sounds like a large, unglamorous undertaking next to “deploy an agent.” But the two aren’t sequential projects, they’re one in the same.
It’s also worth challenging one assumption that’s crept into a lot of AI readiness thinking: that the goal is a single source of truth.
I’m convinced that’s the wrong north star. Multiple knowledge systems identifying and reconciling knowledge is the future, just as humans have done forever; nobody trusts one colleague’s memory over everyone else’s; we cross-reference, we ask around, we reconcile conflicting versions until we’re confident. Agents should work the same way, not be handed one brittle repository and told it’s gospel.
Practical first steps to take
Here’s why: documentation tells you how a problem was supposed to be fixed. Ticket history and chat tell you how it actually got fixed. That gap between the two is real tribal knowledge, the workaround the team quietly settled on because the documented process didn’t quite hold up in practice. It’s the most valuable knowledge in your organisation and the least likely to be written down anywhere official.
To get at it safely:
🔹Mine ticket history for repetition, not recency. The same fix showing up across twenty tickets is a pattern worth turning into a proper knowledge article, in its own right, distinct from whatever the documentation says.
🔹Synthesise across threads, never thread by thread. Extracting one article per conversation, whether you take the fix or the reasoning, barely beats doing nothing; the thread is simply too incomplete a unit to carry knowledge on its own. Group related threads first, then write one article from the group. Amazon tested exactly this on a year of internal support tickets and got a 26% lift in helpful answers, a 77% drop in completely unhelpful ones, and a knowledge base 3% the size of what they started with.
🔹Weight by independent repetition, not by who said it. Five separate threads or tickets converging on the same fix, from different people, is a strong signal.
🔹Treat disagreement with the documentation as a finding, not noise. When the tickets and the docs disagree, that’s exactly the kind of conflict a reconciliation layer should be surfacing for a human to adjudicate, not something to quietly smooth over.
🔹Strip identity and ticket-specific detail before anything gets reused. The value is the fix, not the names, the ticket number, or the “same as last time” shorthand that only makes sense to someone who remembers last time.
🔹Build evals alongside your workflows. Use the power of the same LLMs you are using to perform your workflows to constantly evaluate it.
More on Evals
Evals are what all the cool kids nowadays are calling evaluations, and uncool grownups call testing – measurement of AI agent performance. They need their own article really, but you use the same LLM tech you use to run your workflows to evaluate their performance. This can be binary measure or a scored/rubric measure:
🔹Binary: did it cite a real source?
Pass or fail. Every factual claim in the answer points to a document that exists in the knowledge base. No cited source, or a cited source that isn’t there, is a fail. No judgement needed, and you can automate it.
🔹Scored: was the answer complete enough to act on?
Rated 1 to 5 by a human, or by a second model against a rubric. A ‘5’ means the person could follow it without coming back. A ‘2’ means it was technically correct but left out something they’d need.
Different reviewers will disagree at the margins, which is fine, that’s the nature of it. The value is in the trend over time, not any single score. Worth knowing the trade-off: binary evals are cheap and unambiguous but only catch mechanical failures. Scored evals catch the failures that matter more and cost more to run. Most teams need a handful of each rather than one or the other.
Final thoughts
Do all of the above, before touching your formal documentation, and you’ll have found your organisation’s real tribal knowledge, tested it against itself, and made it usable, all before you’ve written a single new page.
Note what that Amazon result implies, too: the knowledge base that performed best was dramatically smaller than the raw material it came from, not a faithful transcription of it. The compression is where the human judgement lives.
My north star through all of this has been simple: agent resources are abundant, human resources are finite. That statement is easy to agree with and harder to act on, because it means the human effort you do spend has to go on the highest-leverage work:
🔹structuring knowledge
🔹defining ownership
🔹deciding what “good” looks like before you let a model near it.
Skip that step and you haven’t built an AI-native service desk, you’ve built a faster way to be confidently wrong.
What’s your experience been, is knowledge quality the blocker on your AI roadmap too?
Further reading
📕 My take on Open Knowledge Format and why it matters
📘 Atomic Content: Building for humans, machines, and everything in between — Justin Waugh
📕 From Unstructured Communication to Intelligent RAG — Zhang et al., KDD 2025
About the author
Scott Gamble
Scott is an AI adoption strategist and AI knowledge specialist, focused on getting AI to work and deliver value in the enterprise. He has most recently worked at ServiceNow overseeing AI knowledge strategy. Based in the UK, he runs Tech Detectives, teaching children AI literacy and safety.
