Every retrieval-augmented chatbot demos well. You load a hundred documents, ask five questions you already know the answers to, and the thing looks like magic. Then it meets a real support queue and starts confidently citing a policy that was retired in 2023.
The failure is almost never the language model. It is the retrieval layer, the chunking, and the quiet absence of any way to tell that something went wrong.
The three failures that account for most incidents
1. Chunks that split the answer in half
The most common bug we find is boring: a document was chunked at a fixed token count, and the sentence that carries the condition landed in a different chunk than the sentence that carries the rule.
"Refunds are issued within 14 days." (chunk 41) "This applies only to orders placed before the billing cycle closes." (chunk 42)
Retrieve chunk 41 alone and the model will tell your customer something that is technically in your documentation and completely wrong.
Chunk on structure, not on length. Headings, list boundaries, and table rows are natural seams. Fixed-width chunking is a fallback, not a default.
2. Retrieval that is confident about nothing
Vector search always returns its k nearest neighbours. Ask a question your corpus cannot answer and it will still hand back five chunks, each with a similarity score, and the model will dutifully synthesise an answer from them.
The fix is a floor:
hits = index.query(embedding, top_k=5)
relevant = [h for h in hits if h.score >= MIN_SCORE]
if not relevant:
return "I don't have documentation covering that. Escalating to a human."MIN_SCORE is not a constant you can look up. You find it by scoring a few hundred real questions and locating the point where precision falls off.
3. Stale indexes nobody is watching
A knowledge base is a moving target. Someone edits the pricing page, and unless your ingestion pipeline notices, your chatbot keeps serving last quarter's numbers with perfect confidence.
Every document in the index needs a source_updated_at and every pipeline needs an alert when the gap between that and indexed_at crosses a threshold.
What to instrument before you launch
| Signal | What it tells you | Alert when |
|---|---|---|
| Retrieval score distribution | Whether questions are landing in-corpus | p50 drops week over week |
| Escalation rate | How often the bot correctly declines | Sudden drop — it stopped declining |
| Answer-to-source distance | Whether the model is inventing detail | Spikes on any single document |
| Index staleness | Whether ingestion is keeping up | Any doc older than its source |
The counterintuitive one is escalation rate. Teams celebrate when it falls. More often it means a retrieval change quietly lowered the bar and the bot started answering questions it should have handed off.
Evaluate on the questions you are afraid of
A test set built from questions you know the system handles proves nothing. The useful set is adversarial:
- Questions whose answers changed recently
- Questions that span two documents
- Questions that are nearly covered — close enough to retrieve, not close enough to answer
- Questions with a false premise, where the correct answer is a correction
Score those before every deploy. A model swap that improves your happy-path numbers can quietly destroy category four.
None of this is exotic. It is the difference between a chatbot that impresses in a meeting and one that survives contact with customers — and it is almost entirely about the plumbing around the model rather than the model itself.