RAG in production

A live retrieval system is mostly a data pipeline with a model attached. Most of what goes wrong after launch is about the index rather than the model.

• Freshness: when a document changes, its chunks must be re-embedded. A stale index answers confidently from text that no longer exists.
• Deletions: a removed document whose vectors remain will keep being retrieved, and nobody notices until it is quoted back to a customer.
• Drift: changing the embedding model invalidates every stored vector. They are not comparable across models, so it is a full re-index, not a migration.
• Cost: embedding is cheap per call and adds up over a large corpus re-embedded often. Embed on change, not on a schedule.

Watch the retrieval scores in production the way you would watch latency. A falling average top score means queries are drifting away from what the corpus covers — users asking about things you have not documented. That is a content problem surfacing as a retrieval metric, and it is one of the more useful signals a RAG system produces.

It is also worth knowing when not to reach for retrieval. Questions requiring aggregation across many documents — how many orders were late last month — are database questions wearing a natural-language coat; retrieval will hand the model three documents and it will guess. Reasoning that spans a whole corpus is not what top-k similarity does.

• Needs facts from documents: retrieval.
• Needs a count, a sum, or a join: a query, possibly written by the agent, against a real database.
• Needs a consistent format or voice: a prompt or a fine-tune.
• Needs several steps and tools: that is the agent-loops course, with retrieval as one of the tools.