RAG knowledge bases: a field guide

Retrieval-augmented generation is the pattern behind most useful "chat with your documents" systems. The idea is simple: search first, then let a language model write an answer from the search results, with citations. Getting it production-grade is where the work is.

The pipeline

1. Ingest: pull documents from SharePoint, Confluence, Drive, ticketing systems, databases. Preserve permissions and metadata.
2. Chunk: split into passages that carry meaning on their own (headings help; fixed 500-token windows do not).
3. Embed & index: vector index for semantic similarity, keyword index for exact terms; hybrid retrieval beats either alone.
4. Retrieve & re-rank: fetch candidates, re-rank with a cross-encoder, filter by the user's access rights.
5. Generate: answer with citations, or decline if the evidence is thin.
6. Evaluate: retrieval recall and answer faithfulness on a labelled set, every release.

Where RAG systems break

• Bad chunks: tables and multi-column PDFs mangled at ingestion. Fix ingestion before touching prompts.
• Stale content: nightly incremental re-indexing is the minimum.
• Permission leaks: filter at retrieval time, not in the prompt.
• Overconfident answers: enforce "cite or decline".
• No feedback loop: thumbs-up/down feeding an evaluation set is the cheapest quality lever you have.

What "good" looks like

In an enterprise deployment we ran, common policy and process questions were answered around 70% faster than the previous "ask a colleague" route, every answer carried a source, and the content-gap report became the documentation team's backlog. That last effect, RAG revealing what your docs don't say, is often the biggest long-term win.

Build or buy?

Off-the-shelf tools are fine for a single, low-risk source. Build (or heavily customise) when you need access control, multiple sources, actions, or evaluation you can defend to auditors.