RAG
What is RAG in AI, and how does it work step by step?
RAG (Retrieval-Augmented Generation) means the model searches your documents first and writes the answer from what it found. The documents are split into chunks, each chunk becomes an embedding in an index, and for every question the chunks closest in meaning are added to the prompt next to the question.
Updated
How does RAG work, step by step?
- The knowledge base is split into small chunks of text.
- An embedding model turns each chunk into a vector that encodes its meaning, and the vectors go into a vector database.
- When a question comes in, the database returns the chunks closest to it in meaning.
- Those chunks are added to the prompt next to the question, and the model writes the answer from them.
What is an embedding?
A vector: a list of numbers that encodes what a piece of text means. A vector database searches by semantic similarity, so it can find a paragraph even when the question uses different words from the document.
Why does RAG still get things wrong?
- A chunk loses its context once it is cut out: a sentence about revenue growth no longer says which company or which period it refers to.
- Embeddings can miss exact matches such as an error code. Keyword search with BM25 catches those.
- Unless you tell the model to use only the documents, it can fill gaps from its general knowledge.
How does the source end up next to the answer?
Each retrieved chunk can go into the request as its own document. With the citations feature, the model returns the exact passage behind each claim, so the reader can open the document and check.
Frequently asked questions
RAG vs MCP: what is the difference?
They solve different problems. RAG is the technique of searching documents before answering. MCP is an open standard for connecting AI applications to external systems. A document search engine can itself be offered to an agent as an MCP server.
What does a reranker do in RAG?
It filters the retrieved chunks once more so that only the most relevant reach the model. The model reads less text, which makes the answer cheaper, though the reranking step itself adds a little latency.
Do I need RAG for a handful of documents?
Not necessarily. A small knowledge base can go into the prompt whole, and prompt caching makes repeated questions over it cheaper.
How do I know the answer is right?
Check the source. When the answer comes with the passage it was taken from, you can verify it. Citations cut errors a lot, but they do not remove them entirely.
What is contextual retrieval?
A method described by Anthropic: before indexing, a model writes a short note for each chunk explaining where it sits in the document, so the chunk keeps its context at search time.
Sources
Further reading
Related articles
Hybrid search: BM25 and vector search in one query
Hybrid search runs the same query twice: once with BM25, which finds exact words and codes, and once against vectors, which find text that is close in meaning. The two result lists are merged, usually with Reciprocal Rank Fusion, and a reranker picks what finally reaches the model.ReadWhen not to use RAG
You do not need RAG when all your documents fit in the model's context window: put them straight into the prompt, and prompt caching makes repeated questions cheaper and faster. When they do not fit, an agent can search files itself as it goes. RAG with an index is the pick for large corpora when speed matters.ReadAI answers with citations: grounding an LLM in your sources
An answer with citations puts the passage that supports each claim right next to it. The model is allowed to say "I don't know", pulls the quotes first and answers second, and any claim without a quote is retracted. Hallucinations drop a lot but do not disappear, so a person checks the decisions that matter.Read
Case studies
Related services
Want RAG on your company's documents?
We build the assistant on your contracts, procedures and archive, with the source next to every answer.
See the document chatbot