RAG
When not to use RAG
You do not need RAG when all your documents fit in the model's context window: put them straight into the prompt, and prompt caching makes repeated questions cheaper and faster. When they do not fit, an agent can search files itself as it goes. RAG with an index is the pick for large corpora when speed matters.
Updated
RAG vs long context: do you still need RAG if it all fits?
Not necessarily. Anthropic's advice is that a small knowledge base can go into the prompt whole. There is no index to maintain, and the model sees the full text instead of a few chunks picked by a search.
What is prompt caching?
The model resumes from a prompt prefix it has already processed, in the order tools, system, messages, up to the cache marker. Cache reads cost less than regular input, so a fixed set of documents at the start of the prompt gets cheaper on every question that follows.
Agentic search vs RAG: what is agentic search?
The agent keeps lightweight references at hand, such as file paths, stored queries and links, and loads the data only when it needs it. Claude Code works this way: CLAUDE.md goes into context up front, and files are found along the way with glob and grep. The trade-off is speed, because exploring at runtime is slower than reading from a precomputed index.
What is context rot?
As you add more tokens to the context window, the model gets worse at recalling what is in it. Anthropic describes attention as a finite budget, so documents fitting in the window is no reason to load all of them.
Frequently asked questions
How large can a knowledge base be and still go in the prompt?
As much as fits in the model's context window. On your own documents the test is simple: if answers get worse with everything in the prompt, switch to retrieval.
Is agentic search better than RAG?
It needs no index and reads files directly, but it is slower. RAG answers quickly from data prepared ahead of time. Claude Code mixes the two: some context is loaded up front, the rest is searched when needed.
What is a context window?
Everything the model sees on one call: instructions, tools, documents, the conversation so far. It has a limit, and the model uses it less well as it fills up.
How long does the prompt cache last?
Briefly by default, but the lifetime resets each time the cache is used, at no extra cost. For longer gaps there is an extended lifetime that costs more.
Sources
Related articles
What is RAG in AI, and how does it work step by step?
RAG (Retrieval-Augmented Generation) means the model searches your documents first and writes the answer from what it found. The documents are split into chunks, each chunk becomes an embedding in an index, and for every question the chunks closest in meaning are added to the prompt next to the question.ReadHybrid search: BM25 and vector search in one query
Hybrid search runs the same query twice: once with BM25, which finds exact words and codes, and once against vectors, which find text that is close in meaning. The two result lists are merged, usually with Reciprocal Rank Fusion, and a reranker picks what finally reaches the model.Read
Case studies
Related services
Not sure you need RAG?
Tell us what documents you have and what your team asks. We will tell you whether the prompt is enough or you need retrieval.
Write to us