What is RAG in AI, and how does it work?
What is RAG in AI? It stands for retrieval-augmented generation, a method where, before a language model answers, an AI app searches a set of documents you choose and adds the most relevant passages to the prompt. The model then answers from that material instead of relying only on what it learned in training, and it can cite where the answer came from.
The short version
- RAG has three steps: retrieve relevant content, add it to the prompt, then generate the answer.
- It gives a model access to private or recent information without retraining the model.
- Answers can include citations, so you can check the source documents yourself.
- It reduces guessing but doesn't stop it. Microsoft warns that models can still produce inaccurate answers even with retrieved content.
- Use RAG to add knowledge. Use fine-tuning to change how a model behaves or writes.
What RAG means in AI
Large language models learn from a fixed snapshot of data. OpenAI’s GPT-6.1 Terra, for example, lists a knowledge cutoff of June 1, 2026. Anything that happened after that date, and anything that was never public, such as your company’s HR policy or your own notes, isn’t in the model.
Retrieval-augmented generation fills that gap at the moment you ask a question. Microsoft describes RAG as “a pattern that combines search with large language models (LLMs) so responses are grounded in your data.” AWS describes it as a way to make a model reference “an authoritative knowledge base outside of its training data sources” before it responds.
IBM Research uses a simple analogy. In a 2023 explainer on its site, Luis Lastras of IBM Research is quoted calling it “the difference between an open-book and a closed-book exam.” Without RAG, the model answers from memory. With RAG, it gets to look things up first.
The term comes from a 2020 research paper by Patrick Lewis and colleagues, “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks,” which was accepted at the NeurIPS 2020 conference. That system paired a language model with a searchable index of Wikipedia.
How does RAG work?
Microsoft’s documentation splits the process into three steps, and the name follows them:
- Retrieve. When you ask a question, the app searches an index or data store for content that matches it.
- Augment. The app combines your question and the passages it found into a single prompt.
- Generate. The model reads that prompt and writes an answer based on the retrieved content, with citations where the app supports them.
Much of the setup happens before anyone asks a question. Long documents are split into smaller pieces, often called chunks, so each part can be matched on its own. AWS explains that an embedding model then converts the text into numbers, called embeddings, and stores them in a vector database.
When a question comes in, it’s converted into numbers the same way and compared against the database to find passages with similar meaning. Microsoft notes that vector similarity search “matches concepts, not exact terms.” That helps with the kind of mismatch Microsoft describes, where someone asks about “PTO” but the documents say “time off.” Microsoft recommends hybrid search, which combines this similarity search with traditional keyword search, to find as many relevant passages as possible.
The documents also need upkeep. AWS notes that the stored content and its embeddings have to be updated, either automatically as things change or in periodic batches, or the answers drift out of date.
Why RAG is used
AWS, Microsoft and IBM point to similar benefits, and AWS adds a point about control:
- Private and current knowledge. The model can answer from internal documents or frequently updated sources that were never in its training data.
- No retraining. AWS calls RAG a cost-effective approach compared with retraining a model on your own data. To add new information, you update the documents.
- Sources you can check. Answers can cite the documents they came from, so readers can verify them.
- More control. AWS says developers can change which sources the model uses and restrict retrieval by authorization level, so different users only see what they’re allowed to see.
Where you meet RAG in practice
A common example is a company chatbot. AWS describes an HR assistant that, asked how much annual leave an employee has, pulls the company’s leave policy and that employee’s leave record before it answers.
Developers can build the same thing with off-the-shelf tools. OpenAI’s file search tool lets a model “retrieve information in a knowledge base of previously uploaded files” using semantic and keyword search, and its answers carry citations that point to the source file. Microsoft offers Azure AI Search for the same job, including in its Foundry platform.
RAG vs fine-tuning
Both make a general model more useful for a specific task, but they do different jobs. Microsoft’s guidance is to use RAG “when you need answers grounded in private or frequently changing data” and fine-tuning “when you need to change model behavior, style, or task performance, rather than add fresh knowledge.”
| RAG | Fine-tuning | |
|---|---|---|
| What it changes | What the model sees in each prompt | The model itself, through extra training |
| Best for | Private or frequently changing information | Behavior, style or performance on a task |
| Adding new information | Update the documents and their index | Train the model again |
| Citations | Can point to the source passage | No built-in link to a source |
The two can work together. A team might fine-tune a model to follow a house writing style and use RAG to feed it the latest product documentation.
RAG vs MCP
The Model Context Protocol (MCP) solves a different problem. Its official documentation calls it “an open-source standard for connecting AI applications to external systems,” such as files, databases, search engines and calculators, and compares it to a USB-C port for AI apps.
RAG is a technique: search for relevant text and put it in the prompt. MCP is a standard way to plug an AI app into other software. An app could use an MCP connection to reach a search tool and then use what it finds in a RAG-style answer, so the two aren’t rivals.
RAG vs a bigger context window
Models can now read far more text at once. GPT-6.1 Terra has a context window of 1,050,000 tokens and accepts up to 922,000 input tokens, according to OpenAI. So why not paste in every document?
Microsoft gives a practical answer. In its example, GPT-4 accepts about 128,000 tokens but you have 10,000 pages of documentation. Microsoft says sending everything “wastes tokens and degrades quality.” Retrieved passages also add to input tokens, which raises cost, so a good RAG system sends only the most relevant pieces. For a handful of documents, a long prompt can be enough. For a large or growing library, retrieval remains the practical choice.
The limits of RAG
RAG makes a model more reliable, but it isn’t a guarantee. Microsoft lists the main weak spots in its documentation:
- Bad retrieval means bad answers. If the search returns irrelevant or incomplete passages, the model can still give an incomplete or wrong answer. Quality depends on how the documents are prepared and indexed.
- Hallucinations still happen. Even with the right content in the prompt, a model can generate inaccurate responses. Microsoft recommends citations and clear instructions telling the model to stick to the retrieved content.
- Security needs design. Without access controls, a RAG system can leak sensitive content from its index. Microsoft also advises treating retrieved documents as untrusted input because they can carry prompt injection attacks.
- Extra cost and delay. Each search and embedding adds time and compute, and each retrieved passage adds input tokens.
IBM Research made a similar point: RAG is imperfect, and models need to be trained to recognize questions they can’t answer.
If you’re curious about running models yourself, our guide to running an LLM locally on a laptop covers what hardware you need and why longer contexts take more memory.
Frequently asked questions
What does RAG stand for in AI?
RAG stands for retrieval-augmented generation. The name describes the process: retrieve relevant information, augment the prompt with it, then generate an answer.
Is RAG an LLM?
No. RAG is a pattern built around an LLM. The language model writes the answer, while a separate search system finds the material it reads first. You can use RAG with many different models.
What is the difference between RAG and fine-tuning?
RAG adds knowledge by putting relevant documents into each prompt, and the model itself doesn’t change. Fine-tuning trains the model further to change how it behaves or writes. Microsoft recommends RAG for private or frequently changing data and fine-tuning for behavior and style.
Does RAG stop AI hallucinations?
It reduces them but doesn’t eliminate them. Microsoft says models can still produce inaccurate responses even with retrieved content, including when the search returns irrelevant or incomplete passages. Citations let you check the answer against its source.
What is agentic RAG?
Classic RAG often runs a single search per question. Microsoft describes agentic retrieval, also called agentic RAG, as using a model to break a complex question into several focused searches, run them in parallel and return structured results with citations.
More in AI
- What is a large language model (LLM), and how does it work?
- What is an AI agent, and how does it work?
- What is generative AI, and how does it work?
- What is machine learning, and how does it work?
- What is prompt engineering, and how does it work?