What is RAG? Retrieval-augmented generation, explained

How RAG works#

A RAG system has two halves. First, documents are cut into passages and indexed so they can be searched, usually by meaning as well as by keyword. Then, when someone asks a question, the system finds the few passages most likely to help, adds them to the request, and asks the language model to answer using them.

Because the answer is built from text the system can show, it can point to its sources. The idea was described in a 2020 research paper from Facebook AI Research, University College London and New York University, and it has since become the usual way to connect a language model to an organisation’s own information.

Why not just train the model on the documents?#

Training a model on documents changes the model itself, which is slow and costly, and the knowledge can be blurred or lost. Documents also change: a price list, a policy or a manual is out of date as soon as it is edited. With RAG you edit the document and the next answer reflects it. It also keeps private material in your own store, where you decide who may retrieve what.

Where RAG goes wrong#

The model can only answer well from what is retrieved. If the search finds the wrong passage, or misses the right one, the answer will be confidently off. Passages cut in the wrong place lose their context, scanned tables and diagrams are hard to search, and a retrieved document can contain hidden instructions written to steer the model, an attack known as prompt injection. Good systems measure retrieval separately from the answer, keep a log, and say when nothing relevant was found.

Where it is used#

• Customer support Answering questions from the current help articles and product manuals.
• Internal knowledge Finding a policy, a procedure or a past decision in company documents.
• Engineering and maintenance Looking up the right section of a manual for one particular machine.
• Research and compliance Searching long reports and regulations and pointing to the paragraph.

Why has RAG become so common?#

Language models write fluently, but they know only what they saw in training, which ended on a fixed date and never included your private files. RAG fixes both problems with a search step that is cheap to build and easy to update. Tools for indexing and searching by meaning, called vector search, are now widely available, so a useful first version can be built quickly.

The catch is that quick to build is not the same as good. Teams that stop at the first demonstration often find that real questions, abbreviations and old documents break it.

RAG compared with fine-tuning and with a very long prompt#

Fine-tuning teaches a model a style or a narrow skill by adjusting it, while RAG supplies facts at the moment of the question. A very long prompt can hold a whole document, but it costs more with every request and has a limit on length.

Many systems combine them: a model tuned for the task, with a retrieval step for the facts. If your information changes often or must stay private, start with RAG.

Swipe sideways to see every column

How do you know a RAG system is any good?#

Test the two halves separately. For retrieval, collect real questions and check whether the right passage appears in the top few results. For answers, check whether each claim is supported by the retrieved text, and whether the system admits when it found nothing. Keep a set of such test questions and run it after every change.

Also test the unhappy paths: a question with no answer in the documents, two documents that disagree, and a document that tries to give the model instructions. A system that handles those honestly is worth trusting.

Key terms#

Retrieval

Finding the passages most likely to help answer a question.

Embedding

A list of numbers that stands for the meaning of a piece of text, so that similar meanings can be found by search.

Chunking

Cutting documents into passages small enough to search and to fit in a request.

Grounding

Tying an answer to specific source text that can be checked.

See every term from every guide in the glossary

Common questions#

Is RAG the same as fine-tuning?

No. Fine-tuning changes the model’s weights. RAG leaves the model alone and gives it relevant text at the moment of each question.

Does RAG stop AI from making things up?

It reduces the problem, because the answer is anchored in real text, but it does not remove it. The model can still misread a passage, so answers should show their sources.

What kinds of documents work best?

Clear, well-structured text that stays reasonably current, such as manuals, policies, support articles and reports. Scanned images, messy tables and out-of-date files give poor results.

Sources and further reading#

Independent pages we checked while writing this guide. They are not Reinnder products, and Reinnder is not affiliated with them.

• Lewis et al.: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020) (opens in a new tab)
• Anthropic: Introducing Contextual Retrieval (opens in a new tab)
• OWASP: Prompt injection (LLM01) (opens in a new tab)