When RAG Is Useful and When It Is Not

What RAG actually does

Retrieval-augmented generation, usually shortened to RAG, connects a language model to a search step. Before the model answers, the system retrieves relevant passages from a set of approved sources and asks the model to answer from those passages, showing where each part of the answer came from.

The model is not being taught your documents. It is being handed the most relevant extracts at the moment a question is asked, and instructed to stay within them. That distinction matters: it is why RAG can work with knowledge that changes daily, and why the quality of what is retrieved decides the quality of what is answered.

That makes RAG a good fit when the knowledge lives in your documents, changes often, and people need to be able to check where an answer came from.

How a RAG system answers a question

Most RAG systems follow the same broad sequence. Each step is a place where quality is won or lost.

• Collect: approved documents are gathered from the systems where they live, along with information such as owner, date and who may see them.
• Prepare: documents are cleaned and split into passages small enough to retrieve precisely but large enough to keep their meaning.
• Index: each passage is stored so it can be found by meaning, and often by keywords too.
• Retrieve: when a question arrives, the system finds the passages most likely to contain the answer, filtered to what the person asking is allowed to see.
• Answer: the model writes a response from those passages and cites them, or says it cannot find an answer.

When a RAG system gives a poor answer, the cause is usually earlier in this chain than the model: the right document was missing, out of date, badly split, or not retrieved.

Where it helps

RAG earns its place when several of these conditions are true.

• Policies, procedures and product documentation that people search every day.
• Answers that must show their source, so people can verify them before acting.
• Knowledge that changes faster than a model could ever be retrained.
• Different people who are allowed to see different documents.
• Large collections where finding the right paragraph is the slow part of the job.

Typical uses include internal assistants for policies and procedures, support teams answering from product documentation, and sales or operations teams looking up current terms, specifications or processes.

Where it does not

RAG is not a fix for everything. If the source documents are out of date, contradictory or missing, retrieval will faithfully surface the problem rather than solve it.

• The source material is incomplete, contradictory or has no clear owner.
• The job is a calculation, a database lookup or a fixed rule. Those are better answered directly from the system that holds the data.
• The real need is a workflow that takes actions, not an answer to a question.
• Nobody has decided who is allowed to see what, so permissions cannot be enforced.
• The question needs reasoning across an entire collection at once, such as “what has changed across all our contracts this year”, which retrieval of a few passages does not handle well on its own.

What decides answer quality

Teams often assume that a better model will produce better answers. In practice, the largest gains usually come from elsewhere.

• Source quality: one current, owned version of each document beats five conflicting copies.
• Preparation: how documents are split affects whether the right context is retrieved together.
• Retrieval: combining meaning-based and keyword search, and filtering by metadata, often matters more than the model.
• Instructions: the model should be told to answer only from the retrieved sources and to say when it does not know.
• Feedback: a simple way for users to flag a poor answer turns every failure into something that can be fixed.

Alternatives worth considering

RAG is one pattern among several. Depending on the problem, one of these may be simpler or better.

• Better search: if people only need to find the right document, improved search may be enough.
• Direct queries: questions about structured data, such as orders, stock or account status, are best answered from the system of record.
• Fine-tuning: useful for teaching a model a style, format or specialised task, but not a good way to keep it up to date with changing facts.
• Long context: for a small, fixed set of documents, passing them in full can be simpler than building retrieval.
• Workflow automation: if the goal is to get something done rather than answered, design the workflow first.

Security needs more than a pattern

RAG is a retrieval pattern, not a security control. Calling a system “RAG” says nothing about whether it is safe. Safety depends on decisions made around it.

• Which sources are connected, and whether anything sensitive has been included by accident.
• Whether retrieval respects each user’s existing access rights at the moment they ask.
• Where the model runs, what it receives, and whether prompts or documents are stored.
• How document content is treated, since text inside a document can contain instructions that try to change the system’s behaviour.
• What is logged, who can review it, and how long it is kept.

Those choices should be made explicitly and written down, not assumed because a familiar pattern name is in use.

How to evaluate a RAG system

Evaluation should happen before launch and continue after it. A useful starting point is a set of real questions with known correct answers and their sources.

• Did the system retrieve the passages that contain the answer?
• Is the answer supported by those passages, without adding unsupported claims?
• Are the citations correct and useful to the reader?
• Does it say “I don’t know” when the answer is not in the sources?
• Does it refuse to reveal content the user is not permitted to see?

Measuring retrieval and answering separately shows where to improve. If the right passage was never retrieved, a better model will not help.

How to decide

Collect twenty real questions people ask today, identify where the correct answers live, and check who is allowed to see them. That exercise is quick and reveals most of what you need to know.

If the answers exist in documents you trust and can keep current, and access rules are clear, RAG is a strong candidate. If the answers live in structured systems, query those systems instead. If the answers do not exist anywhere reliable, fix the sources first: no retrieval system can return knowledge that was never written down.