Retrieval-Augmented Generation (RAG)

What Is Retrieval-Augmented Generation?

Retrieval-augmented generation is a technique that connects a language model to an external information source. When a question arrives, the system searches a document collection for relevant passages, then supplies them to the model alongside the question with instructions to answer from that material.

The problem RAG addresses is straightforward. A language model knows only what was in its training data, which has a cutoff date and contains none of an organization's internal information. Asked about a company's refund policy or last quarter's figures, a model without access to those documents will either decline or produce something plausible and invented.

RAG changes the task from recall to reading comprehension. The system finds the relevant passages first and asks the model to answer from them. This makes answers current and grounded in specific sources. It also lets the model point to where each claim came from, so users can check the output.

What Can Retrieval-Augmented Generation Do for My Business?

RAG is the standard method for applying language models to proprietary information, where most commercial value lies.

• Strategic Advantage: It makes an organization's accumulated documentation usable through a question-and-answer interface: support knowledge bases, engineering documentation, policies, contracts, research archives. That information is already owned; RAG makes it accessible without anyone knowing which document to open. Because retrieval reads from a live index, updating the system's knowledge means updating a document, not retraining a model.

• The Problem It Solves: It substantially reduces fabrication, which is the primary obstacle to deploying language models in situations where wrong answers have consequences. Grounding responses in retrieved passages, with citations users can check, converts an unverifiable answer into a checkable one.

How Does Retrieval-Augmented Generation Work?

RAG has two phases: an indexing phase that runs when documents change, and a retrieval-and-generation phase that runs per query.

Indexing (offline):

• Ingestion and parsing. Documents are extracted from their source formats: PDF, HTML, Word, Confluence, ticketing systems. Parsing quality matters more than most teams expect, since tables, multi-column layouts, and scanned pages are frequent sources of failure.

• Chunking. Documents are divided into passages small enough to be specific but large enough to retain meaning, commonly 200 to 800 tokens with some overlap. Chunking that splits mid-explanation is one of the most common causes of poor retrieval quality.

• Embedding. Each chunk is converted to a vector that captures its meaning using an embedding model.

• Storage. Vectors and their source text are stored in a vector store: either a vector database or a search engine supporting vector queries, along with metadata such as source, date, author, and access permissions.

Retrieval and generation (per query):

• Query processing. The question is embedded using the same model and may be rewritten to improve retrieval: expanding abbreviations, resolving references to earlier conversation turns, or splitting a compound question.

• Retrieval. The system finds chunks whose vectors are closest to the query vector. Production systems usually combine this with keyword search, since vector search handles paraphrase well but misses exact identifiers, product codes, and rare terms.

• Reranking. A cross-encoder model rescores the top candidates against the query more precisely than the initial search can. This step reliably improves quality and is often the highest-value addition to a basic implementation.

• Prompt assembly. The top passages are placed into a prompt with the question and instructions to answer only from the provided material, to cite sources, and to state when the material does not contain an answer.

• Generation and citation. The model produces an answer with references to the passages used so that the user can verify claims against the original documents.

What Tools Are Used to Build RAG?

• Vector databases: Pinecone, Weaviate, Qdrant, Milvus, Chroma, and pgvector for teams already running PostgreSQL.

• Hybrid search: Elasticsearch and OpenSearch combining BM25 keyword scoring with vector similarity.

• Embedding models: Cohere Embed, OpenAI text-embedding models, and open alternatives including BGE and E5.

• Reranking: Cohere Rerank, and open cross-encoder models from the Sentence Transformers family.

• Orchestration: LangChain, LlamaIndex, Haystack, or direct implementation, which many teams prefer for production because the frameworks add abstraction that complicates debugging.

• Generation models: Claude, GPT, Gemini, and open models such as Llama and Mistral, accessed directly or through Bedrock, Vertex AI, or Azure AI Foundry.

• Evaluation: RAGAS, TruLens, and custom test sets of questions with known correct answers and known source documents.

What Are the Key Characteristics of Retrieval-Augmented Generation?

• Knowledge lives outside the model. Updating what the system knows means changing documents and reindexing, which takes minutes instead of the days or weeks a retraining cycle requires.

• Answers are traceable to sources. Every response can cite the passages it used, letting users verify claims and giving auditors a record of what informed an answer.

• Retrieval quality sets the ceiling. If the correct passage is not retrieved, no model can produce a correct grounded answer. Most RAG failures are search failures.

• Access control is enforceable at retrieval. Because retrieval is a filtered query, permissions can be applied per user so that people receive answers drawn only from documents they are entitled to read.

• Chunking and parsing decisions have an outsized effect. How documents are split and how faithfully they are extracted from their original formats affect final answer quality more than the choice of generation model.

• It reduces but does not eliminate fabrication. A model can still combine two sources incorrectly or fill a gap with invention. Instructions to abstain when material is insufficient help, and evaluation is still required.

What Are the Benefits of Retrieval-Augmented Generation?

• Current information without retraining. The system reflects today's documents. A policy change published this morning is available immediately, which no fine-tuned model can match.

• Verifiable answers. Citations let users check the source, which is what makes the output usable in support, legal, and clinical settings.

• Lower cost than fine-tuning. Building an index costs a fraction of fine-tuning a model, and maintaining it is far cheaper than repeating training cycles as information changes.

• Permission-aware responses. Retrieval filtered by user entitlement means a single system can serve an entire organization without leaking restricted material across teams.

• Narrower scope for hallucination. Constraining the model to supplied passages and instructing it to abstain otherwise removes much of the space in which fabrication occurs.

What Are the Challenges and Trade-offs of RAG?

• Retrieval failure is the dominant failure mode. Vector search misses exact identifiers and rare terms; keyword search misses paraphrases. Systems relying on one or the other perform noticeably worse than hybrid approaches.

• Document parsing is harder than it appears. Tables, scanned PDFs, multi-column layouts, and diagrams routinely produce garbled text that no downstream component can recover from.

• Chunking requires tuning per corpus. Passages that are too small lose context; too large, and they dilute relevance and consume the context window. No setting works well across all document types.

• Latency accumulates across the pipeline. Embedding, searching, reranking, and generating each add time, and a naive implementation can take several seconds per query, which is noticeable in a conversational interface.

• Evaluation is difficult. Assessing whether an answer is correct, complete, and properly grounded requires a curated test set with known answers. Teams that skip this can't tell whether changes improve or degrade the system.

• Conflicting sources cause incorrect answers. When a corpus contains an outdated policy and its replacement, retrieval may return both, and the model has no reliable way to know which supersedes the other. Metadata and index hygiene are necessary.

• Permissions must be applied at retrieval. Filtering results after generation is unsafe, because restricted content may already have influenced the answer.

Should I use RAG or fine-tuning?