Retrieval-Augmented Generation (RAG): Architecture, Chunking, and Evaluation

Retrieval-Augmented Generation (RAG) is the pattern that turned LLMs from impressive demos into systems that can answer questions about your documents, your policies, and your codebase. The idea is simple: instead of relying on what the model memorized during training, you retrieve relevant material at query time and hand it to the model as context. The execution is anything but simple. Years into the RAG era, the pattern is mature, the tooling is abundant — and the majority of underperforming AI assistants still trace their problems to the same place: retrieval quality that nobody measured. This guide covers the architecture, the decisions that matter, and how to evaluate the result like an engineer.

Why RAG persists in the age of huge context windows

A recurring prediction says that ever-larger context windows will make RAG obsolete — just paste everything in. It has not happened, for three stubborn reasons. Cost: processing your entire knowledge base on every query is wildly more expensive than retrieving the relevant slice, even with prompt caching. Quality: models demonstrably attend less reliably to information buried in the middle of enormous contexts, so more context is not automatically better answers. Freshness and access control: a retrieval layer can enforce document permissions per user and reflect updates instantly — capabilities a static context dump cannot offer. Long context and RAG are complements: retrieval selects, the context window receives.

The anatomy of a production RAG pipeline

A production system has two halves: an ingestion pipeline that runs when documents change, and a query pipeline that runs per request.

Ingestion

1. Parsing: converting PDFs, HTML, wikis, and office documents into clean text while preserving structure. Unglamorous, and the source of more quality problems than any other stage — garbage parsing guarantees garbage answers.
2. Chunking: splitting documents into retrievable units (more below).
3. Enrichment: attaching metadata — source, date, section, access permissions — and often a generated summary or contextual header per chunk.
4. Embedding and indexing: converting chunks to vectors and writing them to a vector index, usually alongside a traditional keyword index.

Query time

1. Query processing: optionally rewriting the user\'s question — expanding acronyms, adding context from the conversation, or decomposing multi-part questions.
2. Retrieval: fetching candidate chunks via vector similarity, keyword search, or both.
3. Reranking: scoring candidates with a more precise model and keeping the best few.
4. Generation: prompting the LLM with the question and the selected chunks, with instructions to ground its answer in them and cite sources.

Chunking: the decision everyone underestimates

Chunking determines what your system can and cannot retrieve, and no single strategy wins everywhere. The practical options:

• Fixed-size with overlap: the naive baseline — split every N tokens with some overlap. Easy, and surprisingly serviceable for homogeneous prose.
• Structure-aware: split on document structure — headings, sections, paragraphs — so chunks align with semantic units. The sensible default for documentation, policies, and wikis.
• Semantic chunking: use embedding similarity between sentences to find natural topic boundaries. Better coherence at higher preprocessing cost.
• Contextualized chunks: prepend each chunk with generated context — what document it came from, what the section covers — so the chunk makes sense in isolation. This cheap technique fixes the classic failure where a retrieved paragraph says "this approach" and nobody knows which approach.

Two rules travel well. First, chunk boundaries should respect meaning: a table split in half or a code block severed from its explanation will never retrieve well. Second, retrieve small, generate with more — a common pattern retrieves precise small chunks but hands the model the surrounding section, balancing retrieval precision against generation context.

Retrieval quality: hybrid search and reranking

Pure vector search misses exact identifiers — error codes, product SKUs, function names — that keyword search catches trivially; keyword search misses paraphrases that embeddings handle easily. Production systems therefore run hybrid search: both retrievers in parallel, results fused (reciprocal rank fusion being the standard technique). A reranker — a cross-encoder model that scores each candidate against the full query — then filters the candidate pool down to the handful of chunks the LLM actually sees. Adding a reranker is frequently the single largest quality improvement available for one day of work.

Evaluation: measure retrieval and generation separately

The cardinal sin of RAG development is evaluating only the final answer. When an answer is wrong, you need to know which stage failed, so measure the stages independently.

Retrieval metrics

Build a test set of questions paired with the chunks or documents that should be retrieved. Then track recall@k (is the needed material in the top k results?) and precision or MRR (how highly is it ranked?). This requires labeling, and there is no shortcut — though an LLM can draft question/passage pairs for humans to verify, which cuts the effort dramatically.

Generation metrics

Given correct retrieval, score the answer on groundedness (does every claim trace to the provided context?), relevance (does it answer the actual question?), and completeness. LLM-as-judge evaluation with a clear rubric, spot-checked by humans, is the standard practice.

A minimal evaluation harness is not exotic engineering:

# Pseudocode: staged RAG evaluation
for case in eval_set:
    retrieved = pipeline.retrieve(case.question)
    recall = overlap(retrieved, case.expected_chunks)
    answer = pipeline.generate(case.question, retrieved)
    scores = judge_model.score(answer, retrieved,
                               rubric=[groundedness, relevance])
    log(case.id, recall, scores)

Failure modes and their fixes

• Right document, wrong chunk: revisit chunking; add contextual headers.
• Vocabulary mismatch between questions and documents: query rewriting, or index generated question-forms of each chunk.
• Stale answers: incremental re-indexing on document change, with metadata filters preferring recent versions.
• Confident answers from thin retrieval: instruct the model to refuse when context is insufficient, and test that it actually does.
• Permission leaks: enforce access control in the retriever, per user, at query time — never rely on the model to withhold what it was given.

RAG in 2026 is not a research problem; it is an engineering discipline with known best practices and measurable quality. Teams that treat it that way — parsing carefully, chunking deliberately, searching hybrid, reranking, and evaluating each stage — ship assistants that users trust. Teams that bolt a vector database onto a prompt and hope have already discovered where hope leads.