Production Large Language Model Engineering: Part 4

RAG changes what the model sees, not what it knows. Here's when retrieval-augmented generation is the right tool, what a full RAG pipeline actually involves, and a worked example with country-specific regulations that change monthly.

Retrieval-Augmented Generation, end to end

Retrieval-Augmented Generation pulls relevant external information at request time and adds it to the model’s context before generation happens:

User question → Search or retrieval → Relevant passages
    → Passages added to prompt → LLM generates grounded answer

The important part of that definition is what it doesn’t do: RAG changes the model’s input, not its weights. It’s the right tool the moment “what does the model know” and “what’s actually true right now” start to diverge.

When RAG is the right call

RAG earns its complexity when information changes frequently, has to stay current, needs citations, needs to be auditable, comes from private documents that shouldn’t be baked into a public model, is too large to fit in every prompt, needs to be updatable without retraining, or needs to be corrected or removed on short notice. Medical guidelines, customs regulations, company policy, legal documents, product catalogs, shipment records, technical manuals, and standard operating procedures all fit that shape.

The full pipeline, stage by stage

A production RAG system is a lot more than “embed some documents and query them.” The stages, roughly in order:

Source collection — pull from trusted, approved sources only. Garbage sources produce confidently-wrong answers just as easily as no sources at all.

Parsing — extract usable text out of PDFs, HTML, Word docs, databases, APIs, and scanned files, each of which needs its own extraction path.

Cleaning — strip navigation text, repeated headers, broken formatting, boilerplate, and duplicate content before any of it gets chunked.

Chunking — split by token length, section boundaries, paragraph structure, or semantic continuity, with overlap used carefully so information doesn’t get orphaned at a chunk boundary.

Metadata enrichment — attach document title, country, issuing authority, effective date, version, section, article number, language, source URL, and access permissions. This metadata is what makes filtering, citation, and access control possible later — skip it here and you can’t add it back cheaply.

Embedding — convert each chunk to a vector, as covered in Part 3.

Indexing — store the embedding alongside the chunk text, source ID, and metadata.

Retrieval — embed the incoming query, retrieve the relevant chunks.

Filtering and reranking — apply permission, date, and region filters, then rerank the shortlist if precision matters enough to justify the extra cost.

Generation — pass only the relevant, trusted passages into the model’s context.

Citation — return the source title, section, version, and whatever else the use case requires for someone to verify the answer.

Worked example: regulations that change every month

Say a logistics platform’s assistant has to answer customs questions across a dozen African countries. The regulations differ by country, change regularly, need exact citations, and carry effective dates and version numbers. This is close to the canonical case for RAG over fine-tuning — you cannot retrain a model every time a customs authority updates a threshold, and you absolutely cannot cite a page number from inside a model’s weights.

Collect trusted sources: customs authority publications, government gazettes, regulatory notices, approved legal databases, reviewed internal summaries. Nothing scraped from an unverified forum post.

Process the documents: extract text, clean it, chunk by both tokens and section boundaries so an article doesn’t get split mid-clause, attach the metadata from the pipeline above.

Generate embeddings for each chunk.

Store the embedding alongside the document text, country, authority, effective date, article number, version, and source reference — this is the metadata that lets step five actually work.

Retrieve with filters: country, effective date, regulation type, language, authority, and the requesting user’s access level. A query about Kenyan customs thresholds should never surface a Tanzanian regulation just because the text is semantically similar.

Generate a grounded answer: the model gets the user’s question, the current retrieved passages, the citation metadata, and explicit instructions to stay within the provided sources rather than filling gaps from its own training data.

Return citations: regulation title, article or section, country, effective date, authority, and the official source. If a customs officer can’t trace the answer back to a specific, dated regulation, the answer isn’t done yet.

Fine-tuning was never the right primary solution here — it would bake changing regulations directly into model weights, which makes updates slow and citations close to impossible. That’s a knowledge problem, not a behavior problem, and Part 5 draws that line more precisely: fine-tuning changes how a model answers, RAG changes what it has to answer with.

Quick reference

RAG — retrieve relevant information at request time, add it to the model’s context, generate a grounded answer. Changes context, not weights.

Interview-ready: what is RAG, and when do you reach for it? RAG retrieves relevant external information and injects it into the model’s context before generation. Reach for it when information changes frequently, needs citations, must be auditable, or comes from private or rapidly-updating sources — cases where retraining the model every time the facts change simply isn’t viable.

The pipeline above is only half the picture, though — it assumes the model already knows how to answer in the right format, tone, and structure. Part 5 covers the other half: fine-tuning, LoRA, and preference training, and specifically the failure mode where a well-tuned model still gives a confidently outdated answer because tuning and retrieval solve two completely different problems.