Production Large Language Model Engineering: Part 5

Fine-tuning, LoRA, DPO, RLHF, and putting the whole system together

Parts 1 through 4 covered how a model processes a request, how the system around it survives failure, and how it retrieves relevant knowledge at request time. This last post covers the piece that changes the model itself — and closes with what a complete production architecture looks like once all five layers are wired together.

The question to ask before touching any of this: is this a behavior problem or a knowledge problem?

Need clearer instructions?        → Prompting
Need current external knowledge?  → RAG
Need stable behavior or skill?    → Supervised fine-tuning
Need preference alignment?        → DPO or RLHF

Getting that question wrong is where most of the expensive mistakes in this layer come from.

Supervised fine-tuning: teaching a task, not a fact

SFT trains a model on input-output pairs — {"prompt": "Translate this clinical note into Swahili.", "response": "..."}. The relationship it learns is simply input maps to desired output, applied across enough examples that the pattern generalizes.

It’s genuinely good at teaching new task behavior, response structure, output schemas, domain terminology, tone, classification, translation style, tool-use patterns, and concise summarization — things like “always return JSON using these exact fields” or “write summaries in the clinic’s approved format.”

Where SFT runs out of road: it shows the model a correct answer, but not which of several equally-valid answers is preferable. A model trained this way can end up producing correct information while still being too verbose, overly hedgy, inconsistent in tone, or bad at choosing between two acceptable responses. That’s not a knowledge gap SFT can close — it’s a preference gap, which is a different training problem entirely.

LoRA: adapting without retraining everything

Low-Rank Adaptation freezes the base model’s weights and trains small additional matrices on top instead of touching the full parameter set:

Frozen base model + Trainable LoRA adapters

The full weight update is approximated as a product of two much smaller matrices, ΔW = A × B, which captures most of the useful behavioral change without the cost of training a matrix the size of the original. Practically, that means lower GPU memory, cheaper training runs, smaller checkpoints, faster iteration, and the ability to swap adapters in and out without touching the base model at all. LoRA isn’t its own training method — it’s a cost-reduction technique you layer under SFT, DPO, or any other tuning approach.

Preference training: teaching which answer is better

A preference record is a prompt with a chosen response and a rejected one:

{
  "prompt": "Patient is six months old with a fever. What should I do?",
  "chosen": "Provide a concise, safe, protocol-aligned response.",
  "rejected": "Provide an unnecessarily defensive and unhelpful response."
}

The objective is simply: for this prompt, chosen beats rejected. That’s a meaningfully different training signal than SFT’s “here’s the correct answer” — it’s “between two plausible answers, prefer this one,” which is exactly the gap SFT alone can’t close.

RLHF: the traditional, expensive way to align preferences

RLHF runs in three stages. First, supervised fine-tuning on high-quality demonstrations produces a baseline SFT model. Second, a reward model gets trained on human comparisons — humans rank several responses (Response A > C > B > D), and the reward model learns to predict a score for any prompt-response pair, something like 0.92 for concise-and-correct, 0.63 for correct-but-verbose, 0.15 for wrong. Third, reinforcement learning — typically PPO (Proximal Policy Optimization) — has the model generate responses, the reward model score them, and the policy update toward higher-reward behavior.

That’s three separate models, a full RL training loop, and a fair amount of operational surface area: training instability, reward hacking, mode collapse, high compute cost, sensitive hyperparameters, and evaluation that has to cover a lot of ground. RLHF still earns its complexity when the reward signal genuinely needs to come from an interactive environment or is more complex than a static set of offline preference pairs — but for most teams, that’s not the common case.

DPO: most of the benefit, a fraction of the machinery

Direct Preference Optimization trains straight from chosen/rejected pairs, no separate reward model and no PPO loop:

Traditional RLHF: SFT → Reward model → PPO
DPO:              SFT → Direct preference-pair training

The objective pushes the model toward the chosen response and away from the rejected one, while a regularization term keeps it from drifting too far from the reference model. That’s most of what makes DPO attractive: fewer moving parts, lower cost, more stable training, easier to reproduce, and it composes cleanly with LoRA.

from transformers import AutoModelForCausalLM
from trl import DPOTrainer, DPOConfig
from peft import LoraConfig

model = AutoModelForCausalLM.from_pretrained("./clinic-assistant-sft")

preference_data = [
    {
        "prompt": "Patient: 6 months, fever 39°C. Advice?",
        "chosen": "Concise, safe, clinic-aligned answer.",
        "rejected": "Verbose or unhelpful answer."
    }
]

trainer = DPOTrainer(
    model=model,
    args=DPOConfig(beta=0.1, num_train_epochs=1, per_device_train_batch_size=2),
    train_dataset=preference_data,
    peft_config=LoraConfig(r=8)
)

trainer.train()

(Check this against whatever version of the library you’re actually running — the interface shifts between releases.)

Beta is the knob controlling how strongly the model moves toward the preference data versus how tightly it stays anchored to the reference model — smaller beta generally allows more movement away from reference behavior, larger beta pulls harder toward it, though the precise mechanics depend on the implementation’s exact formulation. It affects preference strength, how much behavior actually shifts, output diversity, training stability, and how much of the base model’s original capability gets retained. Tune it against real evaluation data, not intuition — explanations of what beta “does” get oversimplified often enough that it’s worth verifying against your training library’s actual objective function.

DPO isn’t automatically the better choice in every case, though. Traditional RL still fits better when rewards come from an interactive environment, when long action sequences determine success rather than a single response, when feedback is programmatic rather than expressible as pairwise preferences, or when online adaptation matters. DPO fits naturally when your data can say, plainly, “for this prompt, answer A beats answer B” — which covers a lot of real production use cases, just not all of them.

Fine-tuning vs. RAG: the decision that actually matters

	Fine-tuning	RAG
What changes	Model weights	Input context
Best for	Behavior, skills, style	Facts and external knowledge
Update method	Retrain or retune	Update and re-index documents
Update speed	Slower	Faster
Citations	Difficult	Natural fit
Auditability	Limited	Strong
Real-time information	Poor fit	Strong fit
Data deletion	Difficult	Remove the source record
Primary question	How should it answer?	What information should it use?

Reach for fine-tuning when the actual requirement is about form: answer in exactly two sentences, produce this JSON structure, use approved terminology, apply a specific classification method, prefer concise summaries. It’s justified once prompting and examples alone can’t hold that consistency.

Fine-tuning is a poor primary tool for anything that changes fast — current prices, new regulations, live inventory, updated medical guidelines. Knowledge baked into weights goes stale, updating it means retraining, citing it is close to impossible, and removing outdated knowledge from a trained model is not a solved problem. It also does not, on its own, eliminate hallucination — that requires actual grounding, not just better tuning.

RAG is the better fit when information changes frequently, citations are required, answers need to be auditable, the source is private, or a document needs to be correctable or removable on short notice — removing a bad source record is trivial; removing knowledge from model weights isn’t.

The realistic answer is usually both

Production systems generally use fine-tuning for behavior and RAG for knowledge, layered together:

Fine-tuned behavior: "Answer in two concise sentences. Use approved terminology.
                       Return the required dosage format."

Retrieved knowledge: current clinic protocol, latest approved guidance,
                       drug interaction data, country-specific regulations.

Combined flow:
User question → Retrieve current trusted information → Add to context
    → Fine-tuned model answers in required style → Return citations

Fine-tuning teaches the form and behavior. RAG supplies current, traceable content. Confusing the two is where a specific, very avoidable failure shows up.

The failure mode: a well-tuned model, an outdated answer

Say a model has been DPO-tuned to match a clinic’s communication style. A clinician asks: “What is the current dosage according to the latest approved guideline?” The model answers fluently, confidently, in exactly the right tone — using an old guideline.

DPO changed what the model prefers to say, not what it knows. It learned to be concise, to use the right formatting, to sound confident, to favor certain phrasing — none of which has anything to do with whether its underlying factual knowledge is current. This is a knowledge-freshness failure wearing a preference-training costume, and it’s a very easy one to misdiagnose if you don’t hold the “behavior vs. knowledge” question firmly enough.

The fix is architectural, not a tuning fix: retrieve the latest approved guideline, verify its version and effective date, pass the relevant section into context, generate the response in the tuned style, and return the citation. For anything high-risk, add an approved-source allow-list, version and effective-date checks, citation verification, an explicit no-answer path when evidence is missing, human review where it’s warranted, and full logging.

Mistakes worth naming directly

Fine-tuning frequently changing facts instead of retrieving them. Using RAG to fix every inconsistency problem, when genuinely inconsistent behavior needs SFT or preference training instead. Assuming DPO adds current knowledge — it aligns preferences, it doesn’t hand the model new regulations or prices. Treating RLHF and DPO as interchangeable when one runs a full reward-model-plus-PPO pipeline and the other trains directly on preference pairs. Claiming fine-tuning eliminates hallucination — mitigating it takes trusted retrieval, tool calls, output validation, and source checking working together, not a single training run. Returning vector-search results with no evaluation, on the assumption the nearest match is automatically a good one. Retrying every error, including the ones retrying can’t fix. Treating a timeout as proof of failure, when the operation may well have completed after the caller stopped waiting.

The complete architecture

Putting all five layers of this series into one request path:

Client request
    → Authentication and authorization
    → Rate limiting
    → Input validation
    → Idempotency-key validation
    → Backpressure / bounded queue
    → Query understanding
    → Keyword and vector retrieval
    → Metadata filtering
    → Reranking
    → Trusted context construction
    → Token-budget validation
    → LLM call with timeout
    → Bounded retry with backoff and jitter
    → Circuit breaker
    → Output validation
    → Citation verification
    → Store idempotent result
    → Return response

And the monitoring that has to sit around it: request volume, token usage, cost, rate-limit rejections, queue saturation, timeout and retry rates, circuit-breaker state, retrieval relevance, hallucination indicators, citation correctness, end-to-end latency, error rates, and user feedback. None of that is optional at any meaningful scale — it’s how you find out a component is degrading before a user does.

Quick reference

SFT — teach the model a task or output format, from input-output demonstrations.

DPO — teach which of two valid responses is preferred, directly from chosen/rejected pairs.

RLHF — human preferences, distilled into a reward model, optimized against with reinforcement learning.

Interview-ready: fine-tuning vs. RAG. Fine-tuning changes model weights and is best suited to behavior, style, format, and task skills. RAG changes the model’s context and is best suited to current, auditable, source-grounded knowledge. Ask “how should it answer” for fine-tuning and “what information should it use” for RAG — most real systems need an answer to both.

Interview-ready: why is DPO often preferred over full RLHF? DPO trains directly from chosen and rejected responses, skipping both the separate reward-model training stage and the PPO reinforcement-learning loop that traditional RLHF requires — less infrastructure, lower cost, and generally more stable to actually run.

Interview-ready: can DPO replace RAG? No. DPO adjusts response preferences and behavior; it doesn’t grant access to current external facts. A production system typically uses DPO to shape how it answers and RAG to determine what it answers with.

That closes the series. The through-line across all five parts: a production AI system has to control traffic, protect its own capacity, bound its failures, retrieve trustworthy information, manage its token budget, validate what comes out the other end, and reach for model training only for the behavior that prompting and retrieval genuinely can’t provide on their own.