Production Large Language Model Engineering: Part 1

Tokenization, context windows, sampling, and prompting basics

Being good at prompting doesn’t make someone a production AI engineer, any more than knowing SQL makes someone a database administrator. A reliable AI application sits on top of several distinct disciplines: how language models actually process text, how to keep a system working under failure and traffic, how to give a model current information it wasn’t trained on, and how to change its behavior when prompting alone isn’t enough.

This series works through those four layers in order — foundation, reliability, knowledge, adaptation. This first post covers the foundation: the mechanics of tokenization, context windows, sampling, and prompting that everything downstream depends on.

Tokenization: the unit everything else gets billed and bounded in

Neural networks don’t read text. They do matrix multiplication on numbers. Tokenization is the step that turns “Hello world!” into something a model can actually process — split the text into pieces, then map each piece to a numerical ID:

Raw text → Tokenizer → Tokens → Token IDs → Model processing

"Hello world!" might become ["Hello", " world", "!"], then [15496, 995, 0]. The exact numbers depend entirely on which tokenizer you’re using — they’re not universal constants, just an interview-ready shorthand to say: tokenization splits text into units and maps those units to IDs a model can consume.

A token is not a word. It can be a whole word, part of a word, a word ending, a punctuation mark, a number, a symbol, or a raw byte when nothing else fits. "Hello" is likely one token. "unbelievable" might come out as ["un", "believ", "able"]. A rare technical term can cost even more. Word count and token count are two different numbers, and conflating them is one of the most common mistakes engineers make when budgeting for LLM calls.

Most modern tokenizers build their vocabulary with Byte Pair Encoding (BPE) or something in that family. The training process, simplified:

1. Start from characters or bytes.
2. Count the most frequent adjacent pairs.
3. Merge the most common pair into a new unit.
4. Repeat until you hit the target vocabulary size.

t + h → th, later th + e → the. Common words end up as single tokens; rare ones stay fragmented. That’s the whole trade-off BPE is solving: represent frequent text efficiently, without needing a dedicated vocabulary entry for every conceivable word.

Why this matters once you’re running a production system

Cost. Providers bill per token, not per word — input tokens, output tokens, cached tokens, reasoning tokens, system instructions, conversation history, tool definitions, structured-output schemas. All of it counts.

Context usage. The context window is measured in tokens, and it has to fit the system prompt, developer instructions, history, the user’s query, retrieved documents, tool schemas, examples, and the generated response, all at once.

RAG chunking. Chunk documents by character count and you’ll get wildly inconsistent chunk sizes, because token density varies across languages, code, URLs, and formatting. “500 characters per chunk” is a fragile rule. “400 tokens per chunk,” with overlap also measured in tokens, holds up.

Model migration. Different providers use different tokenizers. A prompt that fits comfortably in one model’s context can produce a meaningfully different token count in another. Migration testing needs to check token counts, context fit, cost estimates, chunk sizes, and output budgets, not just “does the output still look right.”

Multilingual tokenization has no universal multiplier

Tokenization efficiency varies by language, and it’s tempting to memorize a ratio — “non-English costs 1.3x” or similar. Don’t. It depends on the specific tokenizer, the model, the writing system, domain vocabulary, and formatting, and the only way to know is to measure it.

Say a logistics platform processes shipping manifests in both English and Swahili. The right evaluation process looks like this: collect representative documents in each language, tokenize both sets with the actual production tokenizer, calculate average tokens per document, estimate output tokens, include system prompts and schemas, check the longest outlier records, and add a safety margin. A real measured result might come out as 900 tokens average for English manifests versus 1,250 for Swahili — but that number belongs to that dataset and that tokenizer. It’s not a constant you can port to a different pipeline.

This is also why a visible 400-word prompt can turn into 600+ tokens by the time it hits the model: rare words split into subwords, punctuation and numbers consume tokens, JSON and XML add structural characters, chat-role boundaries add overhead, and none of that is visible in the word count you’re eyeballing. A real token budget accounts for the complete request the model receives, not just what the user typed.

Tokens vs. embeddings, briefly

It’s easy to conflate these because both turn language into numbers, but they solve different problems. Tokenization prepares text for model input and output — it’s generally reversible through decoding, and its job is managing cost and context. Embeddings represent meaning as a dense vector for similarity search and retrieval — they’re not reversible into the original text, and their job is semantic relevance, not cost control. Part 3 of this series goes deep on embeddings; for now, the distinction to hold onto is: tokens are about processing text, embeddings are about representing what it means.

A useful mental shortcut: characters are atoms, tokens are molecules. "Hello" is five characters but might be one token — tokens are the reusable combinations a specific vocabulary has learned to recognize, not a fixed physical unit.

Mistakes worth avoiding

• Estimating token count from word count. Use the actual tokenizer for the production model.
• Forgetting hidden overhead — system prompts, tool definitions, few-shot examples, schemas, and history all eat into the budget silently.
• Chunking documents by character count instead of tokens.
• Assuming every language tokenizes at the same rate.
• Forgetting that output tokens share the same budget as input tokens.

Context windows and token budgeting

A context window is the maximum number of tokens a model can consider in one request — both the input you send and, depending on the API, the tokens reserved or generated for the response. It’s not memory. It’s the working space available for the current interaction, full stop.

Here’s where the arithmetic gets uncomfortable fast. Say the context window is 16,000 tokens, the system prompt costs 500, and you retrieve 5 documents at 2,000 tokens each — that’s 10,000 tokens of documents alone. Total usage so far: 10,500 tokens, leaving 5,500. And that remaining space still has to cover the user’s actual question, conversation history, formatting overhead, tool calls, and the model’s response. Retrieve one document too many and the request simply doesn’t fit.

The fix is planning the budget explicitly instead of discovering it by hitting the ceiling. Something like:

Total context:                16,000
System and policy instructions: 1,000
Conversation history:           2,000
User query:                       500
Retrieved context:              8,500
Reserved output:                4,000

The exact split depends on the task, but the rule underneath it doesn’t change: never let retrieved documents or conversation history eat the entire window. Always reserve space for output and a safety margin.

For long-running conversations, the usual tools are: keep only the most recent messages, summarize older ones, store structured user facts separately from raw transcript, retrieve only the historically relevant messages instead of the whole thread, drop repeated tool outputs, and compress large documents before they go back into context. The goal was never “send everything” — it’s “send what’s relevant.”

Temperature and sampling

At each generation step, the model computes a probability distribution over possible next tokens. Temperature reshapes how sharply or broadly that distribution gets sampled.

Lower temperature makes the highest-probability tokens dominate — good for classification, extraction, code generation, structured JSON, compliance workflows, and anything where you want the same input to produce roughly the same output. Higher temperature gives lower-probability tokens a real shot at being picked — good for brainstorming, creative writing, naming, and generating varied alternatives on purpose.

The limitation worth internalizing: temperature doesn’t make a model smarter. It doesn’t increase reasoning depth or accuracy. It changes how predictable the output is, nothing more. Need consistency, go lower. Need variation, go higher. And for anything high-risk, randomness should be tightly controlled and paired with validation — temperature is not a substitute for a guardrail.

Zero-shot and few-shot prompting

Zero-shot prompting gives the model instructions with no examples — “classify the following message as urgent or non-urgent.” It’s the right default when the task is straightforward, the model already understands it, the output format is simple, and examples wouldn’t meaningfully change the behavior.

Few-shot prompting adds worked examples to the instructions:

Message: "The server is unavailable."
Classification: Urgent

Message: "Please update my profile photo."
Classification: Non-urgent

Message: "Production payments are failing."
Classification:

Examples are how you teach formatting, label definitions, tone, edge-case handling, output length, and decision boundaries that plain instructions struggle to pin down. They also cost tokens, so they need to earn their place — concise, representative, and no more of them than necessary.

The switch from zero-shot to few-shot isn’t about task difficulty in the abstract, it’s about observed behavior: reach for few-shot once the model clearly understands the general task but keeps producing inconsistent output or misreading the intended pattern.

Structured reasoning

Complex tasks benefit from decomposition — instead of asking for the answer directly, asking the model to identify the task, extract relevant information, apply the required rules, check constraints, then produce the output.

Worth being deliberate about here: a production system generally shouldn’t expose raw, unrestricted internal reasoning to end users. The better pattern is asking for concise conclusions, evidence, or verifiable intermediate outputs instead — something like:

Return:
1. The classification
2. The supporting evidence
3. Any missing information

That gives you the transparency and auditability you actually want, without turning a chain-of-thought dump into a user-facing feature you now have to support.

Quick reference

Tokenization — text → tokens → IDs. This is the unit cost and context are measured in. Word count is not token count.

Context window — system prompt + history + user input + retrieved documents + output, all under one ceiling. Budget it explicitly; never let retrieval or history consume the whole thing.

Temperature — low for consistency, high for variation. It changes predictability, not intelligence.

Few-shot prompting — instructions plus examples, used when the model understands the task but needs help nailing the pattern.

Interview-ready answer, tokenization: Tokenization breaks text into smaller units and maps those units to numerical IDs a language model can process. It directly drives context usage, API cost, multilingual efficiency, and how you size RAG chunks — which is why “word count” is never a safe stand-in for “token count” in production planning.

Part 2 moves from how the model processes a single request to how the system around it survives failure and load: rate limiting, backpressure, retries, timeouts, and circuit breakers.