Production Large Language Model Engineering: Part 2

Rate limiting, backpressure, retries, timeouts, and circuit breakers

An LLM application isn’t one system, it’s a dependency chain: model providers, databases, vector stores, external APIs, message queues, auth services, file pipelines, search services. Any link in that chain can go slow, overloaded, unavailable, or inconsistent, and the model call is usually the slowest, least predictable link of all.

The reliability patterns in this post each answer a distinct question. Worth holding onto that framing, because it’s easy to reach for “add a retry” as a catch-all when the actual problem is a rate limit, a full queue, or an unhealthy dependency that retrying will only make worse.

Rate limiting     → Who is allowed to send how much traffic?
Backpressure      → Can the system safely accept more work?
Timeout           → How long should one call wait?
Retry             → Could a temporary failure succeed later?
Exponential backoff → How should retry delays increase?
Jitter            → How do we prevent synchronized retries?
Circuit breaker   → Should we stop calling this dependency?
Idempotency       → Can this operation be safely repeated?
Deduplication     → Have we already processed this request?

Rate limiting: who’s sending too much

Rate limiting caps how many requests a client, tenant, or API key can send in a period — 100 requests per minute per client, say. It exists for fairness, abuse prevention, cost control, tenant isolation, and predictable quotas, and it’s usually enforced right at the boundary: API gateway, reverse proxy, edge service, load balancer. Reject the excess early, before it triggers expensive downstream work.

Say one hospital on a telehealth platform uploads 50,000 patient records in one batch. Without a per-tenant limit, that single upload can consume most of the processing capacity, slow down every other hospital on the platform, and spike AI costs unpredictably. A policy like “1,000 document-processing requests per hospital per hour” fixes that — and it applies to that hospital even when the platform has spare physical capacity, the same way a venue capping “100 visitors per hour” for one organization applies regardless of how much floor space is actually free.

RateLimiterConfig config = RateLimiterConfig.custom()
    .limitForPeriod(100)
    .limitRefreshPeriod(Duration.ofMinutes(1))
    .timeoutDuration(Duration.ZERO)
    .build();

RateLimiter limiter = RateLimiter.of("userRateLimit", config);

String result = RateLimiter
    .decorateSupplier(limiter, () -> processDocument(document))
    .get();

limitForPeriod(100) allows 100 permits per refresh window; limitRefreshPeriod resets that count every minute; timeoutDuration(Duration.ZERO) fails immediately instead of queuing when no permit is available. When a client goes over quota, the standard response is 429 Too Many Requests with a Retry-After header telling them when to come back.

Backpressure: can the system take more work right now

Rate limiting asks who’s sending too much. Backpressure asks a different question entirely: is the system itself full? It protects memory, CPU, threads, queues, database connections, worker pools, and — for an AI system specifically — inference capacity and provider quotas.

Say a service can process 200 AI requests per minute, and 800 arrive. Without backpressure, the queue just keeps growing, memory climbs, latency stretches out until requests are effectively stale by the time they’re processed, and eventually the service falls over. With backpressure, the queue hits a cap and new work gets rejected or delayed while the existing work finishes safely — a worse individual experience for the rejected request, but a system that’s still standing five minutes later.

BlockingQueue<DocumentRequest> queue = new ArrayBlockingQueue<>(1000);

boolean accepted = queue.offer(request);

if (!accepted) {
    return ResponseEntity.status(503)
        .header("Retry-After", "30")
        .body("System at capacity");
}

The load-bearing detail is the bound. new LinkedBlockingQueue<>() with no capacity limit doesn’t solve overload, it just delays the failure — and usually makes it worse, since by the time memory runs out you’re dealing with OutOfMemoryError, thread starvation, and stale work all at once instead of a clean, immediate rejection. When the whole system lacks capacity, the honest response is 503 Service Unavailable with Retry-After.

Rate limiting and backpressure aren’t substitutes for each other, and you need both. Say each client is allowed 10,000 documents per hour, and the platform can actually process 50,000 per hour total. Ten clients, each obeying their individual quota perfectly, add up to 100,000 documents per hour — double what the system can handle. Every client played by the rules and the system still drowns. Rate limiting buys fairness. Backpressure buys safety. Neither one buys the other.

Retry, backoff, and jitter: handling failures that might just be temporary

Retry assumes a failure might succeed if you try again — a network blip, a connection reset, a brief service restart, a gateway error, provider throttling. Retryable candidates typically include timeouts, connection resets, 429, 500, 502, 503, and 504; when a 429 or 503 comes with Retry-After, respect it rather than retrying on your own schedule.

What you don’t retry is anything that’s actually your fault: 400, 401, 403, 404, 422. Retrying invalid input, missing auth, or a missing resource doesn’t fix any of those — it just wastes a call.

RetryConfig config = RetryConfig.custom()
    .maxAttempts(3)
    .waitDuration(Duration.ofSeconds(1))
    .retryExceptions(TimeoutException.class, IOException.class)
    .ignoreExceptions(IllegalArgumentException.class)
    .build();

Fixed delay is the simplest version, but it keeps hammering an already-struggling dependency at a constant rate. Exponential backoff fixes that by growing the delay after each failure — 1 second, 2, 4, 8:

RetryConfig config = RetryConfig.custom()
    .maxAttempts(4)
    .intervalFunction(
        IntervalFunction.ofExponentialBackoff(Duration.ofSeconds(1), 2.0)
    )
    .build();

That gives the dependency actual room to recover instead of getting retried into the ground. But it introduces a new problem on its own: if 10,000 clients fail at the same instant, exponential backoff makes them all retry together at 1 second, then all together at 3 seconds, then all together at 7 — synchronized waves hitting a service that’s trying to recover. Jitter breaks the synchronization by randomizing the delay, so an “expected” 4-second delay actually lands somewhere like 3.5, 4.2, or 4.8 seconds across different clients:

RetryConfig config = RetryConfig.custom()
    .maxAttempts(4)
    .intervalFunction(
        IntervalFunction.ofExponentialRandomBackoff(Duration.ofSeconds(1), 2.0, 0.5)
    )
    .build();

Backoff reduces the pressure of any single retry. Jitter stops the retries from arriving in waves. A solid production strategy runs bounded retries with backoff, jitter, and Retry-After support all together — and “bounded” is not optional. Retrying forever leaves threads occupied, queues blocked, costs climbing, and duplicate actions more likely. Always cap the max attempts, the max elapsed time, and the per-attempt timeout explicitly.

Idempotency and deduplication: making retries safe, not just possible

Retry and backpressure both assume it’s safe to try an operation again. That’s true for a GET — reading data twice does nothing extra. It’s true for a PUT that sets a shipment’s status to DELIVERED — repeating it leaves the same end state. It is not true for a POST /bookings — repeating that can create a second booking.

Here’s the actual failure: a booking request succeeds, but the response gets lost on the way back. The client has no way to know whether the write failed or just the acknowledgment did, so it retries — and now there are two bookings for one intent. This is why a non-idempotent write should never be retried blindly unless the system has some way to recognize “I’ve already done this.”

That mechanism is an idempotency key — a unique identifier for one logical action, sent by the client and reused on every retry of that same action (never regenerated per attempt, or the whole scheme is pointless):

Original request → Booking 101 stored under key K
Retry with key K → Return Booking 101 (not a new booking)

The naive server-side implementation is a race condition waiting to happen:

if (!processedRequests.containsKey(key)) {
    String result = processRequest();
    processedRequests.put(key, result);
}

Two concurrent requests can both check the key, both find it missing, and both proceed — the check-then-act isn’t atomic. The key has to be reserved atomically, via a database unique constraint, Redis SET NX, a compare-and-set, or a dedicated idempotency service. A production idempotency table typically tracks the key, a hash of the request body, status, the stored response, and an expiry. The request hash matters because the same key should never be allowed to cover two different requests — if key ABC was used to book container X, it shouldn’t silently also succeed for a request to book container Y. Same key plus same hash returns the stored result; same key plus a different hash gets rejected outright.

Keys should expire — 24 hours, 7 days, 30 days depending on the realistic retry window. Too short and a legitimately delayed retry slips past deduplication; too long and you’re storing keys you’ll never need again.

Put together: timeout makes retries necessary, idempotency makes retries safe. They’re solving adjacent but distinct problems, and you need both, not one or the other.

Timeout: how long should the caller wait

A timeout bounds how long a caller waits before giving up and treating the operation as failed. Without one, a hung call to an external API blocks the calling thread indefinitely, which occupies a connection, which lets more requests pile up behind it, until the thread or connection pool exhausts and one slow dependency has taken the whole service down with it.

@Service
public class ShippingRateService {

    @TimeLimiter(name = "carrierApi", fallbackMethod = "fallbackRate")
    @CircuitBreaker(name = "carrierApi", fallbackMethod = "fallbackRate")
    public CompletableFuture<String> getRate(String shipmentId) {
        return CompletableFuture.supplyAsync(() ->
            externalCarrierApi.fetchRate(shipmentId)
        );
    }

    public CompletableFuture<String> fallbackRate(String shipmentId, Throwable error) {
        return CompletableFuture.completedFuture("Rate unavailable — try again shortly");
    }
}

resilience4j:
  timelimiter:
    instances:
      carrierApi:
        timeout-duration: 3s
        cancel-running-future: true

The detail that trips people up: a timeout stops the caller from waiting, not necessarily the underlying operation. The remote server may have already received and started the work. If the thread ignores interruption, the HTTP client can’t actually cancel the in-flight request, or a database transaction has already committed on the other side, the operation can complete anyway — after the caller has already moved on and marked it failed. Caller timed out does not mean operation definitely failed. That gap is exactly why timeouts, retries, and idempotency have to be designed as one system, not three independent settings.

Timeouts also need to exist at more than one level — connection, read, write, database statement, socket, queue wait, and an overall request deadline. A single wrapper-level timeout doesn’t guarantee every underlying resource actually gets released.

Circuit breaker: when to stop calling a struggling dependency

A retry assumes the failure is temporary. A circuit breaker assumes the opposite — that a dependency has developed a persistent pattern of failure and further calls are likely to fail too, so it’s better to stop calling for a while rather than keep pushing traffic (and wasted retries) into something that’s already down.

It runs through three states. Closed: requests flow normally, failures and slow calls get measured. Open: once a failure threshold is crossed, requests fail immediately without even attempting the call. Half-open: after a cooldown, a small number of test calls go through — succeed, and the circuit closes again; fail, and it reopens.

Timeout and circuit breaker aren’t the same tool wearing two hats. A timeout bounds one request. A circuit breaker decides whether to keep sending new requests to a dependency based on its pattern across many calls over time. And they depend on each other: without timeouts, calls can hang open indefinitely, resources exhaust, and the circuit breaker never gets clean outcomes fast enough to actually open in time.

Retry and circuit breaker are also distinct, working on different assumptions and different timescales — retry treats one 503 as transient and worth a few seconds of patience; a circuit breaker treats a ten-minute provider outage as a pattern worth stopping traffic for entirely.

They compose naturally, circuit breaker on the outside, retry on the inside:

CircuitBreaker breaker = CircuitBreaker.of("azure-breaker", breakerConfig);
Retry retry = Retry.of("azure-retry", retryConfig);

String result = CircuitBreaker.decorateSupplier(
    breaker,
    Retry.decorateSupplier(retry, () -> callAzureOpenAI(document))
).get();

The circuit breaker checks state first — if open, fail immediately without touching retry logic at all. If closed, the call enters the retry policy, gets attempted, and retryable failures get delayed re-attempts. Once every attempt in that retried operation fails, it counts as one failure toward the circuit breaker’s threshold. The decoration order matters here: your team has to decide deliberately whether each individual retry attempt or the whole retried operation is what feeds the circuit breaker’s failure count, because that choice changes how sensitive the breaker actually is.

A more complete version, wired for calling a model provider:

public class AzureOpenAIClient {

    private final RetryConfig retryConfig = RetryConfig.custom()
        .maxAttempts(3)
        .intervalFunction(
            IntervalFunction.ofExponentialRandomBackoff(Duration.ofSeconds(1), 2.0, 0.5)
        )
        .retryOnException(error ->
            error instanceof TimeoutException
            || error instanceof IOException
            || isRetryableStatus(error)
        )
        .build();

    private final Retry retry = Retry.of("azure-retry", retryConfig);
    private final CircuitBreaker breaker = CircuitBreaker.of("azure-breaker", breakerConfig);

    public String classifyDocument(String document) {
        return CircuitBreaker.decorateSupplier(
            breaker,
            Retry.decorateSupplier(retry, () -> azureClient.classify(document))
        ).get();
    }
}

A brief network blip gets absorbed by retry. Temporary provider overload gets breathing room from backoff. A genuine multi-minute outage trips the circuit breaker so you stop wasting calls into a dead dependency. And if a write’s outcome was left ambiguous by a timeout somewhere in that chain, idempotency is what stops a later retry from duplicating it.

Quick reference

Pattern	Answers
Rate limiting	Who is sending too much?
Backpressure	Is the system full right now?
Timeout	How long may one call wait?
Retry	Could this specific failure succeed on a second try?
Backoff	How much should the delay grow after each failure?
Jitter	How do we stop every client retrying at once?
Circuit breaker	Should new calls to this dependency stop entirely?
Idempotency	Can this write be safely repeated?
Deduplication	Have we already handled this exact logical request?

Interview-ready: rate limiting vs. backpressure. Rate limiting controls how much traffic each client may send, mainly for fairness and abuse prevention. Backpressure controls how much total work the system can safely accept, based on its actual current capacity — and a client can be fully within its rate limit while the aggregate traffic from all clients still overwhelms the system, which is exactly why you need both.

Interview-ready: timeout vs. circuit breaker. A timeout limits how long one request waits. A circuit breaker stops new requests from reaching a dependency once it’s shown a repeated pattern of failure or slowness across many calls.

Interview-ready: why idempotency matters. A timeout or dropped response can leave a caller genuinely unsure whether a write succeeded. An idempotency key means a retry of that same action returns the original result instead of repeating the side effect.

Part 3 shifts from keeping the system alive to giving the model something worth saying — embeddings, semantic search, and how retrieval actually finds relevant information instead of just the nearest one.