Your LLM gateway failed over. The failure vanished.

Ten requests hit a dead primary model. All ten returned 200 through the fallback. The billing log recorded ten successes and zero traces of the failed primary attempt.

A gateway that makes this choice is more than a reverse proxy. It selects the answering model, decides whether a key may spend, limits which models that key may reach and reroutes traffic when the preferred provider fails. It governs other systems without performing the business task itself: a control plane.

I tested three of those decisions on my own Vauban reference deployment: a LiteLLM 1.85.1 gateway carrying traffic in my lab, not a client system. The experiment changed no configuration files. It created temporary keys and one deliberately unreachable model, then removed them. The numbers below describe that deployment at that point in time, not LiteLLM or gateways in general.

Three tests, three operational surprises

Failover kept the caller online and erased the failed attempt

The test registered a temporary primary model pointing at a closed local port and disabled retries. Each request named one working fallback. This isolated failover from retry backoff and from a slow provider timeout.

All ten calls returned HTTP 200 from the fallback model group. The median end-to-end round trip was 1,140.1 ms. Subtracting the served model duration reported by the gateway produced a median derived failover overhead of about 49 ms. That number is arithmetic, not a separate timer for the failed attempt.

The caller could see the route change in two response headers: x-litellm-attempted-fallbacks: 1 and x-litellm-model-group: chat-fast. The response body named only the model that answered.

Thirty seconds later, LiteLLM_SpendLogs contained ten success rows for those fallback requests and no rows for the failed primary attempts. As a control, ten calls to the same dead primary without a fallback produced ten failure rows. The gateway could record the failure; this path did not preserve it in the billing store.

Failover preserved availability, not a complete audit trail. Persist the routing headers or emit one event per attempt; otherwise a provider outage can vanish from the store used during incident review. Assembled reports less than 0.001% request failure during a multi-hour provider outage with automated fallback. That shows the availability benefit. It does not answer whether your evidence system can reconstruct each routing decision.

A budget cap controls admission, not the final bill

The second test created a virtual key with a one-cent max_budget and sent one request at a time until the gateway refused it. Thirteen calls were served, consuming 71,726 tokens. Call fourteen returned HTTP 429 in 22.8 ms.

The delivered cost and the counter quoted by the gateway agreed to nine decimal places. The counter was not stale. Request thirteen was admitted while cumulative spend was still below the cap, then its own cost carried the total above it.

In this sequential run, the overshoot was $0.00052464, or 5.25% of the tiny test cap. That percentage is not a reusable forecast. The reusable design rule is structural: a pre-flight budget can be exceeded by the cost of an admitted request whose final token count is known only afterward.

Under concurrency, several requests could be admitted against the same pre-flight state. This experiment did not measure that shape. A serious budget design therefore combines the monetary cap with maximum input and output tokens, request concurrency, model access and an alert before the hard refusal.

The obvious recovery actions reported success but did not restore service

After the key crossed its cap, I tried the two actions an operator would reasonably reach for. Both admin calls returned HTTP 200. Both made /key/info report zero spend. Neither cleared the refusal in this deployment.

A genuinely new key value restored access. That is more disruptive than resetting a dashboard counter, so it belongs in the runbook before an incident. The experiment shows what happened, not why: it did not trace LiteLLM internals, test a proxy restart or establish whether another version behaves the same way.

Revocation was fast on one process; model scope failed clearly

For each of five repeats, the test authenticated a new key, deleted it, then polled every 0.25 seconds. Every key was refused on the first poll, and no request was served after deletion returned. The result bounds the observed window below the polling interval plus one request round trip. It does not prove zero delay.

A horizontally scaled gateway is a different problem because every replica may have its own authentication cache. The operator-facing revocation window is determined by the slowest replica, which this single-process test did not measure.

Model scope was simpler: a key allowed to call chat-fast requested chat-deep. All three attempts were refused with HTTP 403 in about 16 ms, while three matching control requests succeeded. The error named the allowed model, the requested model and key_model_access_denied, enough information for an operator to resolve the request without gateway access.

A fallback ladder is also a data-residency policy

The experiment used one valid fallback in scope. It did not test cross-region routing. The architectural consequence is still important: the same mechanism that moves traffic around a failed provider can move it across a geopolitical boundary.

Microsoft recommends treating geopolitical boundaries as isolated deployment stamps. Clients should reach only an authorized regional endpoint, and the gateway itself should not make a cross-geopolitical request on their behalf. If no compliant fallback remains, returning an error can be safer than silently serving from elsewhere.

Encode region or jurisdiction as an enforceable routing constraint on every primary and fallback candidate, using the mechanisms your gateway version supports. Then test the empty-set case: remove every in-region backend and verify that routing fails closed. A diagram that shows an EU primary and a US fallback is not a residency design; it is an untested exception path.

Where these measurements stop

These results are useful because their limits are visible:

• One LiteLLM 1.85.1 proxy, one Postgres instance and in-process caches were tested at one point in time.
• The dead primary refused TCP immediately. A timeout, throttle or slow response would produce a different failover delay.
• The budget run used one sequential caller, one model, one cap and one request shape. It did not characterize concurrency.
• Revocation tested key deletion on one proxy, not a multi-replica fleet or an already in-flight request.
• The jurisdiction section is an architectural consequence supported by external guidance, not a fourth measurement.

The control-plane acceptance test

Once routing, money and authority share one endpoint, I require five things:

• Record every attempt, not only the answer. Persist the requested model, failed primary, selected fallback, provider, region, latency and reason for rerouting under one correlation ID.
• Bound request cost as well as account spend. Combine budget, token, concurrency and model limits; alert before the hard stop.
• Test the recovery path. Trip a budget in staging and follow the exact runbook an operator will use, including credential rotation and client rollout.
• Make jurisdiction part of routing eligibility. A fallback that violates residency is not healthy. When no compliant candidate exists, fail closed.
• Protect the gateway itself. A central gateway removes client complexity and creates a central failure domain. Replicate it, test cache propagation and keep a bypass from becoming the undocumented disaster-recovery plan.

Configuration can show that fallback, budgets and revocation exist. A control plane must leave enough evidence for an operator to explain what each one did after the request is over.

Sources and reproducibility

1. EXP-04 results: gateway failover, budget and key-control measurements
2. LiteLLM documentation: fallbacks and provider failover
3. LiteLLM documentation: virtual keys and budget parameters
4. Microsoft Architecture Center: gateway topology, data sovereignty and failover
5. Grab Engineering: why a central AI gateway exists
6. Assembled: automated provider fallback during outages