Every company that ships LLM features eventually writes the same program: an HTTP front door, a list of model deployments, and a way to spread traffic across them without falling over. LiteLLM, whose README bills it in one line as an “Open Source AI Gateway for 100+ LLMs. Self-hosted. Enterprise-ready,” is that program industrialized - and its industrial heart is a single Python class, the Router, that has grown to 16,046 lines inside litellm/router.py. It sits beneath a 21,775-line FastAPI proxy in litellm/proxy/proxy_server.py, weighs requests against in-flight counters and rolling latency histograms, decides which of your thirty OpenAI-compatible endpoints gets the next token, and - when that endpoint 429s at 2 a.m. - quietly cools it down, retries elsewhere, and walks a fallback list to another model group entirely. We toured the gateway’s public face when this series first visited LiteLLM; this post goes one level down, into the machinery that decides who serves the next request.

As always in this series, what follows is an educational tour of published source code.

Overview: the FastAPI proxy authenticates and enriches each request; the Router filters cooldowns, consults a strategy selector, and wraps every call in retry and fallback layers; strategy handlers track in-flight requests and latency; failures feed the cooldown cache; the chosen deployment is bridged to the provider.

Reading the overview from left to right:

• litellm/proxy/proxy_server.py is the front door: proxy_startup_event at line 1399 wires Prisma, health checks, and the shared aiohttp session into globals, while the chat route at 12710 stamps key, team, and org ids into request metadata before handing off to ProxyBaseLLMRequestProcessing at 12759.
• litellm/proxy/auth/user_api_key_auth.py, 4,731 lines, is the dependency injected into every route: user_api_key_auth at line 4186 accepts six different credential headers at 4188-4193 and runs the whole auth phase inside an OpenTelemetry span opened at 4214-4216.
• litellm/router.py is the Router class: init at line 1113 accepts the model_list and per-group settings, with allowed_fails configurable at 1405 and a full AllowedFailsPolicy dataclass path at 1560-1567.
• acompletion at line 3127 is the async dispatch entry; it delegates to async_function_with_retries at 8537, whose first move at 8570 is a single make_call attempt and whose failure path re-fetches healthy deployments at 8592.
• litellm/router_strategy/least_busy.py implements least-busy routing in 225 lines: the logging handler increments a per-deployment counter before each call at 124-125 and decrements on success or failure at 128-141, then get_available_deployments at 160-173 reads the counts from Redis or the local cache and picks the minimum.
• litellm/router_strategy/lowest_latency.py, 555 lines, keeps a per-group latency map and, for streaming requests, switches to time-to-first-token percentiles at 451-456; deployments outside a TPM/RPM budget are dropped at 475-478, and the winner is chosen randomly among everything within a configurable buffer of the best at 492-498.
• litellm/router_utils/cooldown_handlers.py is the failure ledger: set_cooldown_deployments at line 498 decides whether a deployment earns a cooldown, writes it into the cache at 545-550, and fires a cooldown callback task at 553-560, while async_get_cooldown_deployments at 568 is what the Router consults before every selection.
• litellm/main.py, 9,294 lines, is the bridge the Router ultimately calls: completion at line 5328 and acompletion at 439 translate the unified OpenAI-style call into the 100+ provider-specific shapes, and litellm/cost_calculator.py prices the result with cost_per_token at line 405.

Why You Need This

A load balancer for LLMs is not a load balancer for web servers, and the Router is the clearest evidence in open source. Web requests are interchangeable; LLM requests differ by input length, so a deployment’s TPM budget is consumed at wildly different rates, and the “cost” of a failure includes all the prefill work the server already did. That is why every strategy in this codebase is a CustomLogger first and a selector second: the least-busy handler in least_busy.py only knows which deployment is free because the same class hooks log_pre_api_call at line 124 and log_success_event at 128 to maintain the counter. The latency handler likewise learns from completed calls, keeping its map keyed by model group in the shared cache. Compare that with the single-node engines this series has covered - vLLM schedules within one process and SGLang within one cluster - and the Router’s niche becomes obvious: it is the layer above them, treating each engine as a deployment with a budget and a mood.

The retry and cooldown machinery is worth reading on its own because it encodes a decade of production scar tissue. num_retries is not a single number: async_function_with_retries pops the request-level override at 8547, falls back to the Router default at 8549-8552, and then lets a per-deployment num_retries attached to the exception win at 8578-8585 - so a flaky endpoint can declare its own patience. Before any retry, the Router re-derives the healthy set at 8592 by subtracting the cooldown ledger from all deployments, an O(1) set filter at 9531-9534 rather than a loop. Cooldowns themselves are not automatic: set_cooldown_deployments consults the allowed-fails policy, and 401-style errors that no retry can fix go straight to cooldown, exactly as the docstring at 508-512 promises. If you have ever bolted tenacity onto an OpenAI client and wondered why your outage cascade got worse, this file is the missing syllabus.

Finally, the strategy dispatch shows how to make a 16,000-line class extensible without chaos. RoutingStrategy in litellm/types/router.py at line 1157 names exactly four built-in picks - least-busy, latency-based-routing, cost-based-routing, and usage-based-routing-v2 - with simple-shuffle as the default at line 139, and _select_deployment_async at 2164 is a plain match statement over those names at 2182-2215. New strategies register through routing_strategy_init at 1809, which builds a selector object and binds it to the Router. The lesson generalizes: enumerate the decisions you actually make, give each a tiny stateful module, and keep the dispatcher dumb. Teams that adopt LiteLLM behind Open WebUI or their own agents get this machinery for free; teams that hand-roll a gateway usually reimplement only the retry half and wonder why latency silently doubles during traffic spikes. And because every hook flows through CustomLogger - 1,137 lines of documented callbacks - wiring observability like Langfuse is a config entry, not a fork.

Detail: the proxy lifecycle builds the Router at startup; auth and budget gates run per request; the retry loop alternates between healthy-set filtering, strategy selection, cooldown writes, and single attempts through the provider bridge.

Start at the gateway, because everything else inherits its globals. proxy_startup_event at line 1399 declares sixteen module-level globals at 1400-1415 - prisma_client, master_key, llm_router, store_model_in_db among them - a deliberate acknowledgment that the proxy is one long-lived process per worker. It marks stale Prometheus multiprocess directories dead at 1424-1426, then runs any LITELLM_WORKER_STARTUP_HOOKS entries at 1429-1447, importing each “module.path:function_name” spec dynamically so operators can inject startup behavior without patching the image. The chat route at 12710 is thin by design: it reads the body, copies user_id, team_id, org_id, and agent_id from the authenticated key into the metadata dict at 12742-12757, and calls base_process_llm_request with route_type “acompletion” at 12761-12765. Everything that looks like business logic lives in the 21k-line file’s other 21k lines, which is exactly why the Router’s separateness matters - the same Router object serves the proxy, the SDK, and the CLI.

The selection moment deserves a slow-motion replay. After async_function_with_retries fetches the healthy list, _select_deployment_async at 2164 asks the strategy selector for a deployment, with simple-shuffle explicitly noted in the docstring at 2175-2178 as bypassing the selector machinery. For latency-based-routing, the case at 2206-2213 forwards request_kwargs, because only the caller knows whether this request streams - and that flag flips the handler to TTFT percentiles at 451-456, a detail that separates p95 prompt-processing pain from p95 time-to-first-word pain in chat UIs. The TPM/RPM guards at 475-478 reject deployments whose budgets the request would blow through, using the rpm and tpm fields any model_list entry may declare at 440-445. Then the buffer logic at 486-498 does something quietly wise: it sorts candidates by latency, keeps everything within lowest_latency_buffer of the best, and picks randomly among them - spreading load across near-equal deployments instead of hammering one winner into its own budget.

Failure handling is the second half of the same loop. When make_call at 8743 raises, the exception handler at 8573 re-enters the loop after three bookkeeping calls: _set_deployment_num_retries_on_exception at 4425 copies the deployment’s declared patience onto the error, failed-deployment stamping at 4444-4474 marks which model_info id failed so retries can skip it, and should-fail checks consult the allowed_fails counters configured back at init 1405. Cooldowns land in cooldown_cache.py - 237 lines that wrap either an in-memory TTL structure or Redis, which is what makes the whole scheme work across the proxy’s horizontal replicas. When an entire model group is exhausted, async_function_with_fallbacks at 8402 walks the configured fallback list, carrying partial usage forward so billing stays honest across the switch - the hidden-params plumbing at 3250-3282 exists precisely so a retried stream can adopt the previous attempt’s token counts.

The bridge and the bill close the tour. main.py’s completion at 5328 is where “100+ providers” becomes real: a giant dispatch by model prefix into per-provider implementations, each normalizing errors back into LiteLLM’s own exception taxonomy so the Router’s cooldown logic can treat a Bedrock ThrottlingException and an OpenAI 429 identically. cost_per_token at cost_calculator.py line 405 reads the model_prices_and_context_window.json registry that ships at the repo root - one JSON file, thousands of models, input and output rates per token - and completion_cost at 1419 assembles the final number that the proxy writes to spend logs per key, team, and user. That registry, updated by bot PRs almost daily, is the quiet moat of the project: routing, retries, and cooldowns are ideas, but knowing what a Llama 4 Maverick call costs on six clouds is data, and data compounds.

From Install to a Balanced Gateway

The README’s quickstart gets a two-deployment gateway balancing with one config (commands abridged):

pip install 'litellm[proxy]'
export LITELLM_MASTER_KEY=sk-1234
# config.yaml: two model_list entries sharing model_name "gpt-4o",
# one openai/, one azure/, plus:
#   router_settings:
#     routing_strategy: latency-based-routing
#     allowed_fails: 3
litellm --config config.yaml --port 4000
curl -X POST http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -d '{"model": "gpt-4o", "messages": [{"role": "user", "content": "hi"}]}'

Both entries answer to the same public name; the Router picks per request, and /health or the cooldown debug endpoint shows which deployment is currently in timeout.

Honest limits: the Router is big because the problem is big, but the size is real - 16,046 lines with streaming fallback edges (the FallbackAwareAnthropicMessagesStream at 955 exists because mid-stream provider deaths are genuinely hard), and you should not expect to internalize it in an afternoon; the in-memory cooldown and usage caches are per-process until you attach Redis, so a single-replica deployment gets weaker guarantees than the diagrams imply; strategy quality depends on honest logging, and silent callback failures degrade routing to shuffle without errors; budget limiter enforcement (941 lines in budget_limiter.py) checks at request time, not mid-stream, so one 100k-token runaway can still overshoot a TPM cap; and the Apache 2.0 license covers the core, with several enterprise features clearly fenced off in the enterprise directory.

Enjoyed this post? Never miss out on future posts by following us