Our first visit to Xinference at /xinference/ walked the wide road: actors, the REST surface, the model catalog, and how one command becomes a cluster. This second tour goes down the narrow stairwell that first visit skipped: placement, the machinery that decides which machine and which GPU a model lands on, and what happens between “launch this model” and “this model is serving requests on device three.” It is the part of the system an operator never sees and a contributor most often has to reason about, and in the current tree it has grown teeth: a formal replica-config validation pass, a pluggable launch strategy that balances GPUs by live allocation snapshots, reservation-before-download ordering on the worker, orphan-GPU reaping, and a prefill/decode disaggregation topology for the vLLM engine. As always, this is an educational walk through the published source of xorbitsai/inference under the Apache-2.0 license.

The cast of files is smaller than the first tour’s, which makes the story easier to hold. The supervisor actor remains the cluster brain at over three hundred kilobytes of source, but placement now has its own vocabulary: launch_strategy.py is 137 lines, status_guard.py is a stateless actor of replica states, replica_config.py carries per-replica placement requests, and the cache ledger lives in cache_tracker.py. The worker actor at over six thousand lines does the device arithmetic. Read them in that order and the whole flow from request to resident model falls out naturally.

Placement from thirty thousand feet: REST requests reach the supervisor, which validates replica configs, consults the launch strategy over allocation snapshots, marks states in the status guard, and dispatches to workers that reserve devices, prepare environments, and report into the cache ledger and PD topology.

Reading the overview from left to right:

• The REST layer at xinference/api/restful_api.py accepts launch and scale-up requests and forwards them into the supervisor actor.
• _resolve_replica_config validates every replica’s worker_ip and gpu_idx before anything launches.
• The IdleFirstLaunchStrategy picks the least-loaded GPU slots from per-worker allocation snapshots.
• StatusGuardActor records each replica’s transition through CREATING and RUNNING.
• The worker’s allocate_devices performs the final idle-first device selection and records the reservation.
• PDModelActor holds the prefill and decode halves of a disaggregated vLLM deployment.
• CacheTrackerActor keeps the version ledger that lets operators list and reclaim disk used by stale weights.
• The router orchestration controller at router_orchestration.py manages edge-router nodes with heartbeats and assignment snapshots.

Validation before vibration

The first thing placement does is refuse bad placements, and it does so before touching a single device. _resolve_replica_config runs entirely before any replica launches, and its docstring states the contract plainly: a bad config fails cleanly, with no partial deployment and no leaked actors. The checks are layered. Worker addresses must resolve to registered workers, with a deliberate exemption for local deployments where a stale ip is downgraded to a warning. Then the supervisor gathers GPU allocation snapshots from every distinct target worker in parallel at line 2675, and uses them to verify that requested GPU indexes actually exist on the target and that two replicas of the same model are not silently handed the same card on workers that disallow sharing, raising a conflict error at line 2718 that names both the duplicate indexes and the policy. The pattern is worth copying in any orchestrator: turn what used to be a runtime surprise partway through a long launch into a typed error at admission time.

Scale-up of a running model flows through _add_model_replica, and it reads like a checklist someone wrote after being burned. A PD topology cannot be resized in place, so that errors first. The new replica id is the max of active ids plus one, and an idempotency check rejects duplicates. The original launch arguments are retrieved from any surviving replica via get_launch_args, then eligibility gates at lines 5264 through 5275 refuse scale-up for weight-cached models, Xavier-distributed models, and sharded multi-worker models, each with its own reason. Only then is a target chosen, either from an explicit replica config or through the strategy, a subtlety about legacy n_gpu=”auto” placements is normalized, and the new replica is marked CREATING in the status guard before the slow RPC, so dashboards never show a phantom gap. The launch and its wait_for_load close the sequence, with best-effort cleanup if either half fails.

Around that checklist sits a small state machine that makes the whole thing observable. StatusGuardActor is deliberately stateless in the Xoscar sense, a plain holder of replica rows with update_replica_status and get_replica_statuses as its main verbs, plus instance info keyed by model uid for the dashboard’s per-model counts. Every transition the placement machinery makes is written here first and read everywhere else, which is why the scale-up path marks CREATING before the slow RPC: any consumer, human or automated, sees intent rather than silence. On top of that sits the autostart subsystem in autostart.py, which persists launch rules and reapplies them when the cluster’s shape changes, so a model that should always be running comes back after a worker rejoins without anyone re-typing a curl command. The division of labor is worth noting: the supervisor decides, the strategy suggests, the worker executes, and the status guard remembers.

The idle-first arithmetic

Placement strategy proper lives in the 137 lines of launch_strategy.py. The abstract base asks for exactly one method: given worker candidates and a requested GPU count, return a worker reference and a list of device indexes. The concrete IdleFirstLaunchStrategy shuffles the candidates to avoid herding every launch onto the alphabetically first worker, then calls _select_least_loaded_gpu: for each worker it scores every device by how many models already sit on it, sorts by load with the device index as a stable tiebreaker, takes the n least-loaded, and compares workers by the sum of those loads, with the worker address breaking ties so the result is reproducible in logs. The charming detail is _reserve_slot: after picking, the strategy appends a literal “reserved” marker into its local snapshot, so a two-replica launch does not hand both replicas the same idle card before either has actually started. Fallbacks are explicit: no allocation snapshot means least-loaded-worker by count, stable by address.

The worker actor has the final word with allocate_devices, and the two-tier policy is worth a close read. If the operator has disabled multi-replica-per-GPU, only currently idle devices are eligible, the request fails with “No available slot found” rather than co-locating, and each selection is recorded immediately. Under the default multi-tenant policy the selection loop at lines 2186 through 2201 sorts devices by load with running model processes as a secondary key, picks the head, increments its load in place, and repeats, so two GPUs each hosting one model are preferred over one GPU hosting two. User-pinned placements route through allocate_devices_with_gpu_idx, which validates the indexes against what the worker can actually see and warns when a requested card is already occupied.

Ordering is the quiet hero of the launch path. launch_builtin_model begins by snapshotting its own arguments into a dict at line 4626, because auto-recovery after a worker crash re-invokes the function with the same payload, and any parameter read before that line is a recovery bug waiting to happen. Then comes the comment block at lines 4842 through 4871 that explains why devices are reserved before the model download and virtualenv preparation: the reservation is recorded synchronously, so concurrent launches up to the configured cap see each other’s picks for idle-first placement instead of all racing off the same stale snapshot after their prep phases finish. A second comment documents the opposite ordering constraint, that the model subprocess must not spawn before its virtualenv is populated, since numpy and torch imported at bootstrap get cached in sys.modules and a later install cannot replace them. These two paragraphs of commentary are a free masterclass in why launch sequences are ordered the way they are. Progress itself is a two-stage tracker split at line 4917 with the download weighted at 0.8 and model load carrying the rest.

Disaggregation, reaping, and the ledger

Two newer subsystems round out the story. The first is prefill/decode disaggregation for vLLM deployments, where prompt processing and token generation run on separate pools of devices. PDModelActor is the registry: it holds prefill and decode actor references per replica uid, with add_prefill_actor and add_decode_actor wiring the halves together and symmetric removals for teardown, while the scheduling policy abstraction at the top of the file defaults to round-robin across replicas. The supervisor registers these replicas at register_pd_replica, and the placement rules from earlier in this post are exactly what decides which machines host which half.

The second is hygiene. GPU processes that outlive their actors are a classic operator support ticket, so the worker snapshots occupying PIDs per device and kills orphans both at startup and during teardown, with a parent-pid variant for stray children. GPU memory reports from model subprocesses flow into the supervisor, which expires stale entries via _expire_worker_model_gpu_memory so a crashed worker cannot inflate the cluster’s view of itself forever. And the disk-side ledger, CacheTrackerActor, records every model version per worker address, answers list_deletable_models by consulting who still references a version, and only then removes artifacts. Placement, hygiene, and accounting are the same problem seen from three angles, and this codebase treats them that way.

Two supporting systems make the rest of the machine livable. The first is environment isolation: engine dependencies are famously incompatible, a vLLM release pinning one transformers version and a custom Python backend needing another, and virtual_env_manager.py gives each model its own prepared environment so installs cannot poison each other, with the launch flow ordering spawn after preparation for the reasons quoted earlier. The second is measurement: gather_node_info samples CPU through psutil and every GPU’s memory and utilization through the device utilities layer, and those readings feed both the web dashboard’s node views and the placement decisions described above. It is a reminder that the glamorous part of a serving system, the arithmetic that picks a device, is only as good as the boring part that keeps the inputs honest, current, and isolated from version drift.

The detail view: the supervisor's validation and scale-up checklist, the strategy's snapshot arithmetic, the worker's reservation-before-download ordering with virtualenv and orphan guards, and the ledgers for PD topology, router nodes, and cached model versions.

The lesson to carry away is that placement is a protocol, not a function. A request passes through validation, strategy, reservation, and confirmation as four distinct, individually testable stages, each with typed failures, and nothing is trusted twice: the strategy’s snapshot is advisory, the worker’s device table is authoritative, and the status guard is the single place a replica’s truth lives. If you enjoyed the cluster-level story in /xinference/, pair this one with the router internals in /litellm-2/ and the admission machinery in /text-generation-inference/ to see how other servers answer the same “where does this model live, and who is allowed to put it there” question.

Enjoyed this post? Never miss out on future posts by following us