Every serving stack we have toured owns its hardware. vLLM assumes a datacenter GPU, llama.cpp assumes your laptop, KTransformers assumes one machine with a big CPU and a modest GPU. Petals throws away that assumption entirely: it runs a single large model across dozens of unrelated machines — volunteers’ gaming rigs, lab servers — stitched together by a peer-to-peer distributed hash table. You call from_pretrained("meta-llama/Llama-2-70b") in PyTorch and get an ordinary-looking module whose layers live in other people’s VRAM. It is, quite literally, a crowd-funded datacenter with no funding.

The trick is architectural honesty about where things run. The client is a thin PyTorch shim: it builds a normal HF model class, replaces the block stack with remote proxies, and routes hidden states through whatever servers are alive. The server is a slice host: it picks a contiguous span of layers, downloads just those weights, quantizes them, and streams RPC inference to strangers while announcing its health to the DHT. Between them sits a routing layer that treats server churn — peers joining, leaving, getting slow — as a first-class problem, not an error case.

The map below follows one hidden state through that swarm: client to router, router through the DHT, session to server, and the server’s inner loop from RPC to cache to block.

Overview of petals: the client library assembles a distributed model whose blocks are remote; the sequence manager keeps a route from DHT records; on the other side, a volunteer server chooses its layer span, downloads the slice, and runs TransformerBackends behind an RPC handler with per-session KV cache allocation.

Reading the overview from left to right:

• The client pretends to be normal: FromPretrainedMixin, L17 patches HF loading, and RemoteSequential, L20 is the block stack reinterpreted as network calls.
• Routing is its own subsystem: RemoteSequenceManager, L59 consults the DHT layer (declare_active_modules, L28) and the records it stores (RemoteModuleInfo, L78).
• On the server, Server, L46 joins the DHT and announces, while choose_best_blocks, L28 picks which layers to host and load_pretrained_block, L35 fetches only that slice.
• Execution per block is a TransformerBackend, L24 with its own KV memory (MemoryCache, L26), driven by rpc_inference, L132.
• The live wire between the two sides is InferenceSession, L220, which holds one open stream per server on the route.

The Client: A Transformer Wearing a Mask

Petals’ client design starts from a clever observation: Hugging Face model classes already separate configuration from weights, and layers from each other. DistributedLlamaForCausalLM, L132 subclasses the real Llama implementation but mixes in the petals loaders, so the tokenizer, config, and generation loop all behave exactly as HF users expect. The block stack becomes RemoteSequential, L20, whose forward, L52 walks the route server by server, carrying hidden states and prompts between them.

Generation rides on top without forking the HF loop: RemoteGenerationMixin, L56 supplies a RemotePastKeyValues object that pretends to hold KV caches while they actually live on servers, and the final logits come from a locally owned LMHead, L23. Even training works: PTuneMixin, L21 trains soft-prompt tensors on the client — the servers stay frozen and never see gradients — and full backprop through the swarm is orchestrated by _RemoteSequentialAutogradFunction, L223, which streams tensors through run_remote_forward, L67 and replays them backward. Compare this to the tooling-first approach of ONNX: petals makes interoperability with the HF ecosystem the entire strategy.

One adapter directory per model family keeps the trick honest — Llama, Falcon, and Mixtral each get a few files that repeat the identical mixin pattern with no serving code duplicated. Supporting a new architecture is a client-side subclass plus a block wrapper, not a fork of the engine. That is the quiet dividend of building petals as a library rather than a service: the ecosystem grew with Hugging Face’s release cadence instead of waiting on the petals maintainers to catch up.

Routing: The Part That Actually Is Hard

Any distributed system’s awkward truth is that peers are flaky. Petals’ routing subsystem is the answer, and it is the repo’s most transferable reading. The DHT stores per-model records — ModelInfo, L21 declares the block count, each server publishes its ServerState, L33 (offline, downloading, reachable) and the spans it serves (compute_spans, L134). The client-side RemoteSequenceManager, L59 turns those records into a route via make_sequence, L138, tracks latency, and heals: on failure it re-queries the DHT and re-routes mid-generation, and on success it rebalances load through on_request_success, L403.

The routing internals reward a second look. Peer records collapse into RemoteSequenceInfo, L14, which keeps block ranges sorted by latency so the route builder can trade speed for load-balancing with a single threshold. Fresh spans arrive from the DHT as ordinary reads rather than a separate monitoring channel, so dead peers are pruned as a side effect of normal traffic, and the client never needs a central tracker to stay current.

The session is where routing meets execution. InferenceSession, L220 wraps a _ServerInferenceSession, L26 per server — one open RPC stream, one cache handle, one promise: your tokens keep flowing without re-sending the prefix. If a peer dies, the outer session migrates the request to a fresh route and the KV cache for past blocks is rebuilt by re-running the prompt. That’s the design bargain: peer churn costs a prompt replay, not a session crash.

The Volunteer Server: One Slice, Fully Owned

Server startup is a choreography worth reading in order. Server, L46 joins the DHT, benchmarks itself through get_server_throughput, L37 (compute and network, measured, not claimed), and asks the swarm where it’s most useful: choose_best_blocks, L28 solves a small optimization — pick a span whose minimum per-peer throughput is best for the whole chain — and periodically reconsiders via should_choose_other_blocks, L40. NAT traversal is validated before claiming anything (check_direct_reachability, L55), because a server behind a hostile firewall is worse than no server.

Then it downloads exactly its slice (load_pretrained_block, L35 pulls shard files and extracts one block’s state dict), converts it — convert_block, L25 with quantize_module, L76 producing 8-bit or 4-bit modules — and hands it to a TransformerBackend, L24. The announcement loop (ModuleAnnouncerThread, L674, republishing via announce, L754) keeps the DHT honest about what’s alive. This is server ownership inverted from every other stack we’ve toured: instead of one process owning a whole model like Triton Inference Server, each process owns a fragment and negotiates its place in the whole.

The Execution Loop: RPC to Cache to Block

The wire protocol is where performance lives. TransformerConnectionHandler, L55 exposes rpc_inference (L132) as a streamed RPC: the client pushes tensors for one micro-step, the server returns the updated hidden state plus cache handles, and the connection stays open for the whole sequence. Inside, iterate_rpc_inference, L144 is the decode loop — collect waiting requests, batch them, run the block once, ship results — and simple forward/backward calls (run_rpc_forward, L32) bypass the stateful machinery entirely.

Stateless traffic takes the short path. The same handler also serves plain forward and backward RPCs (rpc_forward, L352) for callers who don’t need a session — mostly training clients streaming gradients — and they share the backend machinery without touching the cache allocator. Between the two paths, a single server process can host interactive decoding, batched training traffic, and health probes simultaneously, each in its own queue and none able to starve the others.

KV memory is managed with a server-side allocator rather than per-request tensors: MemoryCache, L26 pre-allocates a big pool, allocate_cache, L72 reserves handles per session, and use_cache, L196 hands live tensors to the block for the step. The ideas rhyme with paged attention in vLLM — decouple logical sessions from physical memory — but the implementation is deliberately plain: a Python pool with refcounts, not a CUDA allocator. Batch assembly runs through PrioritizedTaskPool, L29 with pluggable priority (TaskPrioritizerBase, L6), and the decode step can be CUDA-graph captured (make_inference_graphed_callable, L5) to cut launch overhead. Speculative decoding closes the loop: DistributedLlamaForSpeculativeGeneration, L13 drafts locally with a small model and pays network round-trips only to verify.

From Install to Your First Model

Petals is a Python package — client and server in one — so the fast path is two terminals:

pip install petals  # client side
python - <<'PY'
import torch
from transformers import AutoTokenizer
from petals import DistributedLlamaForCausalLM

tok = AutoTokenizer.from_pretrained("petals-team/StableBeluga2")
model = DistributedLlamaForCausalLM.from_pretrained(
    "petals-team/StableBeluga2", torch_dtype=torch.float16)
inputs = tok("The swarm works because", return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=20)
print(tok.decode(out[0]))
PY

Serving a slice is one command (python -m petals.cli.run_server with a model name — main, L19 parses the flags) and the server quietly handles block choice, download, quantization, and announcement. Nothing in that flow needs a database, a scheduler, or an account — the DHT is the control plane, which is both the elegance and, as we’ll see, the constraint.

Honest Limits and Where Petals Fits

Limits first, because the design’s weaknesses flow directly from its strengths. Latency is the ceiling: every block boundary is a network hop, so single-digit-millisecond interconnect assumptions die here — petals is for throughput and access, not interactive speed, unless you draft locally and verify remotely. Privacy is structural: volunteers see the hidden states that cross their slices, which rules out anything confidential. The swarm’s capacity is a commons — it serves whatever models volunteers choose to host, and Popular-model churn is real. And the DHT itself, brilliant for zero administration, means no SLA, no quotas beyond politeness (spending policies exist but trust is still the base layer).

For fit, petals occupies a niche nobody else in our tours touches: it is the only stack where the fleet is a public good. Read it for the routing layer — genuinely the best reference code for churn-tolerant request routing — or to understand how far the PyTorch module abstraction can stretch before it tears. Every other inference catalog entry assumes you own the machines; petals assumes you don’t, and engineers accordingly — the same lesson IREE teaches about compilers, flipped from hardware heterogeneity to ownership heterogeneity.

Enjoyed this post? Never miss out on future posts by following us