Execution Flow & Internals
A frontend initializes one application context, obtains authorization, builds a live plan, and executes that same plan. Rendering, diagnostics and readiness policy belong to the frontend. See Python API for the difference between detached API submission and CLI completion.
Launch lifecycle
Section titled “Launch lifecycle”The following is an illustrative container workflow, not a fixed six-phase contract for every executor or plugin strategy:
Resolve recipe and provenance -> plan hosts, hardware, placement, executor destinations -> authorize all hooks and mods for this launch -> prepare container or native assets -> distribute and verify required resources per host -> synchronize applicable tuning/runtime caches -> validate replacement and submit workload -> observe readiness and run required hooksCLI run and fresh benchmarks share placement/readiness presentation. A live plan
avoids redundant scheduling; it does not prove that cluster capacity can never
change between preview and execution. Reservation and ownership checks still apply.
The trust verdict is resolved once before preparation and before the replacement barrier can evict an existing deployment. Runtime plugins are shared instances; the launcher binds executors on an operation-local copy and passes per-host platform environment explicitly to prevent concurrent launches sharing state.
Phase 1: Preparing
Section titled “Phase 1: Preparing”Resolve effective recipe/runtime overrides, source trust, launch intent, hosts,
transport and executor target. Probe hardware and occupancy as needed. Schedulers
produce RankAssignment (core/scheduler.py, schedulers/); explicit layouts
come from core/layout.py. Do not use the removed core/placement.py path.
Hardware assessments distinguish unsupported/unknown fit from demonstrated
capacity. DGX Spark scheduling and runtime allocation are separate policies.
See Schedulers. Recipe mods expand to pre_exec before
builder work, so resolution/trust failures surface early.
Phase 2: Building image
Section titled “Phase 2: Building image”Builders implement prepare(..., builder_context=None) for image or host
environment preparation. The default preparation leaves image refs unchanged;
distribution owns registry pulls, with --rebuild requesting a fresh pull.
Eugr can select a prebuilt image or perform an explicitly requested build.
See Builders.
The shared image planner resolves per-node images from effective recipe settings
and hardware. Builder output is authoritative. Prepared container distribution
policies are passed to staging instead of mutating recipe declarations.
Image-less native assets have no container image plan; native runs do not require
an unused image default. Container-only materialize() rejects native targets.
Phase 3: Distributing resources
Section titled “Phase 3: Distributing resources”Preparation binds immutable model revisions, selected files and workload-visible paths. Each required target is checked for completeness and access. Built-in local/push/delegated/pull paths and selected providers share those guarantees. A provider’s success receipt or byte counter is insufficient without per-host verification. OCI Relay remains bundled for images; other providers may be plugins. See Model Cache Management.
HF snapshot copies dereference selected shared-blob links into target files. Known damage, permissions, missing dependencies, and mount shadowing fail preparation rather than silently becoming success. Offline mode can copy existing cluster artifacts but does not call origins or install tools. Native and container executors must verify access under their actual workload identity.
Phase 4: Syncing tuning configs
Section titled “Phase 4: Syncing tuning configs”Applicable registry tuning and target runtime compilation caches are separate from
model payloads. --no-sync-tuning skips registry tuning synchronization;
--no-runtime-cache has a different purpose. Native/delegating strategies may use
different preparation paths.
Phase 5: Launching runtime
Section titled “Phase 5: Launching runtime”Generate a nonempty serving command through the runtime’s required
generate_command() hook and submit to the resolved executor destination.
Plugin run handlers call before_start after validation/staging and immediately
before submission so core can perform confirmed replacement on that exact target.
Dry runs pass no launch callback and do not submit. Check public RunResult.rc,
not an assumed private launch handle.
Native destinations need verified ownership/process identity and recovery state; metadata alone is not permission to kill a process or delete files. Kubernetes submission uses the optional integration’s own target and reservation model.
Phase 6: Post-launch hooks
Section titled “Phase 6: Post-launch hooks”Default CLI completion waits for readiness and required hooks. Supported Docker observation measures port-open, HTTP-ready, and first generated token from the serving container start. Endpoint readiness, request latency, and total CLI wall time are distinct. Unsupported observers omit timing rather than inventing it.
--no-follow waits without model logs; --no-ready-wait skips observation but
conflicts with required post hooks; --follow keeps watching after readiness.
A failed wait can leave detached inference running. Ctrl+C cancels observers,
not the detached deployment. See run.
Runtime orchestration internals
Section titled “Runtime orchestration internals”Recipes should use runtime: vllm with defaults.distributed_executor_backend
omitted or set to mp for native distributed execution, or set to ray for Ray.
The names below identify the concrete implementations selected by that alias.
vllm-distributed multi-node
Section titled “vllm-distributed multi-node”vllm-distributed uses vLLM’s built-in multi-node support. Each node runs the full vllm serve command with node-specific arguments:
- Cleanup — remove existing containers for this cluster ID on all hosts
- InfiniBand detection — detect IB interfaces and compute NCCL environment variables
- Head node IP detection — determine the head node’s management IP
- Launch head node (rank 0) with
--nnodes,--node-rank 0, and--master-addr - Wait for master port to become ready
- Launch worker nodes in parallel, each with
--node-rank Nand--headless
No Ray cluster is involved — vLLM coordinates the nodes directly.
vllm-ray multi-node
Section titled “vllm-ray multi-node”vllm-ray forms a Ray cluster first, then runs the serve command on the head node:
- Cleanup — remove existing containers for this cluster ID on all hosts
- InfiniBand detection — detect IB interfaces and compute NCCL environment variables
- Start Ray head node on the first host
- Join workers to the Ray cluster
- Launch vLLM serve on the head container
SGLang multi-node
Section titled “SGLang multi-node”SGLang uses its native distributed backend. sparkrun launches the head node with --dist-init-addr, --nnodes, and --node-rank 0, then launches workers in parallel with their respective --node-rank values.
TensorRT-LLM multi-node (MPI)
Section titled “TensorRT-LLM multi-node (MPI)”TRT-LLM uses a custom MPI-based approach. Instead of installing openssh-server inside containers, sparkrun generates a custom mpirun rsh agent that routes through host-level SSH and docker exec:
- Cleanup — remove existing containers for this cluster ID on all hosts
- InfiniBand detection — detect IB interfaces and compute NCCL environment variables
- IP detection — detect management IPs on all hosts for MPI host list
- Container launch — launch containers with
sleep infinityon all hosts in parallel - Verify — confirm all containers are running
- Write rsh wrapper + config — generate the MPI rsh wrapper script and extra LLM API config YAML, write both into the head container
- Exec mpirun — execute
mpirunon the head container withtrtllm-llmapi-launchwrapping thetrtllm-servecommand
The rsh wrapper script maps each host IP to its container name, then uses ssh <host> docker exec <container> <command> to route MPI commands into the correct worker container.
TRT-LLM environment variables
Section titled “TRT-LLM environment variables”sparkrun sets these environment variables automatically for TRT-LLM clusters:
| Variable | Value | Purpose |
|---|---|---|
OMPI_ALLOW_RUN_AS_ROOT | 1 | Allow mpirun as root |
OMPI_ALLOW_RUN_AS_ROOT_CONFIRM | 1 | Confirm root execution |
NCCL_CUMEM_ENABLE | 0 | Disable NCCL cuMem (compatibility) |
OMPI_MCA_rmaps_ppr_n_pernode | 1 | One process per node |
NCCL-related variables (NCCL_SOCKET_IFNAME, UCX_NET_DEVICES, etc.) and HF_TOKEN are also propagated to all nodes via mpirun -x.
Recipe hooks
Section titled “Recipe hooks”Runtime preparation can declare additional resources such as draft models before
staging. Recipe pre_exec runs before the serve command; mods feed the same path.
post_exec runs inside the head workload after readiness. Registry
post_commands require the applicable source authorization and run in their
specified context. Hook completion is part of the default CLI success boundary.
Use the public feedback interface to report
stages, hosts, counters and outcomes; activity success is not operation success.