Skip to content

Execution Flow & Internals

A frontend initializes one application context, obtains authorization, builds a live plan, and executes that same plan. Rendering, diagnostics and readiness policy belong to the frontend. See Python API for the difference between detached API submission and CLI completion.

The following is an illustrative container workflow, not a fixed six-phase contract for every executor or plugin strategy:

Resolve recipe and provenance
-> plan hosts, hardware, placement, executor destinations
-> authorize all hooks and mods for this launch
-> prepare container or native assets
-> distribute and verify required resources per host
-> synchronize applicable tuning/runtime caches
-> validate replacement and submit workload
-> observe readiness and run required hooks

CLI run and fresh benchmarks share placement/readiness presentation. A live plan avoids redundant scheduling; it does not prove that cluster capacity can never change between preview and execution. Reservation and ownership checks still apply.

The trust verdict is resolved once before preparation and before the replacement barrier can evict an existing deployment. Runtime plugins are shared instances; the launcher binds executors on an operation-local copy and passes per-host platform environment explicitly to prevent concurrent launches sharing state.

Resolve effective recipe/runtime overrides, source trust, launch intent, hosts, transport and executor target. Probe hardware and occupancy as needed. Schedulers produce RankAssignment (core/scheduler.py, schedulers/); explicit layouts come from core/layout.py. Do not use the removed core/placement.py path.

Hardware assessments distinguish unsupported/unknown fit from demonstrated capacity. DGX Spark scheduling and runtime allocation are separate policies. See Schedulers. Recipe mods expand to pre_exec before builder work, so resolution/trust failures surface early.

Builders implement prepare(..., builder_context=None) for image or host environment preparation. The default preparation leaves image refs unchanged; distribution owns registry pulls, with --rebuild requesting a fresh pull. Eugr can select a prebuilt image or perform an explicitly requested build. See Builders.

The shared image planner resolves per-node images from effective recipe settings and hardware. Builder output is authoritative. Prepared container distribution policies are passed to staging instead of mutating recipe declarations. Image-less native assets have no container image plan; native runs do not require an unused image default. Container-only materialize() rejects native targets.

Preparation binds immutable model revisions, selected files and workload-visible paths. Each required target is checked for completeness and access. Built-in local/push/delegated/pull paths and selected providers share those guarantees. A provider’s success receipt or byte counter is insufficient without per-host verification. OCI Relay remains bundled for images; other providers may be plugins. See Model Cache Management.

HF snapshot copies dereference selected shared-blob links into target files. Known damage, permissions, missing dependencies, and mount shadowing fail preparation rather than silently becoming success. Offline mode can copy existing cluster artifacts but does not call origins or install tools. Native and container executors must verify access under their actual workload identity.

Applicable registry tuning and target runtime compilation caches are separate from model payloads. --no-sync-tuning skips registry tuning synchronization; --no-runtime-cache has a different purpose. Native/delegating strategies may use different preparation paths.

Generate a nonempty serving command through the runtime’s required generate_command() hook and submit to the resolved executor destination. Plugin run handlers call before_start after validation/staging and immediately before submission so core can perform confirmed replacement on that exact target. Dry runs pass no launch callback and do not submit. Check public RunResult.rc, not an assumed private launch handle.

Native destinations need verified ownership/process identity and recovery state; metadata alone is not permission to kill a process or delete files. Kubernetes submission uses the optional integration’s own target and reservation model.

Default CLI completion waits for readiness and required hooks. Supported Docker observation measures port-open, HTTP-ready, and first generated token from the serving container start. Endpoint readiness, request latency, and total CLI wall time are distinct. Unsupported observers omit timing rather than inventing it.

--no-follow waits without model logs; --no-ready-wait skips observation but conflicts with required post hooks; --follow keeps watching after readiness. A failed wait can leave detached inference running. Ctrl+C cancels observers, not the detached deployment. See run.

Recipes should use runtime: vllm with defaults.distributed_executor_backend omitted or set to mp for native distributed execution, or set to ray for Ray. The names below identify the concrete implementations selected by that alias.

vllm-distributed uses vLLM’s built-in multi-node support. Each node runs the full vllm serve command with node-specific arguments:

  1. Cleanup — remove existing containers for this cluster ID on all hosts
  2. InfiniBand detection — detect IB interfaces and compute NCCL environment variables
  3. Head node IP detection — determine the head node’s management IP
  4. Launch head node (rank 0) with --nnodes, --node-rank 0, and --master-addr
  5. Wait for master port to become ready
  6. Launch worker nodes in parallel, each with --node-rank N and --headless

No Ray cluster is involved — vLLM coordinates the nodes directly.

vllm-ray forms a Ray cluster first, then runs the serve command on the head node:

  1. Cleanup — remove existing containers for this cluster ID on all hosts
  2. InfiniBand detection — detect IB interfaces and compute NCCL environment variables
  3. Start Ray head node on the first host
  4. Join workers to the Ray cluster
  5. Launch vLLM serve on the head container

SGLang uses its native distributed backend. sparkrun launches the head node with --dist-init-addr, --nnodes, and --node-rank 0, then launches workers in parallel with their respective --node-rank values.

TRT-LLM uses a custom MPI-based approach. Instead of installing openssh-server inside containers, sparkrun generates a custom mpirun rsh agent that routes through host-level SSH and docker exec:

  1. Cleanup — remove existing containers for this cluster ID on all hosts
  2. InfiniBand detection — detect IB interfaces and compute NCCL environment variables
  3. IP detection — detect management IPs on all hosts for MPI host list
  4. Container launch — launch containers with sleep infinity on all hosts in parallel
  5. Verify — confirm all containers are running
  6. Write rsh wrapper + config — generate the MPI rsh wrapper script and extra LLM API config YAML, write both into the head container
  7. Exec mpirun — execute mpirun on the head container with trtllm-llmapi-launch wrapping the trtllm-serve command

The rsh wrapper script maps each host IP to its container name, then uses ssh <host> docker exec <container> <command> to route MPI commands into the correct worker container.

sparkrun sets these environment variables automatically for TRT-LLM clusters:

VariableValuePurpose
OMPI_ALLOW_RUN_AS_ROOT1Allow mpirun as root
OMPI_ALLOW_RUN_AS_ROOT_CONFIRM1Confirm root execution
NCCL_CUMEM_ENABLE0Disable NCCL cuMem (compatibility)
OMPI_MCA_rmaps_ppr_n_pernode1One process per node

NCCL-related variables (NCCL_SOCKET_IFNAME, UCX_NET_DEVICES, etc.) and HF_TOKEN are also propagated to all nodes via mpirun -x.

Runtime preparation can declare additional resources such as draft models before staging. Recipe pre_exec runs before the serve command; mods feed the same path. post_exec runs inside the head workload after readiness. Registry post_commands require the applicable source authorization and run in their specified context. Hook completion is part of the default CLI success boundary. Use the public feedback interface to report stages, hosts, counters and outcomes; activity success is not operation success.