Runtimes Overview

sparkrun supports multiple inference runtimes through a plugin system. Each runtime knows how to launch a specific inference engine, configure networking, and manage containers.

Runtime comparison

Runtime	Engine	Multi-Node	Use Case
vllm-distributed	vLLM	Yes (native)	Default vLLM variant. Uses vLLM’s built-in distributed backend.
vllm-ray	vLLM	Yes (Ray)	vLLM with Ray head/worker orchestration.
sglang	SGLang	Yes (native)	High-throughput inference, structured generation, experimental GGUF
llama-cpp	llama.cpp	Experimental	GGUF quantized models, lightweight deployment
trtllm	TensorRT-LLM	Yes (MPI)	NVIDIA TensorRT-LLM with MPI-based multi-node orchestration
atlas	Atlas	Yes (native)	Pure-Rust inference server with native NCCL distribution. Composes tensor and expert parallelism.
modular-max	Modular MAX	No	Single-node only. tensor_parallel maps to local --devices.

Note

The eugr-vllm runtime has been replaced by the eugr builder plugin, which integrates eugr/spark-vllm-docker container builds directly into the recipe pipeline. Recipes using ghcr.io/spark-arena/dgx-vllm-eugr-nightly images automatically use the eugr builder with vllm-distributed or vllm-ray as the runtime.

Automatic runtime detection

Explicitly setting runtime in recipes is recommended, but when it is omitted sparkrun can infer the runtime from the command field prefix:

Command prefix	Detected runtime
vllm serve ...	vllm
sglang serve ...	sglang
python[3] -m sglang.launch_server ...	sglang
llama-server ...	llama-cpp
trtllm-serve ...	trtllm
mpirun...trtllm	trtllm

If the command doesn’t match any known pattern (or there is no command), the recipe falls through to vllm behavior. An explicit runtime field always takes precedence over command-based detection.

See the recipe format docs for the full detection rules.

The vllm alias

You can write runtime: vllm in a recipe (or let it be auto-detected from a vllm serve command) — sparkrun treats it as a virtual alias and resolves it to a concrete runtime:

1. If the recipe hints at Ray usage (distributed_executor_backend: ray in defaults, or --distributed-executor-backend ray in the command template) → vllm-ray
2. Otherwise → vllm-distributed (the default)

Both vllm-distributed and vllm-ray use the same container images and support the same vLLM flags — they differ only in how multi-node clustering is orchestrated.

How runtimes work

Recipes declare which runtime to use via the runtime field. The runtime handles:

• Container launch configuration (volumes, networking, GPU access)
• InfiniBand/RDMA detection and NCCL environment variables
• Multi-node clustering (native distributed for vllm-distributed and sglang, Ray for vllm-ray, MPI for trtllm)
• Model pre-sync and cache path resolution
• Command generation when no explicit command template is provided

Choosing a runtime

• vLLM is the default and most widely supported. Write runtime: vllm and sparkrun picks the right variant automatically (see the vllm alias above).
  • vllm-distributed (default) uses vLLM’s built-in distributed backend — each node runs the full serve command with node-rank arguments (--nnodes, --node-rank, --master-addr).
  • vllm-ray uses Ray head/worker orchestration — a Ray cluster is formed first, then the serve command runs on the head node.
• SGLang offers competitive performance, good structured generation support, and experimental GGUF quantized model serving.
• llama.cpp is ideal for GGUF quantized models or when you want a lightweight alternative.
• TensorRT-LLM provides NVIDIA’s optimized inference engine with MPI-based multi-node orchestration.
• Atlas is a pure-Rust inference server (see Atlas on GitHub) with native NCCL distribution. It composes tensor parallelism (tensor_parallel) and expert parallelism (ep_size) on either an overlapping mesh (tp == ep) or an orthogonal one (tp * ep), and derives the world size from the two.
• Modular MAX is single-node only: tensor_parallel maps to local --devices rather than to host count, and the runtime refuses a multi-host placement rather than silently truncating it.

Maintained container images

The Spark Arena team maintains several specialized container images for DGX Spark.

Image	Description	Link
ghcr.io/spark-arena/dgx-vllm-eugr-nightly	Optimized eugr vLLM for DGX Spark	GitHub
ghcr.io/spark-arena/dgx-vllm-eugr-nightly-tf5	Optimized eugr vLLM for DGX Spark with Transformers 5.x	GitHub
dgx-spark-sglang	Optimized SGLang for DGX Spark	Link
ghcr.io/spark-arena/dgx-llama-cpp	Optimized llama.cpp for DGX Spark Mirror of scitrera/dgx-spark-llama-cpp	Docker GitHub

Sentinel images

The two dgx-vllm-eugr-nightly* names above are sentinel images: at the :latest tag, sparkrun’s eugr builder resolves them to the authoritative published nightly and pulls it, so you get a stable name that tracks upstream eugr/spark-vllm-docker without managing tags yourself. Any other tag is pulled as-is. Since July 2026 this is a pull, not a local rebuild — building from upstream wheels is opt-in via build_args: ["--use-wheels"]. See vLLM › Sentinel images.