Runtimes Overview
sparkrun supports multiple inference runtimes through a plugin system. Each runtime knows how to launch a specific inference engine, configure networking, and manage containers.
Runtime comparison
Section titled “Runtime comparison”| Runtime | Engine | Multi-Node | Use Case |
|---|---|---|---|
vllm | vLLM | Yes (native or Ray) | Recommended recipe name. Defaults to distributed/mp; set defaults.distributed_executor_backend: ray for Ray. |
sglang | SGLang | Yes (native) | High-throughput inference, structured generation, experimental GGUF |
llama-cpp | llama.cpp | Experimental | GGUF quantized models, lightweight deployment |
trtllm | TensorRT-LLM | Yes (MPI) | NVIDIA TensorRT-LLM with MPI-based multi-node orchestration |
atlas | Atlas | Yes (native) | Pure-Rust inference server with native NCCL distribution. Composes tensor and expert parallelism. |
modular-max | Modular MAX | No | Single-node only. tensor_parallel maps to local --devices. |
Automatic runtime detection
Section titled “Automatic runtime detection”Explicitly setting runtime in recipes is recommended, but when it is omitted sparkrun can infer the runtime from the command field prefix:
| Command prefix | Detected runtime |
|---|---|
vllm serve ... | vllm |
sglang serve ... | sglang |
python[3] -m sglang.launch_server ... | sglang |
llama-server ... | llama-cpp |
trtllm-serve ... | trtllm |
mpirun...trtllm | trtllm |
If the command doesn’t match any known pattern (or there is no command), the recipe falls through to vllm behavior. An explicit runtime field always takes precedence over command-based detection.
See the recipe format docs for the full detection rules.
The vllm alias
Section titled “The vllm alias”For vLLM recipes, prefer runtime: vllm and set the backend through
defaults.distributed_executor_backend:
| Backend setting | Resolved runtime |
|---|---|
Omitted or mp | vllm-distributed — native distributed/multiprocessing (default) |
ray | vllm-ray — Ray head/worker orchestration |
runtime: vllmdefaults: distributed_executor_backend: ray # use mp, or omit, for native distributedThe concrete runtime names remain supported, but using the alias keeps backend selection in defaults. Legacy command templates can also supply a Ray hint; see vLLM backend selection for overrides.
Both vllm-distributed and vllm-ray use the same container images and support the same vLLM flags — they differ only in how multi-node clustering is orchestrated.
To add an engine, see Runtime Implementations.
Recipes without command templates
Section titled “Recipes without command templates”Prefer explicit runtime and supported defaults, allowing sparkrun to generate
the engine command. Use command only for unusual flags absent from its mapping.
Each reference lists the generated flags, their recipe keys, boolean behavior,
and settings managed by placement:
The engine image must support the requested flags. A custom command replaces normal flag generation, so it must contain the serving options you need. See Recipe Format for template behavior.
How runtimes work
Section titled “How runtimes work”Recipes declare which runtime to use via the runtime field. The runtime handles:
- Container launch configuration (volumes, networking, GPU access)
- InfiniBand/RDMA detection and NCCL environment variables
- Multi-node clustering (native distributed for
vllm-distributedandsglang, Ray forvllm-ray, MPI fortrtllm) - Model pre-sync and cache path resolution
- Command generation when no explicit
commandtemplate is provided
Choosing a runtime
Section titled “Choosing a runtime”- vLLM is the default and most widely supported. Use
runtime: vllm; omitdefaults.distributed_executor_backendor set it tompfor native distributed execution. Set it toraywhen you need Ray orchestration. See thevllmalias. - SGLang offers competitive performance, good structured generation support, and experimental GGUF quantized model serving.
- llama.cpp is ideal for GGUF quantized models or when you want a lightweight alternative.
- TensorRT-LLM provides NVIDIA’s optimized inference engine with MPI-based multi-node orchestration.
- Atlas is a pure-Rust inference server (see Atlas on GitHub) with native NCCL distribution. It composes tensor parallelism (
tensor_parallel) and expert parallelism (ep_size) on either an overlapping mesh (tp == ep) or an orthogonal one (tp * ep), and derives the world size from the two. - Modular MAX is single-node only:
tensor_parallelmaps to local--devicesrather than to host count, and the runtime refuses a multi-host placement rather than silently truncating it.
Maintained container images
Section titled “Maintained container images”The Spark Arena team maintains several specialized container images for DGX Spark.
| Image | Description | Link |
|---|---|---|
ghcr.io/spark-arena/dgx-vllm-eugr-nightly | Optimized eugr vLLM for DGX Spark | GitHub |
ghcr.io/spark-arena/dgx-vllm-eugr-nightly-tf5 | Optimized eugr vLLM for DGX Spark with Transformers 5.x | GitHub |
dgx-spark-sglang | Optimized SGLang for DGX Spark | Link |
ghcr.io/spark-arena/dgx-llama-cpp | Optimized llama.cpp for DGX Spark Mirror of scitrera/dgx-spark-llama-cpp | Docker GitHub |
Startup and cache policy
Section titled “Startup and cache policy”Default run completes after declared readiness and required hooks; timing and
inference probes depend on the runtime/executor’s capabilities. Unsupported
observers do not report fabricated startup metrics. Explicit recipe memory
settings override serving defaults and remain distinct from scheduling reservations.
Model manifests and target runtime compilation caches have separate validation
and cleanup policies. See run, Schedulers,
and Model Cache Management. Example image tags and
recipe values on this page are illustrative, not installation-wide defaults.