Skip to content

Runtimes Overview

sparkrun supports multiple inference runtimes through a plugin system. Each runtime knows how to launch a specific inference engine, configure networking, and manage containers.

RuntimeEngineMulti-NodeUse Case
vllmvLLMYes (native or Ray)Recommended recipe name. Defaults to distributed/mp; set defaults.distributed_executor_backend: ray for Ray.
sglangSGLangYes (native)High-throughput inference, structured generation, experimental GGUF
llama-cppllama.cppExperimentalGGUF quantized models, lightweight deployment
trtllmTensorRT-LLMYes (MPI)NVIDIA TensorRT-LLM with MPI-based multi-node orchestration
atlasAtlasYes (native)Pure-Rust inference server with native NCCL distribution. Composes tensor and expert parallelism.
modular-maxModular MAXNoSingle-node only. tensor_parallel maps to local --devices.

Explicitly setting runtime in recipes is recommended, but when it is omitted sparkrun can infer the runtime from the command field prefix:

Command prefixDetected runtime
vllm serve ...vllm
sglang serve ...sglang
python[3] -m sglang.launch_server ...sglang
llama-server ...llama-cpp
trtllm-serve ...trtllm
mpirun...trtllmtrtllm

If the command doesn’t match any known pattern (or there is no command), the recipe falls through to vllm behavior. An explicit runtime field always takes precedence over command-based detection.

See the recipe format docs for the full detection rules.

For vLLM recipes, prefer runtime: vllm and set the backend through defaults.distributed_executor_backend:

Backend settingResolved runtime
Omitted or mpvllm-distributed — native distributed/multiprocessing (default)
rayvllm-ray — Ray head/worker orchestration
runtime: vllm
defaults:
distributed_executor_backend: ray # use mp, or omit, for native distributed

The concrete runtime names remain supported, but using the alias keeps backend selection in defaults. Legacy command templates can also supply a Ray hint; see vLLM backend selection for overrides.

Both vllm-distributed and vllm-ray use the same container images and support the same vLLM flags — they differ only in how multi-node clustering is orchestrated.

To add an engine, see Runtime Implementations.

Prefer explicit runtime and supported defaults, allowing sparkrun to generate the engine command. Use command only for unusual flags absent from its mapping. Each reference lists the generated flags, their recipe keys, boolean behavior, and settings managed by placement:

The engine image must support the requested flags. A custom command replaces normal flag generation, so it must contain the serving options you need. See Recipe Format for template behavior.

Recipes declare which runtime to use via the runtime field. The runtime handles:

  • Container launch configuration (volumes, networking, GPU access)
  • InfiniBand/RDMA detection and NCCL environment variables
  • Multi-node clustering (native distributed for vllm-distributed and sglang, Ray for vllm-ray, MPI for trtllm)
  • Model pre-sync and cache path resolution
  • Command generation when no explicit command template is provided
  • vLLM is the default and most widely supported. Use runtime: vllm; omit defaults.distributed_executor_backend or set it to mp for native distributed execution. Set it to ray when you need Ray orchestration. See the vllm alias.
  • SGLang offers competitive performance, good structured generation support, and experimental GGUF quantized model serving.
  • llama.cpp is ideal for GGUF quantized models or when you want a lightweight alternative.
  • TensorRT-LLM provides NVIDIA’s optimized inference engine with MPI-based multi-node orchestration.
  • Atlas is a pure-Rust inference server (see Atlas on GitHub) with native NCCL distribution. It composes tensor parallelism (tensor_parallel) and expert parallelism (ep_size) on either an overlapping mesh (tp == ep) or an orthogonal one (tp * ep), and derives the world size from the two.
  • Modular MAX is single-node only: tensor_parallel maps to local --devices rather than to host count, and the runtime refuses a multi-host placement rather than silently truncating it.

The Spark Arena team maintains several specialized container images for DGX Spark.

ImageDescriptionLink
ghcr.io/spark-arena/dgx-vllm-eugr-nightlyOptimized eugr vLLM for DGX SparkGitHub
ghcr.io/spark-arena/dgx-vllm-eugr-nightly-tf5Optimized eugr vLLM for DGX Spark with Transformers 5.xGitHub
dgx-spark-sglangOptimized SGLang for DGX SparkLink
ghcr.io/spark-arena/dgx-llama-cppOptimized llama.cpp for DGX Spark
Mirror of scitrera/dgx-spark-llama-cpp
Docker
GitHub

Default run completes after declared readiness and required hooks; timing and inference probes depend on the runtime/executor’s capabilities. Unsupported observers do not report fabricated startup metrics. Explicit recipe memory settings override serving defaults and remain distinct from scheduling reservations. Model manifests and target runtime compilation caches have separate validation and cleanup policies. See run, Schedulers, and Model Cache Management. Example image tags and recipe values on this page are illustrative, not installation-wide defaults.