vLLM
vLLM is the default runtime in sparkrun.
For recipes, use runtime: vllm and select the backend with
defaults.distributed_executor_backend:
mp(default) — native distributed/multiprocessing, resolved asvllm-distributed.ray— Ray head/worker orchestration, resolved asvllm-ray.
Omit the backend setting to use distributed/mp. Keep runtime: vllm when
switching between backends; see Runtime alias resolution.
Features
Section titled “Features”- Solo and multi-node tensor parallelism across DGX Spark nodes
- Broad model support (most HuggingFace models)
- PagedAttention for efficient KV cache management
- Tool calling support
- Two multi-node strategies: vLLM native distributed or Ray clustering
Container images
Section titled “Container images”Both vllm-distributed and vllm-ray use the same container images. sparkrun works with ready-built images including:
- Spark Arena Official Images (built from @eugr’s repo):
ghcr.io/spark-arena/dgx-vllm-eugr-nightly:latestandghcr.io/spark-arena/dgx-vllm-eugr-nightly-tf5:latest - Self-built community images from eugr/spark-vllm-docker (often the first to have the latest fixes)
- Official NVIDIA Images
We recommend that you use the Spark Arena images for sparkrun recipes; when you use the official Spark Arena images, sparkrun will automatically manage using the latest eugr/spark-vllm-docker.
Sentinel images
Section titled “Sentinel images”The Spark Arena nightly image names — ghcr.io/spark-arena/dgx-vllm-eugr-nightly:latest, its -tf5 variant, and the Docker Hub eugr/spark-vllm image — are sentinel images. When you reference one at :latest, sparkrun’s eugr builder resolves it to the authoritative published nightly and pulls it.
This is the default for all official v2 recipes: it gives you a stable, single name to reference while sparkrun handles versioning. The -tf5 and non-tf5 nightlies are identical now, so both resolve the same way.
To pin a specific image rather than tracking the nightly, use any non-:latest tag (e.g. ghcr.io/spark-arena/dgx-vllm-eugr-nightly:2026-03-01) — sparkrun uses that tag as-is. Tags can be moved upstream; use a digest when content identity must be immutable.
See Phase 2: Building image for the full builder lifecycle.
Speculative decoding
Section titled “Speculative decoding”When defaults.speculative_config has a model field, sparkrun pre-syncs that
draft model to all target hosts alongside the primary model. Both YAML mappings
and JSON-object strings are supported, including mappings imported from the lil
catalog. This happens during the
distribution phase
so the container can use the cached drafter under sparkrun’s offline-hub default.
defaults: speculative_config: method: dflash model: z-lab/Qwen3.6-35B-A3B-DFlash num_speculative_tokens: 5Both vllm-distributed and vllm-ray honor this. Sparkrun passes the full mapping
as one JSON argument to --speculative-config and uses model plus the optional
revision for draft preparation. The draft does not inherit the primary model’s
model_revision, since it belongs to a different repository.
Use -o speculative_config.model=org/other-drafter to change the model prepared
for that launch, or -o speculative_config.num_speculative_tokens=3 to tune one
field while preserving the rest. If the original config pins a draft revision,
update that field too when changing repositories. See
mapping overrides.
Runtime alias resolution
Section titled “Runtime alias resolution”We recommend runtime: vllm for both variants. For recipes that let Sparkrun
generate the command, select the backend in defaults:
defaults.distributed_executor_backend | Resolved runtime | Orchestration |
|---|---|---|
| Omitted | vllm-distributed | Native distributed/mp (default) |
mp | vllm-distributed | Native distributed/mp, selected explicitly |
ray | vllm-ray | Ray head/worker cluster |
To select native distributed/mp explicitly:
runtime: vllmdefaults: distributed_executor_backend: mpFor Ray, change only the default:
runtime: vllmdefaults: distributed_executor_backend: rayFor one launch, override that setting without editing the recipe:
sparkrun run my-vllm-recipe -o distributed_executor_backend=raysparkrun run my-vllm-recipe -o distributed_executor_backend=mpThe concrete names vllm-distributed and vllm-ray remain supported, but prefer
the vllm alias so the backend setting controls selection. Older command
templates containing --distributed-executor-backend ray can also select Ray;
an explicit -o distributed_executor_backend=mp overrides that hint.
Example recipe
Section titled “Example recipe”This recipe uses runtime: vllm, which resolves to vllm-distributed by default:
model: Qwen/Qwen3-1.7Bruntime: vllmcontainer: ghcr.io/spark-arena/dgx-vllm-eugr-nightly:latest
defaults: port: 8000 host: 0.0.0.0 tensor_parallel: 1 gpu_memory_utilization: 0.8 max_model_len: 32768 enable_prefix_caching: trueConfiguration options
Section titled “Configuration options”Use runtime and defaults and omit command for the supported settings
below. Add a command only when you need an unusual engine flag that sparkrun
does not map. These are the flags generated by sparkrun; the selected engine
image must also support them.
Set values under defaults, for example port: 8000, or override them with
-o port=9000. Unset values are omitted unless a runtime/platform default
supplies them. Boolean toggles emit a bare flag when true and omit it when false,
except where the table lists an explicit negative or valued form. Mapping/list
values are rendered as one JSON argument for flags that accept JSON. Unknown
keys do not automatically become engine flags.
The generated command is vllm serve MODEL. Set model and model_revision
at the recipe’s top level; model preparation and revision arguments are managed
by sparkrun. Both vLLM variants share these mappings, with different parallelism
constraints described below.
Mappings: sparkrun 0.4.0 source.
Standard vLLM flags
Section titled “Standard vLLM flags”| Engine flag | Key under defaults | Value form |
|---|---|---|
--port | port | Value |
--host | host | Value |
-tp | tensor_parallel | Value |
--gpu-memory-utilization | gpu_memory_utilization | Value |
--max-model-len | max_model_len | Value |
--max-num-batched-tokens | max_num_batched_tokens | Value |
--max-num-seqs | max_num_seqs | Value |
--served-model-name | served_model_name | Value |
--dtype | dtype | Value |
--quantization | quantization | Value |
--enforce-eager | enforce_eager | Boolean toggle; true emits the flag |
--enable-prefix-caching | enable_prefix_caching | Boolean; false emits --no-enable-prefix-caching |
--trust-remote-code | trust_remote_code | Boolean toggle; true emits the flag |
--distributed-executor-backend | distributed_executor_backend | Value |
-pp | pipeline_parallel | Value |
--data-parallel-size | data_parallel | Value |
--kv-cache-dtype | kv_cache_dtype | Value |
--otlp-traces-endpoint | otlp_traces_endpoint | Value |
--api-key | api_key | Value |
--attention-backend | attention_backend | Value |
--load-format | load_format | Value |
--reasoning-parser | reasoning_parser | Value |
--tool-call-parser | tool_call_parser | Value |
--chat-template | chat_template | Value |
--speculative-config | speculative_config | Value |
--tokenizer-mode | tokenizer_mode | Value |
--mm-encoder-tp-mode | mm_encoder_tp_mode | Value |
--block-size | block_size | Value |
--seed | seed | Value |
--enable-auto-tool-choice | enable_auto_tool_choice | Boolean toggle; true emits the flag |
--enable-chunked-prefill | enable_chunked_prefill | Boolean; false emits --no-enable-chunked-prefill |
--async-scheduling | async_scheduling | Boolean; false emits --no-async-scheduling |
--generation-config | generation_config | Value |
--hf-overrides | hf_overrides | Value |
--compilation-config | compilation_config | Value |
--attention-config | attention_config | Value |
--model-loader-extra-config | model_loader_extra_config | Value |
--default-chat-template-kwargs | default_chat_template_kwargs | Value |
--limit-mm-per-prompt | limit_mm_per_prompt | Value |
--mm-processor-cache-gb | mm_processor_cache_gb | Value |
--kv-cache-memory-bytes | kv_cache_memory_bytes | Value |
--max-cudagraph-capture-size | max_cudagraph_capture_size | Value |
--long-prefill-token-threshold | long_prefill_token_threshold | Value |
--mamba-cache-mode | mamba_cache_mode | Value |
--mamba-cache-dtype | mamba_cache_dtype | Value |
--mamba-ssm-cache-dtype | mamba_ssm_cache_dtype | Value |
--reasoning-parser-plugin | reasoning_parser_plugin | Value |
--media-io-kwargs | media_io_kwargs | Value |
--video-pruning-rate | video_pruning_rate | Value |
--allowed-local-media-path | allowed_local_media_path | Value |
--prefix-match-unit | prefix_match_unit | Value |
--enable-mamba-shared-prefix-checkpoint | enable_mamba_shared_prefix_checkpoint | Boolean toggle; true emits the flag |
--safetensors-load-strategy | safetensors_load_strategy | Value |
--moe-backend | moe_backend | Value |
--enable-prompt-tokens-details | enable_prompt_tokens_details | Boolean toggle; true emits the flag |
--enable-force-include-usage | enable_force_include_usage | Boolean toggle; true emits the flag |
--enable-request-id-headers | enable_request_id_headers | Boolean toggle; true emits the flag |
--enable-flashinfer-autotune | enable_flashinfer_autotune | Boolean; false emits --no-enable-flashinfer-autotune |
--disable-custom-all-reduce | disable_custom_all_reduce | Boolean toggle; true emits the flag |
B12X-specific flags
Section titled “B12X-specific flags”These flags require a B12X-compatible image, such as the B12X nightly used by the lil adapter. Stock vLLM images can reject them.
| Engine flag | Key under defaults | Value form |
|---|---|---|
--linear-backend | linear_backend | Value |
--gdn-prefill-backend | gdn_prefill_backend | Value |
--gdn-decode-kernel | gdn_decode_kernel | Value |
--kda-prefill-backend | kda_prefill_backend | Value |
--mamba-backend | mamba_backend | Value |
--recurrent-checkpoint-policy | recurrent_checkpoint_policy | Value |
--prefix-cache-retention-interval | prefix_cache_retention_interval | Value |
--engram-config | engram_config | Value |
--dcp-comm-backend | dcp_comm_backend | Value |
--prefill-policy | prefill_policy | Value |
--prefill-compute-share | prefill_compute_share | Value |
--prefill-compute-half-life | prefill_compute_half_life | Value |
--prefill-schedule-interval | prefill_schedule_interval | Value |
--max-parallel-prefills | max_parallel_prefills | Value |
--decode-refill-target | decode_refill_target | Value |
--scheduler-reserve-full-isl | scheduler_reserve_full_isl | Boolean; false emits --no-scheduler-reserve-full-isl |
--jit-monitor-mode | jit_monitor_mode | Value |
Parallelism and distributed_executor_backend are resolved by the selected
vLLM variant. Sparkrun supplies rank, world-size, rendezvous, and headless-worker
flags from placement. data_parallel_rpc_port configures the DP rendezvous port;
it defaults to 13345. See the parallelism section below for topology limits.
Multi-node with vllm-distributed (default)
Section titled “Multi-node with vllm-distributed (default)”With runtime: vllm and the backend omitted or set to mp, Sparkrun selects
vllm-distributed. It uses vLLM’s built-in multi-node support, with Sparkrun
handling container lifecycle, InfiniBand detection, and node coordination:
sparkrun run qwen3-1.7b-vllm --tp 2This explicitly requests tensor parallelism across two DGX Sparks in the selected cluster.
No Ray cluster is involved — vLLM coordinates the nodes directly via its native distributed backend. See Execution Flow for the full multi-node orchestration details.
Tensor, pipeline, and data parallelism
Section titled “Tensor, pipeline, and data parallelism”vllm-distributed supports three parallelism regimes on DGX Spark (one GPU per node, so each parallel dimension consumes physical hosts):
| Mode | Defaults | Behavior |
|---|---|---|
| Cross-node TP/PP, single replica | tensor_parallel: N or pipeline_parallel: N (DP=1) | sparkrun appends --nnodes, --node-rank, --master-addr, --master-port; worker ranks get --headless. |
| Pure data parallelism | data_parallel: N (TP=PP=1) | Each node is its own replica; sparkrun appends --data-parallel-size, --data-parallel-rank, --data-parallel-address, --data-parallel-rpc-port. |
| Hybrid (TP/PP × DP) | tensor_parallel: T, data_parallel: D | sparkrun maps replicas to host groups of size T·PP; --master-addr is the first host of each replica; both flag sets are emitted. |
Required node count = tensor_parallel × pipeline_parallel × data_parallel. The data_parallel_rpc_port defaults to 13345 and can be overridden in recipe defaults.
Multi-node with vllm-ray
Section titled “Multi-node with vllm-ray”vllm-ray forms a Ray cluster first, then runs the serve command on the head node. sparkrun handles Ray cluster setup, InfiniBand detection, and container lifecycle automatically.
Select Ray through defaults while keeping the recommended runtime name:
runtime: vllmdefaults: distributed_executor_backend: rayRay-specific options
Section titled “Ray-specific options”These options apply only to vllm-ray:
| Option | Default | Description |
|---|---|---|
--ray-port | 46379 | Ray GCS port |
--dashboard | off | Enable Ray dashboard |
--dashboard-port | 8265 | Ray dashboard port |
Choosing between variants
Section titled “Choosing between variants”Both variants support the same models and vLLM flags. The difference is in multi-node orchestration:
- Distributed/mp (default) uses vLLM’s built-in coordination without Ray.
Use
runtime: vllmand omitdistributed_executor_backend, or set it tomp. - Ray provides Ray’s cluster management, dashboard, and monitoring. Keep
runtime: vllmand setdefaults.distributed_executor_backend: raywhen you need those features.
For single-node (solo) workloads, both variants behave identically.
Startup and cache policy
Section titled “Startup and cache policy”Default run follows startup logs until readiness and required hooks complete.
vLLM declares streaming Chat Completions, Responses, and Messages probes plus a
/health endpoint. Which observations run depends on executor support; missing
observers do not produce inferred timing values. See run.
On DGX Spark, sparkrun supplies gpu_memory_utilization: 0.80 only when the
recipe and overrides leave it unset. This serving setting is separate from the
scheduler’s 0.90 cap. Runtime compilation caches are
separate from model files; see Model Cache Management.