Skip to content

vLLM

vLLM is the default runtime in sparkrun. For recipes, use runtime: vllm and select the backend with defaults.distributed_executor_backend:

  • mp (default) — native distributed/multiprocessing, resolved as vllm-distributed.
  • ray — Ray head/worker orchestration, resolved as vllm-ray.

Omit the backend setting to use distributed/mp. Keep runtime: vllm when switching between backends; see Runtime alias resolution.

  • Solo and multi-node tensor parallelism across DGX Spark nodes
  • Broad model support (most HuggingFace models)
  • PagedAttention for efficient KV cache management
  • Tool calling support
  • Two multi-node strategies: vLLM native distributed or Ray clustering

Both vllm-distributed and vllm-ray use the same container images. sparkrun works with ready-built images including:

We recommend that you use the Spark Arena images for sparkrun recipes; when you use the official Spark Arena images, sparkrun will automatically manage using the latest eugr/spark-vllm-docker.

The Spark Arena nightly image names — ghcr.io/spark-arena/dgx-vllm-eugr-nightly:latest, its -tf5 variant, and the Docker Hub eugr/spark-vllm image — are sentinel images. When you reference one at :latest, sparkrun’s eugr builder resolves it to the authoritative published nightly and pulls it.

This is the default for all official v2 recipes: it gives you a stable, single name to reference while sparkrun handles versioning. The -tf5 and non-tf5 nightlies are identical now, so both resolve the same way.

To pin a specific image rather than tracking the nightly, use any non-:latest tag (e.g. ghcr.io/spark-arena/dgx-vllm-eugr-nightly:2026-03-01) — sparkrun uses that tag as-is. Tags can be moved upstream; use a digest when content identity must be immutable.

See Phase 2: Building image for the full builder lifecycle.

When defaults.speculative_config has a model field, sparkrun pre-syncs that draft model to all target hosts alongside the primary model. Both YAML mappings and JSON-object strings are supported, including mappings imported from the lil catalog. This happens during the distribution phase so the container can use the cached drafter under sparkrun’s offline-hub default.

defaults:
speculative_config:
method: dflash
model: z-lab/Qwen3.6-35B-A3B-DFlash
num_speculative_tokens: 5

Both vllm-distributed and vllm-ray honor this. Sparkrun passes the full mapping as one JSON argument to --speculative-config and uses model plus the optional revision for draft preparation. The draft does not inherit the primary model’s model_revision, since it belongs to a different repository.

Use -o speculative_config.model=org/other-drafter to change the model prepared for that launch, or -o speculative_config.num_speculative_tokens=3 to tune one field while preserving the rest. If the original config pins a draft revision, update that field too when changing repositories. See mapping overrides.

We recommend runtime: vllm for both variants. For recipes that let Sparkrun generate the command, select the backend in defaults:

defaults.distributed_executor_backendResolved runtimeOrchestration
Omittedvllm-distributedNative distributed/mp (default)
mpvllm-distributedNative distributed/mp, selected explicitly
rayvllm-rayRay head/worker cluster

To select native distributed/mp explicitly:

runtime: vllm
defaults:
distributed_executor_backend: mp

For Ray, change only the default:

runtime: vllm
defaults:
distributed_executor_backend: ray

For one launch, override that setting without editing the recipe:

Terminal window
sparkrun run my-vllm-recipe -o distributed_executor_backend=ray
sparkrun run my-vllm-recipe -o distributed_executor_backend=mp

The concrete names vllm-distributed and vllm-ray remain supported, but prefer the vllm alias so the backend setting controls selection. Older command templates containing --distributed-executor-backend ray can also select Ray; an explicit -o distributed_executor_backend=mp overrides that hint.

This recipe uses runtime: vllm, which resolves to vllm-distributed by default:

model: Qwen/Qwen3-1.7B
runtime: vllm
container: ghcr.io/spark-arena/dgx-vllm-eugr-nightly:latest
defaults:
port: 8000
host: 0.0.0.0
tensor_parallel: 1
gpu_memory_utilization: 0.8
max_model_len: 32768
enable_prefix_caching: true

Use runtime and defaults and omit command for the supported settings below. Add a command only when you need an unusual engine flag that sparkrun does not map. These are the flags generated by sparkrun; the selected engine image must also support them.

Set values under defaults, for example port: 8000, or override them with -o port=9000. Unset values are omitted unless a runtime/platform default supplies them. Boolean toggles emit a bare flag when true and omit it when false, except where the table lists an explicit negative or valued form. Mapping/list values are rendered as one JSON argument for flags that accept JSON. Unknown keys do not automatically become engine flags.

The generated command is vllm serve MODEL. Set model and model_revision at the recipe’s top level; model preparation and revision arguments are managed by sparkrun. Both vLLM variants share these mappings, with different parallelism constraints described below.

Mappings: sparkrun 0.4.0 source.

Engine flagKey under defaultsValue form
--portportValue
--hosthostValue
-tptensor_parallelValue
--gpu-memory-utilizationgpu_memory_utilizationValue
--max-model-lenmax_model_lenValue
--max-num-batched-tokensmax_num_batched_tokensValue
--max-num-seqsmax_num_seqsValue
--served-model-nameserved_model_nameValue
--dtypedtypeValue
--quantizationquantizationValue
--enforce-eagerenforce_eagerBoolean toggle; true emits the flag
--enable-prefix-cachingenable_prefix_cachingBoolean; false emits --no-enable-prefix-caching
--trust-remote-codetrust_remote_codeBoolean toggle; true emits the flag
--distributed-executor-backenddistributed_executor_backendValue
-pppipeline_parallelValue
--data-parallel-sizedata_parallelValue
--kv-cache-dtypekv_cache_dtypeValue
--otlp-traces-endpointotlp_traces_endpointValue
--api-keyapi_keyValue
--attention-backendattention_backendValue
--load-formatload_formatValue
--reasoning-parserreasoning_parserValue
--tool-call-parsertool_call_parserValue
--chat-templatechat_templateValue
--speculative-configspeculative_configValue
--tokenizer-modetokenizer_modeValue
--mm-encoder-tp-modemm_encoder_tp_modeValue
--block-sizeblock_sizeValue
--seedseedValue
--enable-auto-tool-choiceenable_auto_tool_choiceBoolean toggle; true emits the flag
--enable-chunked-prefillenable_chunked_prefillBoolean; false emits --no-enable-chunked-prefill
--async-schedulingasync_schedulingBoolean; false emits --no-async-scheduling
--generation-configgeneration_configValue
--hf-overrideshf_overridesValue
--compilation-configcompilation_configValue
--attention-configattention_configValue
--model-loader-extra-configmodel_loader_extra_configValue
--default-chat-template-kwargsdefault_chat_template_kwargsValue
--limit-mm-per-promptlimit_mm_per_promptValue
--mm-processor-cache-gbmm_processor_cache_gbValue
--kv-cache-memory-byteskv_cache_memory_bytesValue
--max-cudagraph-capture-sizemax_cudagraph_capture_sizeValue
--long-prefill-token-thresholdlong_prefill_token_thresholdValue
--mamba-cache-modemamba_cache_modeValue
--mamba-cache-dtypemamba_cache_dtypeValue
--mamba-ssm-cache-dtypemamba_ssm_cache_dtypeValue
--reasoning-parser-pluginreasoning_parser_pluginValue
--media-io-kwargsmedia_io_kwargsValue
--video-pruning-ratevideo_pruning_rateValue
--allowed-local-media-pathallowed_local_media_pathValue
--prefix-match-unitprefix_match_unitValue
--enable-mamba-shared-prefix-checkpointenable_mamba_shared_prefix_checkpointBoolean toggle; true emits the flag
--safetensors-load-strategysafetensors_load_strategyValue
--moe-backendmoe_backendValue
--enable-prompt-tokens-detailsenable_prompt_tokens_detailsBoolean toggle; true emits the flag
--enable-force-include-usageenable_force_include_usageBoolean toggle; true emits the flag
--enable-request-id-headersenable_request_id_headersBoolean toggle; true emits the flag
--enable-flashinfer-autotuneenable_flashinfer_autotuneBoolean; false emits --no-enable-flashinfer-autotune
--disable-custom-all-reducedisable_custom_all_reduceBoolean toggle; true emits the flag

These flags require a B12X-compatible image, such as the B12X nightly used by the lil adapter. Stock vLLM images can reject them.

Engine flagKey under defaultsValue form
--linear-backendlinear_backendValue
--gdn-prefill-backendgdn_prefill_backendValue
--gdn-decode-kernelgdn_decode_kernelValue
--kda-prefill-backendkda_prefill_backendValue
--mamba-backendmamba_backendValue
--recurrent-checkpoint-policyrecurrent_checkpoint_policyValue
--prefix-cache-retention-intervalprefix_cache_retention_intervalValue
--engram-configengram_configValue
--dcp-comm-backenddcp_comm_backendValue
--prefill-policyprefill_policyValue
--prefill-compute-shareprefill_compute_shareValue
--prefill-compute-half-lifeprefill_compute_half_lifeValue
--prefill-schedule-intervalprefill_schedule_intervalValue
--max-parallel-prefillsmax_parallel_prefillsValue
--decode-refill-targetdecode_refill_targetValue
--scheduler-reserve-full-islscheduler_reserve_full_islBoolean; false emits --no-scheduler-reserve-full-isl
--jit-monitor-modejit_monitor_modeValue

Parallelism and distributed_executor_backend are resolved by the selected vLLM variant. Sparkrun supplies rank, world-size, rendezvous, and headless-worker flags from placement. data_parallel_rpc_port configures the DP rendezvous port; it defaults to 13345. See the parallelism section below for topology limits.

Multi-node with vllm-distributed (default)

Section titled “Multi-node with vllm-distributed (default)”

With runtime: vllm and the backend omitted or set to mp, Sparkrun selects vllm-distributed. It uses vLLM’s built-in multi-node support, with Sparkrun handling container lifecycle, InfiniBand detection, and node coordination:

Terminal window
sparkrun run qwen3-1.7b-vllm --tp 2

This explicitly requests tensor parallelism across two DGX Sparks in the selected cluster.

No Ray cluster is involved — vLLM coordinates the nodes directly via its native distributed backend. See Execution Flow for the full multi-node orchestration details.

vllm-distributed supports three parallelism regimes on DGX Spark (one GPU per node, so each parallel dimension consumes physical hosts):

ModeDefaultsBehavior
Cross-node TP/PP, single replicatensor_parallel: N or pipeline_parallel: N (DP=1)sparkrun appends --nnodes, --node-rank, --master-addr, --master-port; worker ranks get --headless.
Pure data parallelismdata_parallel: N (TP=PP=1)Each node is its own replica; sparkrun appends --data-parallel-size, --data-parallel-rank, --data-parallel-address, --data-parallel-rpc-port.
Hybrid (TP/PP × DP)tensor_parallel: T, data_parallel: Dsparkrun maps replicas to host groups of size T·PP; --master-addr is the first host of each replica; both flag sets are emitted.

Required node count = tensor_parallel × pipeline_parallel × data_parallel. The data_parallel_rpc_port defaults to 13345 and can be overridden in recipe defaults.

vllm-ray forms a Ray cluster first, then runs the serve command on the head node. sparkrun handles Ray cluster setup, InfiniBand detection, and container lifecycle automatically.

Select Ray through defaults while keeping the recommended runtime name:

runtime: vllm
defaults:
distributed_executor_backend: ray

These options apply only to vllm-ray:

OptionDefaultDescription
--ray-port46379Ray GCS port
--dashboardoffEnable Ray dashboard
--dashboard-port8265Ray dashboard port

Both variants support the same models and vLLM flags. The difference is in multi-node orchestration:

  • Distributed/mp (default) uses vLLM’s built-in coordination without Ray. Use runtime: vllm and omit distributed_executor_backend, or set it to mp.
  • Ray provides Ray’s cluster management, dashboard, and monitoring. Keep runtime: vllm and set defaults.distributed_executor_backend: ray when you need those features.

For single-node (solo) workloads, both variants behave identically.

Default run follows startup logs until readiness and required hooks complete. vLLM declares streaming Chat Completions, Responses, and Messages probes plus a /health endpoint. Which observations run depends on executor support; missing observers do not produce inferred timing values. See run.

On DGX Spark, sparkrun supplies gpu_memory_utilization: 0.80 only when the recipe and overrides leave it unset. This serving setting is separate from the scheduler’s 0.90 cap. Runtime compilation caches are separate from model files; see Model Cache Management.