SGLang
SGLang is a first-class runtime in sparkrun with support for solo and multi-node inference using SGLang’s native distributed backend.
Features
Section titled “Features”- Solo and multi-node clustering via SGLang’s built-in distribution
- Native distributed backend (
--dist-init-addr,--nnodes,--node-rank) - High-throughput inference
- Structured generation support
- Experimental GGUF support (quantized models via SGLang)
Container images
Section titled “Container images”sparkrun works with ready-built images including:
Example recipe
Section titled “Example recipe”model: Qwen/Qwen3-1.7Bruntime: sglangmin_nodes: 1container: scitrera/dgx-spark-sglang:0.5.8-t5
metadata: description: Qwen3 1.7B via SGLang model_params: 1.7B model_dtype: bf16
defaults: port: 8000 host: 0.0.0.0 tensor_parallel: 1Configuration options
Section titled “Configuration options”Use runtime and defaults and omit command for the supported settings
below. Add a command only when you need an unusual engine flag that sparkrun
does not map. These are the flags generated by sparkrun; the selected engine
image must also support them.
Set values under defaults, for example port: 8000, or override them with
-o port=9000. Unset values are omitted unless a runtime/platform default
supplies them. Boolean toggles emit a bare flag when true and omit it when false,
except where the table lists an explicit negative or valued form. Mapping/list
values are rendered as one JSON argument for flags that accept JSON. Unknown
keys do not automatically become engine flags.
The generated command is sglang serve --model-path MODEL. The top-level
model supplies the model path; prepared snapshots and GGUF selections are
resolved by sparkrun. Use top-level model_revision to pin acquisition. The
revision default below is a serving-engine flag, not a substitute for that pin.
Mappings: sparkrun 0.4.0 source.
| Engine flag | Key under defaults | Value form |
|---|---|---|
--port | port | Value |
--host | host | Value |
--tp-size | tensor_parallel | Value |
--pp-size | pipeline_parallel | Value |
--ep-size | expert_parallel | Value |
--dp-size | data_parallel | Value |
--enable-dp-attention | enable_dp_attention | Boolean toggle; true emits the flag |
--mem-fraction-static | gpu_memory_utilization | Value |
--context-length | max_model_len | Value |
--max-running-requests | max_num_seqs | Value |
--served-model-name | served_model_name | Value |
--dtype | dtype | Value |
--quantization | quantization | Value |
--trust-remote-code | trust_remote_code | Boolean toggle; true emits the flag |
--chunked-prefill-size | chunked_prefill | Value |
--kv-cache-dtype | kv_cache_dtype | Value |
--tokenizer-path | tokenizer_path | Value |
--api-key | api_key | Value |
--attention-backend | attention_backend | Value |
--load-format | load_format | Value |
--revision | revision | Value |
--model-type | model_type | Value |
--max-total-tokens | max_total_tokens | Value |
--random-seed | random_seed | Value |
--reasoning-parser | reasoning_parser | Value |
--tool-call-parser | tool_call_parser | Value |
--mm-feature-transport | mm_feature_transport | Value |
--speculative-algorithm | speculative_algorithm | Value |
--speculative-draft-model-path | speculative_draft_model_path | Value |
--speculative-draft-model-revision | speculative_draft_model_revision | Value |
--speculative-num-steps | speculative_num_steps | Value |
--speculative-eagle-topk | speculative_eagle_topk | Value |
--speculative-num-draft-tokens | speculative_num_draft_tokens | Value |
--speculative-dspark-block-size | speculative_dspark_block_size | Value |
--cuda-graph-bs | cuda_graph_bs | Value |
--cuda-graph-max-bs | cuda_graph_max_bs | Value |
--cuda-graph-max-bs-decode | cuda_graph_max_bs_decode | Value |
--torch-compile-max-bs | torch_compile_max_bs | Value |
--num-continuous-decode-steps | num_continuous_decode_steps | Value |
--fp8-gemm-backend | fp8_gemm_backend | Value |
--fp4-gemm-backend | fp4_gemm_backend | Value |
--moe-runner-backend | moe_runner_backend | Value |
--mamba-ssm-dtype | mamba_ssm_dtype | Value |
--mamba-full-memory-ratio | mamba_full_memory_ratio | Value |
--enable-torch-compile | enable_torch_compile | Boolean toggle; true emits the flag |
--disable-radix-cache | disable_radix_cache | Boolean toggle; true emits the flag |
--disable-prefill-cuda-graph | disable_prefill_cuda_graph | Boolean toggle; true emits the flag |
--enable-decoder-swa-bounded-replay | enable_decoder_swa_bounded_replay | Boolean toggle; true emits the flag |
speculative_draft_model is accepted as an alias for
speculative_draft_model_path. Sparkrun derives per-node --nnodes,
--node-rank, and --dist-init-addr values from placement. data_parallel
and enable_dp_attention are validated against the launch topology before
--dp-size is emitted. cuda_graph_bs is a space-separated argument string
(for example "1 2 4 8"), not a YAML list of separate CLI arguments.
Multi-node inference
Section titled “Multi-node inference”For multi-node, sparkrun uses SGLang’s native distribution:
sparkrun run my-sglang-recipe --tp 2SGLang-specific options
Section titled “SGLang-specific options”| Option | Default | Description |
|---|---|---|
--init-port | 25000 | SGLang distributed init port |
Speculative decoding
Section titled “Speculative decoding”When a recipe sets speculative_draft_model_path (or the alias speculative_draft_model) in defaults, sparkrun automatically pre-syncs that model to all target hosts alongside the primary model. This happens during the distribution phase so the container never has to fetch the draft from HuggingFace at launch time (which would fail under sparkrun’s offline-hub default).
defaults: speculative_algorithm: EAGLE speculative_draft_model_path: lmsys/sglang-EAGLE-llama2-chat-7B speculative_num_steps: 3 speculative_eagle_topk: 4The value is also rendered into the generated command as --speculative-draft-model-path <model>. The alias key speculative_draft_model is accepted for convenience and normalized to the canonical key at command-render time.
Experimental: GGUF support
Section titled “Experimental: GGUF support”SGLang has experimental support for GGUF quantized models. This lets you run quantized models through SGLang’s high-performance serving stack instead of llama.cpp.
Example: Qwen3.5-397B MoE (Q6_K GGUF, 4-node)
Section titled “Example: Qwen3.5-397B MoE (Q6_K GGUF, 4-node)”model: unsloth/Qwen3.5-397B-A17B-GGUF:Q6_Kruntime: sglangmin_nodes: 4container: scitrera/dgx-spark-sglang:0.5.8-t5
defaults: port: 8000 host: 0.0.0.0 tensor_parallel: 4 gpu_memory_utilization: 0.8 max_model_len: 200000 served_model_name: qwen3.5-397b attention_backend: triton tool_call_parser: qwen3_coder tokenizer_path: Qwen/Qwen3.5-397B-A17BKey notes for SGLang GGUF
Section titled “Key notes for SGLang GGUF”tokenizer_path— GGUF files don’t bundle tokenizer configs, so you must point to the original / non-GGUF HuggingFace repo (e.g.Qwen/Qwen3.5-397B-A17B)- Model field uses the same
repo:quantsyntax as llama-cpp GGUF recipes (e.g.unsloth/Qwen3.5-397B-A17B-GGUF:Q6_K)
Kernel tuning
Section titled “Kernel tuning”sparkrun tune sglang supports fused MoE tuning and dense block-FP8 tuning.
FP8 mode takes explicit per-rank shapes and installs compatible saved configs
into future Docker serving containers. See SGLang FP8 tuning
for options and an example.
Startup and cache policy
Section titled “Startup and cache policy”SGLang declares a streaming Chat Completions readiness probe and /health.
Default run returns after supported readiness checks and required hooks; use
--follow to continue observing logs. See run.
On DGX Spark, sparkrun supplies gpu_memory_utilization: 0.80 only when the
recipe and overrides leave it unset. That does not change the separate
scheduler’s 0.90 cap. Model inventory and runtime
compilation caches have separate validation and cleanup.