Skip to content

SGLang

SGLang is a first-class runtime in sparkrun with support for solo and multi-node inference using SGLang’s native distributed backend.

  • Solo and multi-node clustering via SGLang’s built-in distribution
  • Native distributed backend (--dist-init-addr, --nnodes, --node-rank)
  • High-throughput inference
  • Structured generation support
  • Experimental GGUF support (quantized models via SGLang)

sparkrun works with ready-built images including:

model: Qwen/Qwen3-1.7B
runtime: sglang
min_nodes: 1
container: scitrera/dgx-spark-sglang:0.5.8-t5
metadata:
description: Qwen3 1.7B via SGLang
model_params: 1.7B
model_dtype: bf16
defaults:
port: 8000
host: 0.0.0.0
tensor_parallel: 1

Use runtime and defaults and omit command for the supported settings below. Add a command only when you need an unusual engine flag that sparkrun does not map. These are the flags generated by sparkrun; the selected engine image must also support them.

Set values under defaults, for example port: 8000, or override them with -o port=9000. Unset values are omitted unless a runtime/platform default supplies them. Boolean toggles emit a bare flag when true and omit it when false, except where the table lists an explicit negative or valued form. Mapping/list values are rendered as one JSON argument for flags that accept JSON. Unknown keys do not automatically become engine flags.

The generated command is sglang serve --model-path MODEL. The top-level model supplies the model path; prepared snapshots and GGUF selections are resolved by sparkrun. Use top-level model_revision to pin acquisition. The revision default below is a serving-engine flag, not a substitute for that pin.

Mappings: sparkrun 0.4.0 source.

Engine flagKey under defaultsValue form
--portportValue
--hosthostValue
--tp-sizetensor_parallelValue
--pp-sizepipeline_parallelValue
--ep-sizeexpert_parallelValue
--dp-sizedata_parallelValue
--enable-dp-attentionenable_dp_attentionBoolean toggle; true emits the flag
--mem-fraction-staticgpu_memory_utilizationValue
--context-lengthmax_model_lenValue
--max-running-requestsmax_num_seqsValue
--served-model-nameserved_model_nameValue
--dtypedtypeValue
--quantizationquantizationValue
--trust-remote-codetrust_remote_codeBoolean toggle; true emits the flag
--chunked-prefill-sizechunked_prefillValue
--kv-cache-dtypekv_cache_dtypeValue
--tokenizer-pathtokenizer_pathValue
--api-keyapi_keyValue
--attention-backendattention_backendValue
--load-formatload_formatValue
--revisionrevisionValue
--model-typemodel_typeValue
--max-total-tokensmax_total_tokensValue
--random-seedrandom_seedValue
--reasoning-parserreasoning_parserValue
--tool-call-parsertool_call_parserValue
--mm-feature-transportmm_feature_transportValue
--speculative-algorithmspeculative_algorithmValue
--speculative-draft-model-pathspeculative_draft_model_pathValue
--speculative-draft-model-revisionspeculative_draft_model_revisionValue
--speculative-num-stepsspeculative_num_stepsValue
--speculative-eagle-topkspeculative_eagle_topkValue
--speculative-num-draft-tokensspeculative_num_draft_tokensValue
--speculative-dspark-block-sizespeculative_dspark_block_sizeValue
--cuda-graph-bscuda_graph_bsValue
--cuda-graph-max-bscuda_graph_max_bsValue
--cuda-graph-max-bs-decodecuda_graph_max_bs_decodeValue
--torch-compile-max-bstorch_compile_max_bsValue
--num-continuous-decode-stepsnum_continuous_decode_stepsValue
--fp8-gemm-backendfp8_gemm_backendValue
--fp4-gemm-backendfp4_gemm_backendValue
--moe-runner-backendmoe_runner_backendValue
--mamba-ssm-dtypemamba_ssm_dtypeValue
--mamba-full-memory-ratiomamba_full_memory_ratioValue
--enable-torch-compileenable_torch_compileBoolean toggle; true emits the flag
--disable-radix-cachedisable_radix_cacheBoolean toggle; true emits the flag
--disable-prefill-cuda-graphdisable_prefill_cuda_graphBoolean toggle; true emits the flag
--enable-decoder-swa-bounded-replayenable_decoder_swa_bounded_replayBoolean toggle; true emits the flag

speculative_draft_model is accepted as an alias for speculative_draft_model_path. Sparkrun derives per-node --nnodes, --node-rank, and --dist-init-addr values from placement. data_parallel and enable_dp_attention are validated against the launch topology before --dp-size is emitted. cuda_graph_bs is a space-separated argument string (for example "1 2 4 8"), not a YAML list of separate CLI arguments.

For multi-node, sparkrun uses SGLang’s native distribution:

Terminal window
sparkrun run my-sglang-recipe --tp 2
OptionDefaultDescription
--init-port25000SGLang distributed init port

When a recipe sets speculative_draft_model_path (or the alias speculative_draft_model) in defaults, sparkrun automatically pre-syncs that model to all target hosts alongside the primary model. This happens during the distribution phase so the container never has to fetch the draft from HuggingFace at launch time (which would fail under sparkrun’s offline-hub default).

defaults:
speculative_algorithm: EAGLE
speculative_draft_model_path: lmsys/sglang-EAGLE-llama2-chat-7B
speculative_num_steps: 3
speculative_eagle_topk: 4

The value is also rendered into the generated command as --speculative-draft-model-path <model>. The alias key speculative_draft_model is accepted for convenience and normalized to the canonical key at command-render time.

SGLang has experimental support for GGUF quantized models. This lets you run quantized models through SGLang’s high-performance serving stack instead of llama.cpp.

Example: Qwen3.5-397B MoE (Q6_K GGUF, 4-node)

Section titled “Example: Qwen3.5-397B MoE (Q6_K GGUF, 4-node)”
model: unsloth/Qwen3.5-397B-A17B-GGUF:Q6_K
runtime: sglang
min_nodes: 4
container: scitrera/dgx-spark-sglang:0.5.8-t5
defaults:
port: 8000
host: 0.0.0.0
tensor_parallel: 4
gpu_memory_utilization: 0.8
max_model_len: 200000
served_model_name: qwen3.5-397b
attention_backend: triton
tool_call_parser: qwen3_coder
tokenizer_path: Qwen/Qwen3.5-397B-A17B
  • tokenizer_path — GGUF files don’t bundle tokenizer configs, so you must point to the original / non-GGUF HuggingFace repo (e.g. Qwen/Qwen3.5-397B-A17B)
  • Model field uses the same repo:quant syntax as llama-cpp GGUF recipes (e.g. unsloth/Qwen3.5-397B-A17B-GGUF:Q6_K)

sparkrun tune sglang supports fused MoE tuning and dense block-FP8 tuning. FP8 mode takes explicit per-rank shapes and installs compatible saved configs into future Docker serving containers. See SGLang FP8 tuning for options and an example.

SGLang declares a streaming Chat Completions readiness probe and /health. Default run returns after supported readiness checks and required hooks; use --follow to continue observing logs. See run.

On DGX Spark, sparkrun supplies gpu_memory_utilization: 0.80 only when the recipe and overrides leave it unset. That does not change the separate scheduler’s 0.90 cap. Model inventory and runtime compilation caches have separate validation and cleanup.