sparkrun tune
sparkrun tune sglang <recipe> [options]sparkrun tune vllm <recipe> [options]Run kernel autotuning on a single host using a recipe’s container image. Both
SGLang and vLLM support fused MoE and dense FP8 kernels. Results are saved in
the tuning cache for automatic reuse by future sparkrun run invocations.
Tuning is particularly beneficial for Mixture-of-Experts (MoE) models and FP8 quantized models, where the fused-MoE / FP8 GEMM kernels are performance-critical paths. Default Triton configs are generic; tuning finds the best configuration for your specific hardware and model combination.
How it works
Section titled “How it works”sparkrun tune sglang defaults to MoE tuning: it launches a tuning container,
clones SGLang’s benchmark scripts, detects Triton, and tunes each requested TP
size. Dense FP8 mode uses explicit per-rank shapes and the kernels in the serving
image; it does not use the cloned MoE benchmark scripts.
sparkrun tune vllm shells out to the vllm-tune backing engine on the target host. Sparkrun ensures the pinned vllm-tune git ref is checked out under ~/.cache/sparkrun/vllm-tune/<ref>/, then invokes vllm-tune.sh --standalone --foreground --mode {moe|fp8|all} once per TP size. vllm-tune manages its own standalone tuning container per TP, covers both fused MoE and FP8 dense GEMM kernels, and writes results into sparkrun’s flat tuning cache via its --export-sparkrun integration point. Sparkrun then rsyncs that cache back to the control machine.
Subsequent runs mount MoE configs through SGLANG_MOE_CONFIG_DIR for SGLang and
use VLLM_TUNED_CONFIG_FOLDER for vLLM. Matching SGLang dense FP8 configs are
installed into the Docker serving image’s configuration directories before
serving, preserving bundled configurations for other shapes.
Prerequisites
Section titled “Prerequisites”sparkrun tune vllm requires jq, docker, and git on the remote host (the DGX Spark, not the control machine). The tuner runs a preflight check and bails with an actionable error if any are missing.
Options
Section titled “Options”| Option | Description | Applies to |
|---|---|---|
--hosts / -H | Comma-separated host list (only the first host is used) | both |
--hosts-file | File with hosts (one per line) | both |
--cluster | Use a saved cluster by name | both |
--tp | TP sizes (repeatable; normally 1,2,4,8). SGLang FP8 accepts one value, defaulting to recipe TP or 1. | both |
--image | Override container image | both |
--output-dir | Override tuning config output directory | both |
--parallel / -j | Run N tuning jobs concurrently (default: 1 = sequential) | both |
--dry-run / -n | Show what would be done without executing | both |
--skip-clone | Skip cloning benchmark scripts (if already in image) | sglang only |
--mode {moe,fp8,all} | Kernels to tune (default: all) | vllm only |
--mode {moe,fp8} | Kernels to tune (default: moe) | sglang only |
--shape N K | Per-rank FP8 weight shape; repeatable and required for FP8 | sglang FP8 |
--block-shape N K | Quantization block dimensions (default: 128 128); powers of two, at least 16 | sglang FP8 |
--batch-size M | Token count to tune; repeatable | sglang FP8 |
--timeout | Per-TP timeout in seconds; 0 means no timeout | both |
--vllm-tune-ref | Override the vllm-tune git ref pinned in config | vllm only |
SGLang tuning
Section titled “SGLang tuning”# Tune for all default TP sizes (1, 2, 4, 8)sparkrun tune sglang qwen3.5-35b-bf16-sglang --cluster mylab
# Tune for a specific TP sizesparkrun tune sglang qwen3.5-35b-bf16-sglang --cluster mylab --tp 2
# Tune for multiple specific TP sizessparkrun tune sglang qwen3.5-35b-bf16-sglang --cluster mylab --tp 1 --tp 2 --tp 4
# Tune with 4 parallel jobssparkrun tune sglang qwen3.5-35b-bf16-sglang --cluster mylab -j4
# Preview without runningsparkrun tune sglang qwen3.5-35b-bf16-sglang --cluster mylab --dry-runRequires an SGLang recipe (the recipe’s runtime must be sglang). MoE configs are saved to ~/.cache/sparkrun/tuning/sglang/ and auto-mounted via SGLANG_MOE_CONFIG_DIR.
Dense FP8 tuning
Section titled “Dense FP8 tuning”Use the per-rank weight shapes reported by the serving runtime. Choose the same image, tensor parallelism, and quantization block shape as the intended deployment. The dimensions below are illustrative; replace them with your model’s shapes:
sparkrun tune sglang my-fp8-recipe --cluster mylab --mode fp8 \ --tp 2 --shape 4096 4096 --shape 14336 4096 \ --block-shape 128 128 --batch-size 1 --batch-size 16FP8 mode accepts one TP value and requires --parallel 1 (the default), because
the shapes are already per rank. MoE mode does not accept shape/batch options.
Results live under ~/.cache/sparkrun/tuning/sglang/fp8/. Future Docker launches
install matching configurations before the server starts, including when the
server runs as a non-root user. Installation failure produces a warning; configs
already bundled in the image remain available. Keep the tuning and serving
images compatible.
vLLM tuning
Section titled “vLLM tuning”# Tune both MoE and FP8 kernels for all default TP sizes (1, 2, 4, 8)sparkrun tune vllm qwen3-moe-vllm --cluster mylab
# FP8 dense GEMM only — much faster (~25 min) for non-MoE FP8 modelssparkrun tune vllm qwen3-4b-fp8 --cluster mylab --mode fp8 --tp 1
# MoE onlysparkrun tune vllm qwen3-moe-vllm --cluster mylab --mode moe --tp 4
# Tune for multiple specific TP sizessparkrun tune vllm qwen3-moe-vllm --cluster mylab --tp 1 --tp 2 --tp 4
# Tune with 4 parallel jobssparkrun tune vllm qwen3-moe-vllm --cluster mylab -j4
# Pin an in-flight vllm-tune branch for testingsparkrun tune vllm qwen3-moe-vllm --cluster mylab --vllm-tune-ref my-pr-branch
# Preview without runningsparkrun tune vllm qwen3-moe-vllm --cluster mylab --dry-runUse runtime: vllm in recipes; tuning accepts both its distributed/mp and Ray
variants. Select the serving backend with defaults.distributed_executor_backend
(mp or ray), leaving it unset for distributed/mp. Configs are saved to
~/.cache/sparkrun/tuning/vllm/ and auto-mounted via VLLM_TUNED_CONFIG_FOLDER.
The vllm-tune git URL and ref can be pinned in ~/.config/sparkrun/config.yaml:
tuning: vllm_tune_repo: https://github.com/SeraphimSerapis/vllm-tune.git vllm_tune_ref: mainTuning output
Section titled “Tuning output”Configs are stored on the host at:
| Runtime | Host path | Container path | Env variable |
|---|---|---|---|
| SGLang | ~/.cache/sparkrun/tuning/sglang/ | /tuning/sglang/configs | SGLANG_MOE_CONFIG_DIR |
| vLLM | ~/.cache/sparkrun/tuning/vllm/ | /tuning/vllm | VLLM_TUNED_CONFIG_FOLDER |
vLLM tuning configs are stored flat (one JSON per kernel shape); MoE filenames start with E=..., FP8 dense GEMM filenames with N=....
Once tuning configs exist on a host, every sparkrun run for the matching runtime will mount them automatically. No recipe changes or extra flags are needed.
- Tuning runs on a single host even if you specify a cluster — only the first host is used.
- Each TP size produces its own config. Tune for the TP sizes you actually plan to use.
- Use
-j4for parallel MoE/vLLM tuning; SGLang FP8 requires--parallel 1. - Use
--dry-runto preview the Docker and tuning commands before committing to a long run. - Tuning can take hours per TP size depending on the model.