Skip to content

sparkrun tune

Terminal window
sparkrun tune sglang <recipe> [options]
sparkrun tune vllm <recipe> [options]

Run kernel autotuning on a single host using a recipe’s container image. Both SGLang and vLLM support fused MoE and dense FP8 kernels. Results are saved in the tuning cache for automatic reuse by future sparkrun run invocations.

Tuning is particularly beneficial for Mixture-of-Experts (MoE) models and FP8 quantized models, where the fused-MoE / FP8 GEMM kernels are performance-critical paths. Default Triton configs are generic; tuning finds the best configuration for your specific hardware and model combination.

sparkrun tune sglang defaults to MoE tuning: it launches a tuning container, clones SGLang’s benchmark scripts, detects Triton, and tunes each requested TP size. Dense FP8 mode uses explicit per-rank shapes and the kernels in the serving image; it does not use the cloned MoE benchmark scripts.

sparkrun tune vllm shells out to the vllm-tune backing engine on the target host. Sparkrun ensures the pinned vllm-tune git ref is checked out under ~/.cache/sparkrun/vllm-tune/<ref>/, then invokes vllm-tune.sh --standalone --foreground --mode {moe|fp8|all} once per TP size. vllm-tune manages its own standalone tuning container per TP, covers both fused MoE and FP8 dense GEMM kernels, and writes results into sparkrun’s flat tuning cache via its --export-sparkrun integration point. Sparkrun then rsyncs that cache back to the control machine.

Subsequent runs mount MoE configs through SGLANG_MOE_CONFIG_DIR for SGLang and use VLLM_TUNED_CONFIG_FOLDER for vLLM. Matching SGLang dense FP8 configs are installed into the Docker serving image’s configuration directories before serving, preserving bundled configurations for other shapes.

sparkrun tune vllm requires jq, docker, and git on the remote host (the DGX Spark, not the control machine). The tuner runs a preflight check and bails with an actionable error if any are missing.

OptionDescriptionApplies to
--hosts / -HComma-separated host list (only the first host is used)both
--hosts-fileFile with hosts (one per line)both
--clusterUse a saved cluster by nameboth
--tpTP sizes (repeatable; normally 1,2,4,8). SGLang FP8 accepts one value, defaulting to recipe TP or 1.both
--imageOverride container imageboth
--output-dirOverride tuning config output directoryboth
--parallel / -jRun N tuning jobs concurrently (default: 1 = sequential)both
--dry-run / -nShow what would be done without executingboth
--skip-cloneSkip cloning benchmark scripts (if already in image)sglang only
--mode {moe,fp8,all}Kernels to tune (default: all)vllm only
--mode {moe,fp8}Kernels to tune (default: moe)sglang only
--shape N KPer-rank FP8 weight shape; repeatable and required for FP8sglang FP8
--block-shape N KQuantization block dimensions (default: 128 128); powers of two, at least 16sglang FP8
--batch-size MToken count to tune; repeatablesglang FP8
--timeoutPer-TP timeout in seconds; 0 means no timeoutboth
--vllm-tune-refOverride the vllm-tune git ref pinned in configvllm only
Terminal window
# Tune for all default TP sizes (1, 2, 4, 8)
sparkrun tune sglang qwen3.5-35b-bf16-sglang --cluster mylab
# Tune for a specific TP size
sparkrun tune sglang qwen3.5-35b-bf16-sglang --cluster mylab --tp 2
# Tune for multiple specific TP sizes
sparkrun tune sglang qwen3.5-35b-bf16-sglang --cluster mylab --tp 1 --tp 2 --tp 4
# Tune with 4 parallel jobs
sparkrun tune sglang qwen3.5-35b-bf16-sglang --cluster mylab -j4
# Preview without running
sparkrun tune sglang qwen3.5-35b-bf16-sglang --cluster mylab --dry-run

Requires an SGLang recipe (the recipe’s runtime must be sglang). MoE configs are saved to ~/.cache/sparkrun/tuning/sglang/ and auto-mounted via SGLANG_MOE_CONFIG_DIR.

Use the per-rank weight shapes reported by the serving runtime. Choose the same image, tensor parallelism, and quantization block shape as the intended deployment. The dimensions below are illustrative; replace them with your model’s shapes:

Terminal window
sparkrun tune sglang my-fp8-recipe --cluster mylab --mode fp8 \
--tp 2 --shape 4096 4096 --shape 14336 4096 \
--block-shape 128 128 --batch-size 1 --batch-size 16

FP8 mode accepts one TP value and requires --parallel 1 (the default), because the shapes are already per rank. MoE mode does not accept shape/batch options.

Results live under ~/.cache/sparkrun/tuning/sglang/fp8/. Future Docker launches install matching configurations before the server starts, including when the server runs as a non-root user. Installation failure produces a warning; configs already bundled in the image remain available. Keep the tuning and serving images compatible.

Terminal window
# Tune both MoE and FP8 kernels for all default TP sizes (1, 2, 4, 8)
sparkrun tune vllm qwen3-moe-vllm --cluster mylab
# FP8 dense GEMM only — much faster (~25 min) for non-MoE FP8 models
sparkrun tune vllm qwen3-4b-fp8 --cluster mylab --mode fp8 --tp 1
# MoE only
sparkrun tune vllm qwen3-moe-vllm --cluster mylab --mode moe --tp 4
# Tune for multiple specific TP sizes
sparkrun tune vllm qwen3-moe-vllm --cluster mylab --tp 1 --tp 2 --tp 4
# Tune with 4 parallel jobs
sparkrun tune vllm qwen3-moe-vllm --cluster mylab -j4
# Pin an in-flight vllm-tune branch for testing
sparkrun tune vllm qwen3-moe-vllm --cluster mylab --vllm-tune-ref my-pr-branch
# Preview without running
sparkrun tune vllm qwen3-moe-vllm --cluster mylab --dry-run

Use runtime: vllm in recipes; tuning accepts both its distributed/mp and Ray variants. Select the serving backend with defaults.distributed_executor_backend (mp or ray), leaving it unset for distributed/mp. Configs are saved to ~/.cache/sparkrun/tuning/vllm/ and auto-mounted via VLLM_TUNED_CONFIG_FOLDER.

The vllm-tune git URL and ref can be pinned in ~/.config/sparkrun/config.yaml:

tuning:
vllm_tune_repo: https://github.com/SeraphimSerapis/vllm-tune.git
vllm_tune_ref: main

Configs are stored on the host at:

RuntimeHost pathContainer pathEnv variable
SGLang~/.cache/sparkrun/tuning/sglang//tuning/sglang/configsSGLANG_MOE_CONFIG_DIR
vLLM~/.cache/sparkrun/tuning/vllm//tuning/vllmVLLM_TUNED_CONFIG_FOLDER

vLLM tuning configs are stored flat (one JSON per kernel shape); MoE filenames start with E=..., FP8 dense GEMM filenames with N=....

Once tuning configs exist on a host, every sparkrun run for the matching runtime will mount them automatically. No recipe changes or extra flags are needed.

  • Tuning runs on a single host even if you specify a cluster — only the first host is used.
  • Each TP size produces its own config. Tune for the TP sizes you actually plan to use.
  • Use -j4 for parallel MoE/vLLM tuning; SGLang FP8 requires --parallel 1.
  • Use --dry-run to preview the Docker and tuning commands before committing to a long run.
  • Tuning can take hours per TP size depending on the model.