Benchmark on Spark Arena
Spark Arena is a community leaderboard for DGX Spark inference performance. sparkrun integrates directly with Spark Arena — authenticate, benchmark, and submit results all from the CLI.
Use sparkrun arena for login, logout, and status. After logging in,
sparkrun benchmark <recipe> --arena runs the benchmark and submits its results:
- Launches the inference server
- Benchmarks using the standardized
@official/spark-arena-v2profile - Collects results (CSV, metadata, effective recipe)
- Stops the inference server launched for the benchmark
- Uploads the submission to the Spark Arena leaderboard
1. Log in to Spark Arena
Section titled “1. Log in to Spark Arena”Before submitting benchmarks, authenticate with your Spark Arena account:
sparkrun arena loginThis opens your browser for OAuth authentication. Sign in with your preferred provider (GitHub, Google, etc.) and sparkrun stores a refresh token locally.
Verify your login status anytime:
sparkrun arena status2. Run a Spark Arena benchmark
Section titled “2. Run a Spark Arena benchmark”With authentication in place, run a benchmark and submit in one step:
sparkrun benchmark qwen3-1.7b-sglang --arenaThis uses your default cluster. To target another saved group:
# Use a saved clustersparkrun benchmark qwen3-1.7b-sglang --arena --cluster mylab
# Or use a saved pairsparkrun benchmark qwen3-1.7b-sglang --arena --cluster pair-aFor multi-node benchmarks, set tensor parallelism:
sparkrun benchmark qwen3-1.7b-sglang --arena --tp 2--arena enables Spark Arena submission and defaults to the standardized
@official/spark-arena-v2 profile. You can use the same option with the explicit
benchmark perf subcommand. An explicit --profile takes precedence.
3. What gets submitted
Section titled “3. What gets submitted”When the benchmark completes, sparkrun uploads three files to Spark Arena:
| File | Contents |
|---|---|
recipe.yaml | The effective recipe (with all overrides applied) |
benchmark.csv | Raw benchmark results (prompt sizes, token counts, latencies, throughput) |
metadata.json | System metadata — sparkrun version, cluster info, runtime details, container image |
All files are cached locally under ~/.cache/sparkrun/benchmarks/<submission-id>/ so you always have a copy.
4. Recipe overrides
Section titled “4. Recipe overrides”You can apply the same recipe overrides as sparkrun run:
# Override GPU memory utilizationsparkrun benchmark qwen3-1.7b-sglang --arena -o gpu_memory_utilization=0.9
# Override max model lengthsparkrun benchmark qwen3-1.7b-sglang --arena -o max_model_len=8192
# Override container imagesparkrun benchmark qwen3-1.7b-sglang --arena --image my-custom-image:latestThese overrides are captured in the submission metadata, so the leaderboard accurately reflects your exact configuration.
5. Resumable submissions
Section titled “5. Resumable submissions”Long @official/spark-arena-v2 sweeps are checkpointed after every benchmark task.
If a run is interrupted, use the same recipe and settings plus --resume to
continue without prompting, skipping completed tasks.
# First attempt — interrupted partway throughsparkrun benchmark qwen3-1.7b-sglang --arena --cluster mylab --tp 2
# Resume without prompting, then submit when completesparkrun benchmark qwen3-1.7b-sglang --arena --cluster mylab --tp 2 --resumesparkrun also pins the benchmarking framework version (e.g. the llama-benchy release) on the first task of a run and reuses that exact version for every subsequent task — even after a resume — so the numbers you submit are internally consistent. The pinned version is captured in the uploaded metadata.json.
State files live under ~/.cache/sparkrun/benchmarks/. See Resumable runs in the benchmarking tutorial for more detail.
6. Log out (optional)
Section titled “6. Log out (optional)”Your credentials persist across sessions — you only need to log in once. If you want to remove stored credentials (e.g., switching accounts or on a shared machine):
sparkrun arena logoutNext steps
Section titled “Next steps”- Browse the Spark Arena leaderboard to see community results
- Try different recipes and runtimes to find the best configuration for your workload
- See Performance Benchmarking for more on sparkrun’s benchmark system (custom profiles, parameters, comparing configurations)
Publication recovery
Section titled “Publication recovery”Measurement completion is independent of upload success. Keep the saved benchmark or submission reference after an interrupted upload; retry completed-state publication without launching inference or remeasuring. Reacquire credentials at execution time rather than embedding them in benchmark checkpoints. Resume the saved benchmark by ID:
sparkrun benchmark resume <benchmark-id>The saved run retains its Arena integration and submission ID, so this retries
publication without adding --arena again. See
Benchmark API.