Skip to content

Benchmark on Spark Arena

Spark Arena is a community leaderboard for DGX Spark inference performance. sparkrun integrates directly with Spark Arena — authenticate, benchmark, and submit results all from the CLI.

Use sparkrun arena for login, logout, and status. After logging in, sparkrun benchmark <recipe> --arena runs the benchmark and submits its results:

  1. Launches the inference server
  2. Benchmarks using the standardized @official/spark-arena-v2 profile
  3. Collects results (CSV, metadata, effective recipe)
  4. Stops the inference server launched for the benchmark
  5. Uploads the submission to the Spark Arena leaderboard

Before submitting benchmarks, authenticate with your Spark Arena account:

Terminal window
sparkrun arena login

This opens your browser for OAuth authentication. Sign in with your preferred provider (GitHub, Google, etc.) and sparkrun stores a refresh token locally.

Verify your login status anytime:

Terminal window
sparkrun arena status

With authentication in place, run a benchmark and submit in one step:

Terminal window
sparkrun benchmark qwen3-1.7b-sglang --arena

This uses your default cluster. To target another saved group:

Terminal window
# Use a saved cluster
sparkrun benchmark qwen3-1.7b-sglang --arena --cluster mylab
# Or use a saved pair
sparkrun benchmark qwen3-1.7b-sglang --arena --cluster pair-a

For multi-node benchmarks, set tensor parallelism:

Terminal window
sparkrun benchmark qwen3-1.7b-sglang --arena --tp 2

--arena enables Spark Arena submission and defaults to the standardized @official/spark-arena-v2 profile. You can use the same option with the explicit benchmark perf subcommand. An explicit --profile takes precedence.

When the benchmark completes, sparkrun uploads three files to Spark Arena:

FileContents
recipe.yamlThe effective recipe (with all overrides applied)
benchmark.csvRaw benchmark results (prompt sizes, token counts, latencies, throughput)
metadata.jsonSystem metadata — sparkrun version, cluster info, runtime details, container image

All files are cached locally under ~/.cache/sparkrun/benchmarks/<submission-id>/ so you always have a copy.

You can apply the same recipe overrides as sparkrun run:

Terminal window
# Override GPU memory utilization
sparkrun benchmark qwen3-1.7b-sglang --arena -o gpu_memory_utilization=0.9
# Override max model length
sparkrun benchmark qwen3-1.7b-sglang --arena -o max_model_len=8192
# Override container image
sparkrun benchmark qwen3-1.7b-sglang --arena --image my-custom-image:latest

These overrides are captured in the submission metadata, so the leaderboard accurately reflects your exact configuration.

Long @official/spark-arena-v2 sweeps are checkpointed after every benchmark task. If a run is interrupted, use the same recipe and settings plus --resume to continue without prompting, skipping completed tasks.

Terminal window
# First attempt — interrupted partway through
sparkrun benchmark qwen3-1.7b-sglang --arena --cluster mylab --tp 2
# Resume without prompting, then submit when complete
sparkrun benchmark qwen3-1.7b-sglang --arena --cluster mylab --tp 2 --resume

sparkrun also pins the benchmarking framework version (e.g. the llama-benchy release) on the first task of a run and reuses that exact version for every subsequent task — even after a resume — so the numbers you submit are internally consistent. The pinned version is captured in the uploaded metadata.json.

State files live under ~/.cache/sparkrun/benchmarks/. See Resumable runs in the benchmarking tutorial for more detail.

Your credentials persist across sessions — you only need to log in once. If you want to remove stored credentials (e.g., switching accounts or on a shared machine):

Terminal window
sparkrun arena logout
  • Browse the Spark Arena leaderboard to see community results
  • Try different recipes and runtimes to find the best configuration for your workload
  • See Performance Benchmarking for more on sparkrun’s benchmark system (custom profiles, parameters, comparing configurations)

Measurement completion is independent of upload success. Keep the saved benchmark or submission reference after an interrupted upload; retry completed-state publication without launching inference or remeasuring. Reacquire credentials at execution time rather than embedding them in benchmark checkpoints. Resume the saved benchmark by ID:

Terminal window
sparkrun benchmark resume <benchmark-id>

The saved run retains its Arena integration and submission ID, so this retries publication without adding --arena again. See Benchmark API.