Performance Benchmarking
Use sparkrun benchmark perf to measure inference throughput and latency with
llama-benchy. Sparkrun launches the recipe, waits for readiness, runs the tests,
saves results, and stops the workload it launched.
Start with a recipe that works on your configured hosts. The commands below use
normal placement on your default cluster; add --cluster mylab to select another
cluster. For tool-calling accuracy, follow Tool Call Benchmarking.
For one-node and larger setups, save your hosts as a cluster first; see Recommended Cluster Setups.
1. Run a first benchmark
Section titled “1. Run a first benchmark”Run a small test with a 2,048-token prompt, 128 generated tokens, and one request at a time:
sparkrun benchmark perf qwen3-1.7b-sglang \ -b pp=2048 -b tg=128 -b depth=0 -b concurrency=1 -b runs=3Replace the recipe name with your chosen recipe. Watch the launch and readiness output, then the benchmark task results. The command finishes by saving results and cleaning up the inference workload it started. Keep the printed benchmark ID and result paths.
perf is an alias for performance. sparkrun benchmark <recipe> also selects
performance benchmarking. Add --dry-run to preview a command before executing it.
2. Read the measurements
Section titled “2. Read the measurements”Compare rows with the same prompt size, generated tokens, context depth, and concurrency:
| Measurement | What it tells you |
|---|---|
| Time to first token (TTFT) | How long the client waits before generation starts; lower is better |
| Generation rate | How quickly a request receives output tokens; higher is better |
| Total throughput | How much work the deployment completes across concurrent requests |
| Latency | How long requests take under the chosen test load |
Higher total throughput can come with slower individual requests. Choose the tradeoff that fits your application, and compare repeated measurements with the same model, runtime settings, and test inputs. A server that fails readiness or a benchmark task is not a completed measurement; inspect its diagnostics before retrying.
3. Test concurrency and longer contexts
Section titled “3. Test concurrency and longer contexts”Expand the test into a matrix when the first benchmark completes:
sparkrun benchmark perf qwen3-1.7b-sglang \ -b pp=2048 -b tg=128 -b depth=0,4096 -b concurrency=1,2,5 -b runs=3Comma-separated values create multiple test points. This example measures three concurrency levels at two context depths. Make sure the recipe’s context limit can hold the requested context, prompt, and generated output.
Use -b for benchmark inputs and -o for the model’s serving configuration:
sparkrun benchmark perf qwen3-1.7b-sglang \ -o max_model_len=32768 \ -b pp=2048 -b tg=128 -b depth=0,4096 -b concurrency=1,2,54. Compare configurations with one profile
Section titled “4. Compare configurations with one profile”A profile keeps benchmark inputs consistent between runs. Browse the profiles available from your registries and inspect one before using it:
sparkrun registry list-benchmark-profilessparkrun registry show-benchmark-profile spark-arena-v1Run the profile with your recipe’s normal placement:
sparkrun benchmark perf qwen3-1.7b-sglang --profile spark-arena-v1If you have enough hosts and the runtime supports it, compare a two-node configuration using the same profile:
sparkrun benchmark perf qwen3-1.7b-sglang --tp 2 --profile spark-arena-v1Check the recorded hosts and parallelism for each run. Normal placement follows the recipe and available capacity, so it does not necessarily mean one node. Change one setting at a time and compare equivalent test rows.
5. Inspect and keep the results
Section titled “5. Inspect and keep the results”The printed output paths identify the YAML metadata, JSON results, and CSV measurements. YAML/JSON preserve recipe and host details; CSV is convenient for comparing test points in a spreadsheet. Default filenames include the recipe, profile, and parallelism. To choose an output path:
sparkrun benchmark perf qwen3-1.7b-sglang --output my-performance.yamlKeep the metadata alongside the measurements. Sparkrun pins the benchmark framework version and resolved container image for a resumable run so a later resume uses the same versions.
Benchmark an existing workload
Section titled “Benchmark an existing workload”To leave a newly benchmarked workload running, add --no-stop:
sparkrun benchmark perf qwen3-1.7b-sglang --no-stopFor a workload already started by Sparkrun with the same recipe and settings,
use --skip-run:
sparkrun benchmark perf qwen3-1.7b-sglang --skip-runThis verifies the recorded deployment and leaves it running after measurement. A reachable endpoint alone is insufficient: the saved workload must match the selected recipe and execution provenance.
Resume an interrupted run
Section titled “Resume an interrupted run”Rerun the same recipe and benchmark settings with --resume:
sparkrun benchmark perf qwen3-1.7b-sglang --profile spark-arena-v1 --resumeCompleted tasks with usable results are reused; failed tasks are retried.
State is saved under ~/.cache/sparkrun/benchmarks/<benchmark_id>/. Without an
explicit choice, interactive runs prompt and non-interactive runs default to
resume. Use --fresh when you intend to discard saved progress and remeasure;
it cannot be combined with --resume.
Measurement completion and export/publication are separate. If finalization fails, retain the benchmark ID and completed results so finalization can be retried without repeating measurements. See benchmark recovery.
Share results or automate the workflow
Section titled “Share results or automate the workflow”To publish performance results, log in and enable the Spark Arena integration:
sparkrun arena loginsparkrun benchmark qwen3-1.7b-sglang --arenaReal submissions default to @official/spark-arena-v2; an explicit --profile
takes precedence. Follow Benchmark on Spark Arena for
submission and upload recovery.
For all CLI options, see benchmark. For typed results, callbacks, and automation examples, see Benchmark API.