Skip to content

Performance Benchmarking

Use sparkrun benchmark perf to measure inference throughput and latency with llama-benchy. Sparkrun launches the recipe, waits for readiness, runs the tests, saves results, and stops the workload it launched.

Start with a recipe that works on your configured hosts. The commands below use normal placement on your default cluster; add --cluster mylab to select another cluster. For tool-calling accuracy, follow Tool Call Benchmarking.

For one-node and larger setups, save your hosts as a cluster first; see Recommended Cluster Setups.

Run a small test with a 2,048-token prompt, 128 generated tokens, and one request at a time:

Terminal window
sparkrun benchmark perf qwen3-1.7b-sglang \
-b pp=2048 -b tg=128 -b depth=0 -b concurrency=1 -b runs=3

Replace the recipe name with your chosen recipe. Watch the launch and readiness output, then the benchmark task results. The command finishes by saving results and cleaning up the inference workload it started. Keep the printed benchmark ID and result paths.

perf is an alias for performance. sparkrun benchmark <recipe> also selects performance benchmarking. Add --dry-run to preview a command before executing it.

Compare rows with the same prompt size, generated tokens, context depth, and concurrency:

MeasurementWhat it tells you
Time to first token (TTFT)How long the client waits before generation starts; lower is better
Generation rateHow quickly a request receives output tokens; higher is better
Total throughputHow much work the deployment completes across concurrent requests
LatencyHow long requests take under the chosen test load

Higher total throughput can come with slower individual requests. Choose the tradeoff that fits your application, and compare repeated measurements with the same model, runtime settings, and test inputs. A server that fails readiness or a benchmark task is not a completed measurement; inspect its diagnostics before retrying.

Expand the test into a matrix when the first benchmark completes:

Terminal window
sparkrun benchmark perf qwen3-1.7b-sglang \
-b pp=2048 -b tg=128 -b depth=0,4096 -b concurrency=1,2,5 -b runs=3

Comma-separated values create multiple test points. This example measures three concurrency levels at two context depths. Make sure the recipe’s context limit can hold the requested context, prompt, and generated output.

Use -b for benchmark inputs and -o for the model’s serving configuration:

Terminal window
sparkrun benchmark perf qwen3-1.7b-sglang \
-o max_model_len=32768 \
-b pp=2048 -b tg=128 -b depth=0,4096 -b concurrency=1,2,5

4. Compare configurations with one profile

Section titled “4. Compare configurations with one profile”

A profile keeps benchmark inputs consistent between runs. Browse the profiles available from your registries and inspect one before using it:

Terminal window
sparkrun registry list-benchmark-profiles
sparkrun registry show-benchmark-profile spark-arena-v1

Run the profile with your recipe’s normal placement:

Terminal window
sparkrun benchmark perf qwen3-1.7b-sglang --profile spark-arena-v1

If you have enough hosts and the runtime supports it, compare a two-node configuration using the same profile:

Terminal window
sparkrun benchmark perf qwen3-1.7b-sglang --tp 2 --profile spark-arena-v1

Check the recorded hosts and parallelism for each run. Normal placement follows the recipe and available capacity, so it does not necessarily mean one node. Change one setting at a time and compare equivalent test rows.

The printed output paths identify the YAML metadata, JSON results, and CSV measurements. YAML/JSON preserve recipe and host details; CSV is convenient for comparing test points in a spreadsheet. Default filenames include the recipe, profile, and parallelism. To choose an output path:

Terminal window
sparkrun benchmark perf qwen3-1.7b-sglang --output my-performance.yaml

Keep the metadata alongside the measurements. Sparkrun pins the benchmark framework version and resolved container image for a resumable run so a later resume uses the same versions.

To leave a newly benchmarked workload running, add --no-stop:

Terminal window
sparkrun benchmark perf qwen3-1.7b-sglang --no-stop

For a workload already started by Sparkrun with the same recipe and settings, use --skip-run:

Terminal window
sparkrun benchmark perf qwen3-1.7b-sglang --skip-run

This verifies the recorded deployment and leaves it running after measurement. A reachable endpoint alone is insufficient: the saved workload must match the selected recipe and execution provenance.

Rerun the same recipe and benchmark settings with --resume:

Terminal window
sparkrun benchmark perf qwen3-1.7b-sglang --profile spark-arena-v1 --resume

Completed tasks with usable results are reused; failed tasks are retried. State is saved under ~/.cache/sparkrun/benchmarks/<benchmark_id>/. Without an explicit choice, interactive runs prompt and non-interactive runs default to resume. Use --fresh when you intend to discard saved progress and remeasure; it cannot be combined with --resume.

Measurement completion and export/publication are separate. If finalization fails, retain the benchmark ID and completed results so finalization can be retried without repeating measurements. See benchmark recovery.

To publish performance results, log in and enable the Spark Arena integration:

Terminal window
sparkrun arena login
sparkrun benchmark qwen3-1.7b-sglang --arena

Real submissions default to @official/spark-arena-v2; an explicit --profile takes precedence. Follow Benchmark on Spark Arena for submission and upload recovery.

For all CLI options, see benchmark. For typed results, callbacks, and automation examples, see Benchmark API.