llama.cpp
llama.cpp support in sparkrun provides a lightweight runtime for GGUF quantized models via llama-server.
Features
Section titled “Features”- Solo mode with GGUF quantized models
- Loads models directly from HuggingFace (e.g.
Qwen/Qwen3-1.7B-GGUF:Q8_0) - Automatic model pre-sync with selective quant file download
- Lightweight alternative to vLLM/SGLang
Container images
Section titled “Container images”sparkrun works with ready-built images including:
- Spark Arena Official Image:
ghcr.io/spark-arena/dgx-llama-cpp:latest - scitrera/dgx-spark-llama-cpp
- Build your own
We recommend you use the Spark Arena image for sparkrun recipes.
GGUF model syntax
Section titled “GGUF model syntax”GGUF models use colon syntax to select a quantization variant:
model: Qwen/Qwen3-1.7B-GGUF:Q8_0sparkrun pre-downloads only the matching quant files and resolves the local cache path so the container doesn’t need to re-download at serve time.
Example recipe
Section titled “Example recipe”model: Qwen/Qwen3-1.7B-GGUF:Q8_0runtime: llama-cppmax_nodes: 1container: ghcr.io/spark-arena/dgx-llama-cpp:latest
metadata: description: Qwen3 1.7B (Q8_0 GGUF) model_params: 1.7B model_dtype: q8_0
defaults: port: 8000 host: 0.0.0.0 n_gpu_layers: 99 ctx_size: 8192 flash_attn: "on" jinja: true webui: falseConfiguration options
Section titled “Configuration options”Use runtime and defaults and omit command for the supported settings
below. Add a command only when you need an unusual engine flag that sparkrun
does not map. These are the flags generated by sparkrun; the selected engine
image must also support them.
Set values under defaults, for example port: 8000, or override them with
-o port=9000. Unset values are omitted unless a runtime/platform default
supplies them. Boolean toggles emit a bare flag when true and omit it when false,
except where the table lists an explicit negative or valued form. Mapping/list
values are rendered as one JSON argument for flags that accept JSON. Unknown
keys do not automatically become engine flags.
The generated command is llama-server. The top-level model selects its
-m file or -hf repository argument; prepared GGUF files use -m. A resolved
multimodal projector adds --mmproj automatically.
Mappings: sparkrun 0.4.0 source.
| Engine flag | Key under defaults | Value form |
|---|---|---|
--port | port | Value |
--host | host | Value |
--ctx-size | ctx_size | Value |
--n-gpu-layers | n_gpu_layers | Value |
--parallel | parallel | Value |
--threads | threads | Value |
--chat-template | chat_template | Value |
--reasoning-format | reasoning_format | Value |
--split-mode | split_mode | Value |
--alias | served_model_name | Value |
--api-key | api_key | Value |
--top-p | top_p | Value |
--top-k | top_k | Value |
--temp | temperature | Value |
--min-p | min_p | Value |
--presence-penalty | presence_penalty | Value |
--flash-attn | flash_attn | "on", "off", "auto"; booleans become on/off |
--cont-batching | cont_batching | Boolean toggle; true emits the flag |
--no-webui | no_webui | Boolean toggle; true emits the flag |
--jinja | jinja | Boolean toggle; true emits the flag |
Additional settings with special handling:
| Engine flag | Key under defaults | Behavior |
|---|---|---|
--ctx-size | max_model_len | Portable alias when ctx_size is unset; an explicit CLI context override takes precedence |
--no-webui | webui | Emitted when explicitly false |
--no-mmap | mmap | Emitted when explicitly false; the DGX Spark platform defaults to false |
--split-mode row | tensor_parallel | TP above one selects row splitting |
--split-mode layer | pipeline_parallel | PP above one selects layer splitting |
TP and PP cannot both exceed one. An explicit split_mode is used when
parallelism does not force a mode; otherwise the fallback is layer.
The experimental multi-node path supplies --rpc endpoints automatically.
Key parameters
Section titled “Key parameters”| Parameter | Description |
|---|---|
n_gpu_layers | Number of layers offloaded to GPU (99 = all) |
ctx_size | Context window size in tokens |
Running
Section titled “Running”sparkrun run qwen3-1.7b-llama-cppExperimental: Multi-node via RPC
Section titled “Experimental: Multi-node via RPC”llama.cpp has an experimental RPC backend for multi-node inference. Worker nodes run rpc-server and the head node connects via --rpc.
Startup and cache policy
Section titled “Startup and cache policy”llama.cpp does not declare the streaming readiness observers used by vLLM and SGLang. Startup uses the available endpoint checks without inventing inference or first-token timings. See run.
On DGX Spark, mmap defaults to off unless the recipe explicitly sets mmap: true.
GGUF selections and multimodal projectors must be available at the resolved
revision on every required target. See GGUF Recipes and
Model Cache Management for inventory validation and repair.