Skip to content

llama.cpp

llama.cpp support in sparkrun provides a lightweight runtime for GGUF quantized models via llama-server.

  • Solo mode with GGUF quantized models
  • Loads models directly from HuggingFace (e.g. Qwen/Qwen3-1.7B-GGUF:Q8_0)
  • Automatic model pre-sync with selective quant file download
  • Lightweight alternative to vLLM/SGLang

sparkrun works with ready-built images including:

We recommend you use the Spark Arena image for sparkrun recipes.

GGUF models use colon syntax to select a quantization variant:

model: Qwen/Qwen3-1.7B-GGUF:Q8_0

sparkrun pre-downloads only the matching quant files and resolves the local cache path so the container doesn’t need to re-download at serve time.

model: Qwen/Qwen3-1.7B-GGUF:Q8_0
runtime: llama-cpp
max_nodes: 1
container: ghcr.io/spark-arena/dgx-llama-cpp:latest
metadata:
description: Qwen3 1.7B (Q8_0 GGUF)
model_params: 1.7B
model_dtype: q8_0
defaults:
port: 8000
host: 0.0.0.0
n_gpu_layers: 99
ctx_size: 8192
flash_attn: "on"
jinja: true
webui: false

Use runtime and defaults and omit command for the supported settings below. Add a command only when you need an unusual engine flag that sparkrun does not map. These are the flags generated by sparkrun; the selected engine image must also support them.

Set values under defaults, for example port: 8000, or override them with -o port=9000. Unset values are omitted unless a runtime/platform default supplies them. Boolean toggles emit a bare flag when true and omit it when false, except where the table lists an explicit negative or valued form. Mapping/list values are rendered as one JSON argument for flags that accept JSON. Unknown keys do not automatically become engine flags.

The generated command is llama-server. The top-level model selects its -m file or -hf repository argument; prepared GGUF files use -m. A resolved multimodal projector adds --mmproj automatically.

Mappings: sparkrun 0.4.0 source.

Engine flagKey under defaultsValue form
--portportValue
--hosthostValue
--ctx-sizectx_sizeValue
--n-gpu-layersn_gpu_layersValue
--parallelparallelValue
--threadsthreadsValue
--chat-templatechat_templateValue
--reasoning-formatreasoning_formatValue
--split-modesplit_modeValue
--aliasserved_model_nameValue
--api-keyapi_keyValue
--top-ptop_pValue
--top-ktop_kValue
--temptemperatureValue
--min-pmin_pValue
--presence-penaltypresence_penaltyValue
--flash-attnflash_attn"on", "off", "auto"; booleans become on/off
--cont-batchingcont_batchingBoolean toggle; true emits the flag
--no-webuino_webuiBoolean toggle; true emits the flag
--jinjajinjaBoolean toggle; true emits the flag

Additional settings with special handling:

Engine flagKey under defaultsBehavior
--ctx-sizemax_model_lenPortable alias when ctx_size is unset; an explicit CLI context override takes precedence
--no-webuiwebuiEmitted when explicitly false
--no-mmapmmapEmitted when explicitly false; the DGX Spark platform defaults to false
--split-mode rowtensor_parallelTP above one selects row splitting
--split-mode layerpipeline_parallelPP above one selects layer splitting

TP and PP cannot both exceed one. An explicit split_mode is used when parallelism does not force a mode; otherwise the fallback is layer. The experimental multi-node path supplies --rpc endpoints automatically.

ParameterDescription
n_gpu_layersNumber of layers offloaded to GPU (99 = all)
ctx_sizeContext window size in tokens
Terminal window
sparkrun run qwen3-1.7b-llama-cpp

llama.cpp has an experimental RPC backend for multi-node inference. Worker nodes run rpc-server and the head node connects via --rpc.

llama.cpp does not declare the streaming readiness observers used by vLLM and SGLang. Startup uses the available endpoint checks without inventing inference or first-token timings. See run.

On DGX Spark, mmap defaults to off unless the recipe explicitly sets mmap: true. GGUF selections and multimodal projectors must be available at the resolved revision on every required target. See GGUF Recipes and Model Cache Management for inventory validation and repair.