Skip to content

GGUF Recipes

GGUF recipes use colon syntax in the model field to select a quantization variant. The primary runtime for GGUF is llama-cpp, but SGLang also has experimental GGUF support — see SGLang GGUF for details.

model: Qwen/Qwen3-1.7B-GGUF:Q8_0
runtime: llama-cpp
max_nodes: 1
container: scitrera/dgx-spark-llama-cpp:b8076-cu131
metadata:
description: Qwen3 1.7B (Q8_0 GGUF) -- small test model via llama.cpp
maintainer: user@example.com
model_params: 1.7B
model_dtype: q8_0
defaults:
port: 8000
host: 0.0.0.0
n_gpu_layers: 99
ctx_size: 8192
flash_attn: "on"
jinja: true
webui: false
  • Model field uses repo:quant syntax (e.g. Qwen/Qwen3-1.7B-GGUF:Q8_0). sparkrun parses this to download only the matching quantization files.
  • max_nodes: 1 — llama-cpp solo mode does not support multi-node distribution.
  • ctx_size replaces max_model_len — the context window size in tokens.
  • n_gpu_layers controls how many layers are offloaded to GPU. Set to 99 to offload all layers.

When model pre-sync is enabled (the default), sparkrun:

  1. Selects the matching GGUF quantization, multipart files, and optional projector from one immutable revision
  2. Acquires/copies them under the effective transfer policy and validates every required target
  3. Rewrites -hf to -m with the validated workload cache path

The container serves from the local copy without re-downloading.

Terminal window
sparkrun run qwen3-1.7b-llama-cpp

llama.cpp has an experimental RPC backend for multi-node inference. Worker nodes run rpc-server and the head node connects via --rpc.

Use matching projector/file-selection settings when inspecting or importing inventory. See Model Cache Management for offline reuse and targeted repair.