GGUF Recipes

GGUF recipes use colon syntax in the model field to select a quantization variant. The primary runtime for GGUF is llama-cpp, but SGLang also has experimental GGUF support — see SGLang GGUF for details.

Example

model: Qwen/Qwen3-1.7B-GGUF:Q8_0
runtime: llama-cpp
max_nodes: 1
container: scitrera/dgx-spark-llama-cpp:b8076-cu131

metadata:
  description: Qwen3 1.7B (Q8_0 GGUF) -- small test model via llama.cpp
  maintainer: scitrera.ai <open-source-team@scitrera.com>
  model_params: 1.7B
  model_dtype: q8_0

defaults:
  port: 8000
  host: 0.0.0.0
  n_gpu_layers: 99
  ctx_size: 8192

command: |
  llama-server \
      -hf {model} \
      --host {host} \
      --port {port} \
      --n-gpu-layers {n_gpu_layers} \
      --ctx-size {ctx_size} \
      --flash-attn on \
      --jinja \
      --no-webui

Key differences from vLLM/SGLang recipes

• Model field uses repo:quant syntax (e.g. Qwen/Qwen3-1.7B-GGUF:Q8_0). sparkrun parses this to download only the matching quantization files.
• max_nodes: 1 — llama-cpp solo mode does not support multi-node distribution.
• ctx_size replaces max_model_len — the context window size in tokens.
• n_gpu_layers controls how many layers are offloaded to GPU. Set to 99 to offload all layers.

Model pre-sync

When model pre-sync is enabled (the default), sparkrun:

1. Downloads only the matching GGUF quant files locally
2. Distributes them to target hosts via rsync
3. Rewrites -hf to -m with the resolved container cache path

The container serves from the local copy without re-downloading.

Running

sparkrun run qwen3-1.7b-llama-cpp
sparkrun run qwen3-1.7b-llama-cpp --solo

Experimental: Multi-node via RPC

llama.cpp has an experimental RPC backend for multi-node inference. Worker nodes run rpc-server and the head node connects via --rpc.

Caution

This is still evolving both upstream and in sparkrun. The fastest DGX Spark interconnect communication uses NCCL and RoCE — llama.cpp’s RPC mechanism involves more overhead.