Atlas
Atlas is a pure-Rust LLM inference server with a native NCCL-based distribution layer. sparkrun’s atlas runtime launches spark serve and orchestrates the container lifecycle, NCCL bootstrap, and OpenAI-compatible HTTP endpoint.
Features
Section titled “Features”- Pure-Rust inference server with low overhead
- Native NCCL bootstrap via
--rank,--world-size,--master-addr, and--master-port - Tensor parallelism (
--tp-size) and expert parallelism (--ep-size) — both can share the same physical ranks (overlapping groups) or form an orthogonal mesh - Optional speculative decoding (
--speculative,--ngram-speculative,--self-speculative), DFlash attention, and prefix caching --high-speed-swappath that usesio_uringfor fast KV/cache spill, supported by sparkrun’s shared Docker seccomp profile
Atlas derives its world size from the two parallelism dimensions, so you set
tensor_parallel and ep_size and sparkrun maps the resulting ranks onto
physical hosts:
world_size == tensor_parallel × ep_size— orthogonal meshworld_size == tensor_parallel == ep_size— overlapping groups
Atlas requires a GB10 host: the runtime declares a gb10 capability
requirement, and a launch placed on a host without it fails the compatibility
check before any side effects.
Official recipe registry
Section titled “Official recipe registry”The Atlas maintainers publish an official sparkrun recipe registry at Atlas-Inf/sparkrun-recipes. Treat this as the authoritative source for official Atlas recipes — recipe formats, supported flags, and container tags there are kept in sync with upstream Atlas releases.
The @atlas registry is bundled as a default registry in sparkrun v0.2.31 and later, so it’s discoverable out of the box without manual sparkrun registry add:
# List recipes from the official Atlas registrysparkrun recipe list --registry atlas
# Run a recipe from the @atlas registrysparkrun run @atlas/<recipe-name>If you’re on an older sparkrun release, you’ll need to update to use the atlas runtime and registry.
Running
Section titled “Running”sparkrun run @atlas/qwen3.5-35b-a3b-nvfp4Container images
Section titled “Container images”Default image prefix: azeezish/atlas-gb10
container: azeezish/atlas-gb10:latestSee the Atlas GitHub repository for image tags and release notes.
Example recipe
Section titled “Example recipe”This single-node example illustrates Atlas settings. Inspect the current @atlas recipe before choosing an image tag or model configuration:
recipe_version: "2"model: Sehyo/Qwen3.5-35B-A3B-NVFP4runtime: atlascontainer: azeezish/atlas-gb10:latestmax_nodes: 1
metadata: description: | Example Qwen3.5-35B (A3B MoE) NVFP4 + MTP configuration on one GB10. category: agent model_params: 35B model_dtype: nvfp4 quantization: nvfp4 kv_dtype: nvfp4
defaults: port: 8888 host: 0.0.0.0 max_model_len: 8192 kv_cache_dtype: nvfp4 gpu_memory_utilization: 0.88 scheduling_policy: slai speculative: true mtp_quantization: nvfp4 enable_prefix_caching: trueWhen no command template is provided, sparkrun generates spark serve {model} with flags derived from defaults via the Atlas flag map.
Configuration options
Section titled “Configuration options”Use runtime and defaults and omit command for the supported settings
below. Add a command only when you need an unusual engine flag that sparkrun
does not map. These are the flags generated by sparkrun; the selected engine
image must also support them.
Set values under defaults, for example port: 8000, or override them with
-o port=9000. Unset values are omitted unless a runtime/platform default
supplies them. Boolean toggles emit a bare flag when true and omit it when false,
except where the table lists an explicit negative or valued form. Mapping/list
values are rendered as one JSON argument for flags that accept JSON. Unknown
keys do not automatically become engine flags.
The generated command is spark serve MODEL; model is a top-level recipe
field. Value-taking boolean options pass a lowercase true or false, while
boolean toggles only emit a bare flag when enabled.
Mappings: sparkrun 0.4.0 source.
| Engine flag | Key under defaults | Value form |
|---|---|---|
--port | port | Value |
--host | host | Value |
--tp-size | tensor_parallel | Value |
--gpu-memory-utilization | gpu_memory_utilization | Value |
--max-seq-len | max_model_len | Value |
--max-num-seqs | max_num_seqs | Value |
--max-prefill-tokens | max_num_batched_tokens | Value |
--model-name | served_model_name | Value |
--kv-cache-dtype | kv_cache_dtype | Value |
--ep-size | ep_size | Value |
--max-batch-size | max_batch_size | Value |
--block-size | block_size | Value |
--kv-high-precision-layers | kv_high_precision_layers | Value |
--tool-call-parser | tool_call_parser | Value |
--tool-max-tokens | tool_max_tokens | Value |
--max-inter-tool-prose | max_inter_tool_prose | Value |
--fp8-kv-calibration-tokens | fp8_kv_calibration_tokens | Value |
--fp8-kv-headroom | fp8_kv_headroom | Value |
--scheduling-policy | scheduling_policy | Value |
--tbt-deadline-ms | tbt_deadline_ms | Value |
--max-prefill-tokens | max_prefill_tokens | Value |
--prefill-varlen-batch | prefill_varlen_batch | Explicit true / false value |
--oom-guard-mb | oom_guard_mb | Value |
--request-timeout | request_timeout | Value |
--swap-space-gb | swap_space_gb | Value |
--warmup-prompt | warmup_prompt | Value |
--auto-compact | auto_compact | Value |
--kernel-target | kernel_target | Value |
--lm-head-dtype | lm_head_dtype | Value |
--ssm-cache-slots | ssm_cache_slots | Value |
--ssm-checkpoint-interval | ssm_checkpoint_interval | Value |
--ssm-h-dtype | ssm_h_dtype | Value |
--ssm-rollback-mode | ssm_rollback_mode | Value |
--ssm-batched-recurrent | ssm_batched_recurrent | Explicit true / false value |
--ssm-tail-midchunk | ssm_tail_midchunk | Explicit true / false value |
--gdn-fused-norm | gdn_fused_norm | Explicit true / false value |
--mtp-quantization | mtp_quantization | Value |
--mtp-vocab | mtp_vocab | Value |
--mtp-gate | mtp_gate | Value |
--num-drafts | num_drafts | Value |
--draft-model | draft_model | Value |
--exact-verify | exact_verify | Explicit true / false value |
--dflash-gamma | dflash_gamma | Value |
--dflash-window-size | dflash_window_size | Value |
--max-thinking-budget | max_thinking_budget | Value |
--default-top-n-sigma | default_top_n_sigma | Value |
--default-min-p | default_min_p | Value |
--content-loop-watchdog | content_loop_watchdog | Explicit true / false value |
--content-loop-min-repeats | content_loop_min_repeats | Value |
--default-chat-template-kwargs | default_chat_template_kwargs | Value |
--model-from-path | model_from_path | Value |
--cache-dir | cache_dir | Value |
--gpu-ordinal | gpu_ordinal | Value |
--dump | dump | Value |
--vision-max-pixels | vision_max_pixels | Value |
--vision-remote-image-max-mb | vision_remote_image_max_mb | Value |
--vision-remote-image-timeout-s | vision_remote_image_timeout_s | Value |
--video-ffmpeg-path | video_ffmpeg_path | Value |
--video-fps | video_fps | Value |
--video-max-frames | video_max_frames | Value |
--video-decode-timeout-s | video_decode_timeout_s | Value |
--lora-adapter | lora_adapter | Value |
--max-lora-rank | max_lora_rank | Value |
--max-loras | max_loras | Value |
--lora-stageable | lora_stageable | Value |
--lora-stageable-disk | lora_stageable_disk | Value |
--src-lang | src_lang | Value |
--tgt-lang | tgt_lang | Value |
--high-speed-swap-dir | high_speed_swap_dir | Value |
--high-speed-swap-gb | high_speed_swap_gb | Value |
--high-speed-swap-resident-blocks | high_speed_swap_resident_blocks | Value |
--high-speed-swap-rank | high_speed_swap_rank | Value |
--high-speed-swap-qd | high_speed_swap_qd | Value |
--high-speed-swap-cache-blocks-per-seq | high_speed_swap_cache_blocks_per_seq | Value |
--high-speed-swap-graph | high_speed_swap_graph | Explicit true / false value |
--disable-tool-grammar | disable_tool_grammar | Explicit true / false value |
--enable-prefix-caching | enable_prefix_caching | Boolean toggle; true emits the flag |
--speculative | speculative | Boolean toggle; true emits the flag |
--self-speculative | self_speculative | Boolean toggle; true emits the flag |
--ngram-speculative | ngram_speculative | Boolean toggle; true emits the flag |
--dflash | dflash | Boolean toggle; true emits the flag |
--disable-thinking | disable_thinking | Boolean toggle; true emits the flag |
--high-speed-swap | high_speed_swap | Boolean toggle; true emits the flag |
--require-auth | require_auth | Boolean toggle; true emits the flag |
--adaptive-sampling | adaptive_sampling | Boolean toggle; true emits the flag |
--auto-swap | auto_swap | Boolean toggle; true emits the flag |
--no-auto-swap | no_auto_swap | Boolean toggle; true emits the flag |
--auth-tokens-file | auth_tokens_file | Value |
--disable-template-overrides | disable_template_overrides | Boolean toggle; true emits the flag |
--vision-allow-remote-images | vision_allow_remote_images | Boolean toggle; true emits the flag |
--vision-remote-image-allow-private | vision_remote_image_allow_private | Boolean toggle; true emits the flag |
--video-allow-ffmpeg | video_allow_ffmpeg | Boolean toggle; true emits the flag |
--no-fast-load | no_fast_load | Boolean toggle; true emits the flag |
--fast-load-prefetch-shards | fast_load_prefetch_shards | Boolean toggle; true emits the flag |
--no-tui | no_tui | Boolean toggle; true emits the flag |
--profile | profile | Boolean toggle; true emits the flag |
--check-kernels | check_kernels | Boolean toggle; true emits the flag |
--dangerously-allow-unresolved-kernel-lookups | dangerously_allow_unresolved_kernel_lookups | Boolean toggle; true emits the flag |
--auth-token | api_key | Value |
auth_token is an alias for api_key. For the prefill budget, choose either
max_num_batched_tokens or max_prefill_tokens: both map to the same engine
flag. Expert parallelism uses ep_size for this runtime. draft_model_revision
pins draft-model acquisition and has no serve flag.
Sparkrun generates --rank, --world-size, --master-addr, and --master-port
from placement; worker ranks receive --port 0. The parallelism section below
explains overlapping and orthogonal TP/EP layouts.
Speculative decoding
Section titled “Speculative decoding”When a recipe sets draft_model in defaults, sparkrun automatically pre-syncs that model to all target hosts alongside the primary model — the same way it pre-syncs draft models for vLLM and SGLang speculative recipes. This happens during the distribution phase so the container never has to fetch the draft from HuggingFace at launch time (which would fail under sparkrun’s offline-hub default).
defaults: speculative: true draft_model: avarok/qwen3.5-35b-a3b-nvfp4-draft num_drafts: 2This works with any of the speculative modes (speculative, self_speculative, ngram_speculative, dflash); the only requirement is that the draft model name is set in defaults.draft_model.
Parallelism model
Section titled “Parallelism model”Atlas derives world_size from tensor_parallel and ep_size:
world_size == tp_size * ep_size— orthogonal mesh (TP and EP on independent ranks)world_size == tp_size == ep_size— overlapping groups (both groups share the same physical ranks)
sparkrun maps the resulting world size to physical hosts automatically and launches the ranks through the native NCCL path. The earlier single-node restriction is removed in the pinned 0.4.0 source. For a single-node recipe, set tensor_parallel: 1 and omit ep_size (or set it to 1).
Docker options
Section titled “Docker options”Atlas uses the shared Docker seccomp profile,
which permits io_uring_setup, io_uring_enter, and io_uring_register while
retaining the rest of the bundled Moby policy. The Atlas runtime does not add
seccomp=unconfined; disabling seccomp is an explicit executor configuration choice.
The Atlas runtime adds these capabilities on every Docker launch, including when
high_speed_swap is disabled:
| Option | Capability |
|---|---|
--cap-add=IPC_LOCK | Permits locking memory, used by pinned-memory and RDMA paths |
--cap-add=SYS_NICE | Permits elevated scheduling and priority operations |
These are Atlas additions. The shared Docker settings supply unlimited locked
memory with --ulimit memlock=-1:-1 for normal non-privileged launches; that limit
is separate from the IPC_LOCK capability. See
Capabilities and locked memory.
SYS_NICE describes the current launch configuration, not a universal requirement
for high-speed swap: Linux 5.13 and later allow SQPOLL without special privileges.
See the Linux io_uring documentation.
References
Section titled “References”Startup and cache policy
Section titled “Startup and cache policy”Atlas does not declare streaming inference readiness observers. Default run
uses supported endpoint checks and waits for required hooks; it does not report
unavailable first-token timings. See run.
The example’s memory utilization is an explicit recipe setting, separate from scheduling reservations. Model files, draft models, and runtime caches have different validation and cleanup needs; see Model Cache Management.