Skip to content

Atlas

Atlas is a pure-Rust LLM inference server with a native NCCL-based distribution layer. sparkrun’s atlas runtime launches spark serve and orchestrates the container lifecycle, NCCL bootstrap, and OpenAI-compatible HTTP endpoint.

  • Pure-Rust inference server with low overhead
  • Native NCCL bootstrap via --rank, --world-size, --master-addr, and --master-port
  • Tensor parallelism (--tp-size) and expert parallelism (--ep-size) — both can share the same physical ranks (overlapping groups) or form an orthogonal mesh
  • Optional speculative decoding (--speculative, --ngram-speculative, --self-speculative), DFlash attention, and prefix caching
  • --high-speed-swap path that uses io_uring for fast KV/cache spill, supported by sparkrun’s shared Docker seccomp profile

Atlas derives its world size from the two parallelism dimensions, so you set tensor_parallel and ep_size and sparkrun maps the resulting ranks onto physical hosts:

  • world_size == tensor_parallel × ep_size — orthogonal mesh
  • world_size == tensor_parallel == ep_size — overlapping groups

Atlas requires a GB10 host: the runtime declares a gb10 capability requirement, and a launch placed on a host without it fails the compatibility check before any side effects.

The Atlas maintainers publish an official sparkrun recipe registry at Atlas-Inf/sparkrun-recipes. Treat this as the authoritative source for official Atlas recipes — recipe formats, supported flags, and container tags there are kept in sync with upstream Atlas releases.

The @atlas registry is bundled as a default registry in sparkrun v0.2.31 and later, so it’s discoverable out of the box without manual sparkrun registry add:

Terminal window
# List recipes from the official Atlas registry
sparkrun recipe list --registry atlas
# Run a recipe from the @atlas registry
sparkrun run @atlas/<recipe-name>

If you’re on an older sparkrun release, you’ll need to update to use the atlas runtime and registry.

Terminal window
sparkrun run @atlas/qwen3.5-35b-a3b-nvfp4

Default image prefix: azeezish/atlas-gb10

container: azeezish/atlas-gb10:latest

See the Atlas GitHub repository for image tags and release notes.

This single-node example illustrates Atlas settings. Inspect the current @atlas recipe before choosing an image tag or model configuration:

recipe_version: "2"
model: Sehyo/Qwen3.5-35B-A3B-NVFP4
runtime: atlas
container: azeezish/atlas-gb10:latest
max_nodes: 1
metadata:
description: |
Example Qwen3.5-35B (A3B MoE) NVFP4 + MTP configuration on one GB10.
category: agent
model_params: 35B
model_dtype: nvfp4
quantization: nvfp4
kv_dtype: nvfp4
defaults:
port: 8888
host: 0.0.0.0
max_model_len: 8192
kv_cache_dtype: nvfp4
gpu_memory_utilization: 0.88
scheduling_policy: slai
speculative: true
mtp_quantization: nvfp4
enable_prefix_caching: true

When no command template is provided, sparkrun generates spark serve {model} with flags derived from defaults via the Atlas flag map.

Use runtime and defaults and omit command for the supported settings below. Add a command only when you need an unusual engine flag that sparkrun does not map. These are the flags generated by sparkrun; the selected engine image must also support them.

Set values under defaults, for example port: 8000, or override them with -o port=9000. Unset values are omitted unless a runtime/platform default supplies them. Boolean toggles emit a bare flag when true and omit it when false, except where the table lists an explicit negative or valued form. Mapping/list values are rendered as one JSON argument for flags that accept JSON. Unknown keys do not automatically become engine flags.

The generated command is spark serve MODEL; model is a top-level recipe field. Value-taking boolean options pass a lowercase true or false, while boolean toggles only emit a bare flag when enabled.

Mappings: sparkrun 0.4.0 source.

Engine flagKey under defaultsValue form
--portportValue
--hosthostValue
--tp-sizetensor_parallelValue
--gpu-memory-utilizationgpu_memory_utilizationValue
--max-seq-lenmax_model_lenValue
--max-num-seqsmax_num_seqsValue
--max-prefill-tokensmax_num_batched_tokensValue
--model-nameserved_model_nameValue
--kv-cache-dtypekv_cache_dtypeValue
--ep-sizeep_sizeValue
--max-batch-sizemax_batch_sizeValue
--block-sizeblock_sizeValue
--kv-high-precision-layerskv_high_precision_layersValue
--tool-call-parsertool_call_parserValue
--tool-max-tokenstool_max_tokensValue
--max-inter-tool-prosemax_inter_tool_proseValue
--fp8-kv-calibration-tokensfp8_kv_calibration_tokensValue
--fp8-kv-headroomfp8_kv_headroomValue
--scheduling-policyscheduling_policyValue
--tbt-deadline-mstbt_deadline_msValue
--max-prefill-tokensmax_prefill_tokensValue
--prefill-varlen-batchprefill_varlen_batchExplicit true / false value
--oom-guard-mboom_guard_mbValue
--request-timeoutrequest_timeoutValue
--swap-space-gbswap_space_gbValue
--warmup-promptwarmup_promptValue
--auto-compactauto_compactValue
--kernel-targetkernel_targetValue
--lm-head-dtypelm_head_dtypeValue
--ssm-cache-slotsssm_cache_slotsValue
--ssm-checkpoint-intervalssm_checkpoint_intervalValue
--ssm-h-dtypessm_h_dtypeValue
--ssm-rollback-modessm_rollback_modeValue
--ssm-batched-recurrentssm_batched_recurrentExplicit true / false value
--ssm-tail-midchunkssm_tail_midchunkExplicit true / false value
--gdn-fused-normgdn_fused_normExplicit true / false value
--mtp-quantizationmtp_quantizationValue
--mtp-vocabmtp_vocabValue
--mtp-gatemtp_gateValue
--num-draftsnum_draftsValue
--draft-modeldraft_modelValue
--exact-verifyexact_verifyExplicit true / false value
--dflash-gammadflash_gammaValue
--dflash-window-sizedflash_window_sizeValue
--max-thinking-budgetmax_thinking_budgetValue
--default-top-n-sigmadefault_top_n_sigmaValue
--default-min-pdefault_min_pValue
--content-loop-watchdogcontent_loop_watchdogExplicit true / false value
--content-loop-min-repeatscontent_loop_min_repeatsValue
--default-chat-template-kwargsdefault_chat_template_kwargsValue
--model-from-pathmodel_from_pathValue
--cache-dircache_dirValue
--gpu-ordinalgpu_ordinalValue
--dumpdumpValue
--vision-max-pixelsvision_max_pixelsValue
--vision-remote-image-max-mbvision_remote_image_max_mbValue
--vision-remote-image-timeout-svision_remote_image_timeout_sValue
--video-ffmpeg-pathvideo_ffmpeg_pathValue
--video-fpsvideo_fpsValue
--video-max-framesvideo_max_framesValue
--video-decode-timeout-svideo_decode_timeout_sValue
--lora-adapterlora_adapterValue
--max-lora-rankmax_lora_rankValue
--max-lorasmax_lorasValue
--lora-stageablelora_stageableValue
--lora-stageable-disklora_stageable_diskValue
--src-langsrc_langValue
--tgt-langtgt_langValue
--high-speed-swap-dirhigh_speed_swap_dirValue
--high-speed-swap-gbhigh_speed_swap_gbValue
--high-speed-swap-resident-blockshigh_speed_swap_resident_blocksValue
--high-speed-swap-rankhigh_speed_swap_rankValue
--high-speed-swap-qdhigh_speed_swap_qdValue
--high-speed-swap-cache-blocks-per-seqhigh_speed_swap_cache_blocks_per_seqValue
--high-speed-swap-graphhigh_speed_swap_graphExplicit true / false value
--disable-tool-grammardisable_tool_grammarExplicit true / false value
--enable-prefix-cachingenable_prefix_cachingBoolean toggle; true emits the flag
--speculativespeculativeBoolean toggle; true emits the flag
--self-speculativeself_speculativeBoolean toggle; true emits the flag
--ngram-speculativengram_speculativeBoolean toggle; true emits the flag
--dflashdflashBoolean toggle; true emits the flag
--disable-thinkingdisable_thinkingBoolean toggle; true emits the flag
--high-speed-swaphigh_speed_swapBoolean toggle; true emits the flag
--require-authrequire_authBoolean toggle; true emits the flag
--adaptive-samplingadaptive_samplingBoolean toggle; true emits the flag
--auto-swapauto_swapBoolean toggle; true emits the flag
--no-auto-swapno_auto_swapBoolean toggle; true emits the flag
--auth-tokens-fileauth_tokens_fileValue
--disable-template-overridesdisable_template_overridesBoolean toggle; true emits the flag
--vision-allow-remote-imagesvision_allow_remote_imagesBoolean toggle; true emits the flag
--vision-remote-image-allow-privatevision_remote_image_allow_privateBoolean toggle; true emits the flag
--video-allow-ffmpegvideo_allow_ffmpegBoolean toggle; true emits the flag
--no-fast-loadno_fast_loadBoolean toggle; true emits the flag
--fast-load-prefetch-shardsfast_load_prefetch_shardsBoolean toggle; true emits the flag
--no-tuino_tuiBoolean toggle; true emits the flag
--profileprofileBoolean toggle; true emits the flag
--check-kernelscheck_kernelsBoolean toggle; true emits the flag
--dangerously-allow-unresolved-kernel-lookupsdangerously_allow_unresolved_kernel_lookupsBoolean toggle; true emits the flag
--auth-tokenapi_keyValue

auth_token is an alias for api_key. For the prefill budget, choose either max_num_batched_tokens or max_prefill_tokens: both map to the same engine flag. Expert parallelism uses ep_size for this runtime. draft_model_revision pins draft-model acquisition and has no serve flag.

Sparkrun generates --rank, --world-size, --master-addr, and --master-port from placement; worker ranks receive --port 0. The parallelism section below explains overlapping and orthogonal TP/EP layouts.

When a recipe sets draft_model in defaults, sparkrun automatically pre-syncs that model to all target hosts alongside the primary model — the same way it pre-syncs draft models for vLLM and SGLang speculative recipes. This happens during the distribution phase so the container never has to fetch the draft from HuggingFace at launch time (which would fail under sparkrun’s offline-hub default).

defaults:
speculative: true
draft_model: avarok/qwen3.5-35b-a3b-nvfp4-draft
num_drafts: 2

This works with any of the speculative modes (speculative, self_speculative, ngram_speculative, dflash); the only requirement is that the draft model name is set in defaults.draft_model.

Atlas derives world_size from tensor_parallel and ep_size:

  • world_size == tp_size * ep_size — orthogonal mesh (TP and EP on independent ranks)
  • world_size == tp_size == ep_size — overlapping groups (both groups share the same physical ranks)

sparkrun maps the resulting world size to physical hosts automatically and launches the ranks through the native NCCL path. The earlier single-node restriction is removed in the pinned 0.4.0 source. For a single-node recipe, set tensor_parallel: 1 and omit ep_size (or set it to 1).

Atlas uses the shared Docker seccomp profile, which permits io_uring_setup, io_uring_enter, and io_uring_register while retaining the rest of the bundled Moby policy. The Atlas runtime does not add seccomp=unconfined; disabling seccomp is an explicit executor configuration choice.

The Atlas runtime adds these capabilities on every Docker launch, including when high_speed_swap is disabled:

OptionCapability
--cap-add=IPC_LOCKPermits locking memory, used by pinned-memory and RDMA paths
--cap-add=SYS_NICEPermits elevated scheduling and priority operations

These are Atlas additions. The shared Docker settings supply unlimited locked memory with --ulimit memlock=-1:-1 for normal non-privileged launches; that limit is separate from the IPC_LOCK capability. See Capabilities and locked memory.

SYS_NICE describes the current launch configuration, not a universal requirement for high-speed swap: Linux 5.13 and later allow SQPOLL without special privileges. See the Linux io_uring documentation.

Atlas does not declare streaming inference readiness observers. Default run uses supported endpoint checks and waits for required hooks; it does not report unavailable first-token timings. See run.

The example’s memory utilization is an explicit recipe setting, separate from scheduling reservations. Model files, draft models, and runtime caches have different validation and cleanup needs; see Model Cache Management.