What's New in 0.4.0

sparkrun 0.4.0 brings OCI Relay image distribution, managed plugin installation, an optional daemon, and more visibility into startup, model caches, and benchmarks. It also changes the Python and plugin APIs. These highlights cover changes since 0.3.10.

Startup waits for readiness

By default, sparkrun run follows startup logs and returns once the model is ready and required hooks finish. Use --no-follow to wait quietly, --follow to keep watching after readiness, or --no-ready-wait to return after launch checks. Startup timing reports show the observations supported by the runtime and executor.

A failed readiness check or Ctrl+C can leave the detached workload running. See run for lifecycle and exit behavior.

Reusable recipes and hardware-specific settings

Recipes can use include: to extend another recipe and conditional overrides: to select settings for the hardware, runtime, or parallelism of a launch. Explicit environment templates can reference effective configuration and prepared model paths, node ranks, and runtime caches. Authors can declare min_sparkrun_version to catch incompatible core versions before preparation. See Recipe Format.

Model variants retain inherited metadata, including declared weight and KV-cache dtypes. Review model-specific estimates when changing model; override fields, remove individual values with null, or clear the block with metadata: null to use automatic detection.

Use -o key.field=value to adjust one field of a mapping-valued default while preserving its other fields, including nested settings such as speculative_config.draft_load_config.load_format. See mapping overrides.

Offline launches and reproducible exports

run, build, benchmark, and proxy load accept --offline to reuse existing recipes, images, and models, including copies within the cluster. Clusters can save an offline default; --online overrides it for one operation. Recipe loading honors this policy too: offline operations do not fetch recipe URLs or refresh registries.

sparkrun export recipe --realize resolves a recipe against selected hardware and bakes in matching overrides. It pins container digests and the primary model revision by default, with an offline mode for resident artifacts. See Offline launches and Realized exports for requirements and pinning limits.

Prepare assets before launching

The new advanced sparkrun build command prepares images, model files, tuning configurations, and native environments without starting inference. It prepares all selected hosts, including hosts whose GPUs are busy, and retains completed work for reuse. See build for its scope and options.

Managed plugins

The new sparkrun plugins commands install trusted Git sources, inspect compatibility, update plugins, and roll back to a retained generation. Core updates also reconcile compatible managed plugins; --no-update-plugins skips fetching updates while retaining the local compatibility audit.

plugins enable and plugins disable now control loading across bundled, managed, manual, and installed plugins, independently of capability flags. Inventory distinguishes loading state, compatibility, and declared features. Plugin and feature lists show enabled entries by default; add --all to inspect disabled entries too, including in JSON output. Explicitly enabling a gateway capability selects its provider. Plugin authors can use static manifests and lifecycle hooks to plan settings for management actions.

See Managed Plugins for controls, updates, and rollback, and Plugin Development for manifests and extension examples.

The bundled, opt-in lil plugin reads the catalog used by the lil launcher: launch @lil/<Entry> directly or export it as a sparkrun recipe. It defaults on only for alpha. See The lil catalog.

Container image distribution with OCI Relay

0.4.0 includes OCI Relay, enabled by default on every release channel. Run recipes normally: sparkrun sets up the relay and coordinates image delivery to Linux Docker hosts automatically.

• Stream images directly from registries or a local Docker store. Registry transfers can reach receivers without first importing the image on the source.
• Reuse compatible layers already on each host, including across classic Docker and containerd image stores, reducing repeated data transfers.
• Use parallel transfers with bounded resource usage. The relay supports multiple network paths and parallel downloads of large registry blobs.
• See transfer and verification progress per host. Rates, cache reuse, and verified outcomes are available through CLI output and structured events for applications and daemon clients.

sparkrun downloads and verifies the required relay binaries, then stages them to execution hosts; those hosts do not need internet access for relay installation. Cached binaries and resident source images can support offline distribution. For more background, read the OCI Relay blog discussion.

Shared diagnostics and workload logs

CLI, Python, and daemon operations share opt-in, bounded diagnostic bundles. Capture includes operation-attributed DEBUG logs independently of console verbosity, feedback, and plugin-owned snapshots with shared redaction. Benchmark captures save workload log tails before cleanup, preserving evidence for later finalization failures. Daemon jobs expose authenticated bundle downloads.

Applications can resolve reusable log targets, follow output, page older history, cancel readers, and export complete logs with LogTarget.dump(). Exports publish only after a complete read and discard partial output on cancellation or failure. Native Kubernetes logs share the same source-aware API; transport errors stay separate from workload output. See Diagnostics, Diagnostics API, and Log Streaming API.

Improved cluster monitor TUI

The cluster monitor now combines live host metrics, workload occupancy, sensor history, and interactive logs in one terminal interface:

sparkrun cluster monitor --cluster mylab

Host metrics, sensor history, and workload logs. Select the image for full size. Screenshot from PR #322.

• Sensor sparklines in host details show CPU/GPU utilization, temperatures, GPU power and clock, RAM, and network send/receive rates alongside hardware identity.
• Configurable per-host history retains 120 samples by default, about four minutes at the two-second sampling interval. Set monitor.history_points to change the window; collection continues as you switch hosts or view logs. Stalled streams do not fill the history with repeated stale samples.
• Interactive workload logs open with Enter or a mouse click, with runtime colors, regex search and highlighting, container/node/cluster scopes, older history, and file exports.

See cluster monitor controls. Applications can read the same history through the Python API, and daemon subscribers receive recent history before live frames.

Hardware-aware setup and SSH identity checks

Setup now uses shared hardware and executor plans. On DGX Spark, Docker defaults to gpu_access_mode: gpus; CDI setup applies when selected or required. Read-only checks can report duplicate SSH server host keys and machine IDs. A separate, explicitly confirmed rotation workflow handles remediation.

Wizard reruns reuse working SSH meshes and saved CX7 topology, prompting when a configuration change is needed. Setup also handles different sudo passwords per host and reuses accepted credentials for the rest of the command.

DGX Spark’s scheduling cap is now 0.90. The default vLLM/SGLang serving utilization is 0.80 when the recipe and overrides leave it unset. These settings serve different purposes; explicit recipe memory settings still take precedence for the engine. GPU reservations persist across Docker/native workload observations, and launches are bound to their assigned devices. Ordinary workloads reserve GPUs exclusively. See the Setup Wizard, SSH Setup, and Schedulers.

Docker launches now default to a bundled seccomp profile that allows io_uring. Custom profiles and explicit policies remain available; see Docker launch defaults.

Proxy services

sparkrun proxy systemd installs, inspects, and uninstalls a persistent gateway service on Linux. Normal proxy start/stop commands use the installed unit, and foreground supervision follows the gateway across managed restarts. See Proxy service management.

Inference service exports retain the complete cluster definition and choose placement each time the service starts. See export systemd.

Gateway-specific settings now live under gateways.<name> in proxy.yaml. LiteLLM accepts router_settings and optional balancing: least-in-flight to route requests to the least-loaded backend serving their model. Discovery reports successful imports with routing warnings separately from failed sweeps. See proxy configuration.

Trust before replacing a workload

Untrusted pre_exec, mods, post_exec, and post_commands now share one authorization decision before preparation or eviction. Interactive launches list the hooks and ask once; non-interactive launches require explicit authorization. Dry runs report the required trust without executing hooks. proxy load also accepts --trust. See Security & Trust.

stop --dry-run previews target resolution without stopping workloads. Unreadable registry and proxy configuration files are preserved rather than overwritten with partial defaults.

Optional daemon

Applications can use an authenticated HTTP API with durable jobs, structured events, prompts, and cancellation. Domain-based operations and convenient HTTP paths cover workload lifecycle, catalog, cluster, setup, and benchmark actions. Plugins receive their own API namespaces. Gateway discovery and workload recovery share core ownership and lifecycle controls, with dedicated service-job capacity.

The daemon is enabled by default on alpha and opt-in on stable/beta; ordinary CLI commands still run directly. See Daemon Mode for connections, job lifecycle, and platform support, and Daemon API for the reference.

Cache validation and benchmark progress

Model manifests record expected files at an immutable revision. New cache commands inspect inventory, resolve metadata, import manifests, and repair damaged or missing files. Metadata and checksum validation are available alongside the legacy default. Cache-local usage records now track model preparation and workload use. The model-cache group is visible in normal help and supports named/default clusters, readable tables, and --json reports.

• Adopt existing downloads from Hugging Face tools or notebooks into usage tracking without downloading them again. Preview adoption, then apply it; optionally pin revisions that other applications still use.
• Import expected inventory directly from the origin when you do not have a manifest file. Review the resolved commit before saving; supplied manifests remain the reproducible, offline option. Import validates files without downloading weights or enrolling them in retention.
• Choose retention or explicit deletion. List usage, pin revisions, preview LRU cleanup, or delete a specific model/revision. Cleanup can reclaim blobs used only by the removed snapshot while preserving blobs referenced elsewhere.
• Handle shared caches once. Inventory and retention group hosts sharing a recognized cache location, avoiding repeated actions and duplicate byte counts.
• Reuse unchanged weights across revisions. Preparation can link existing HF blobs and matching files from other prepared revisions before transferring missing content. Parallel copies retain configurable per-host and total limits.

Optional automatic cleanup remains off by default and preserves pins, workload claims, and unknown content. Runtime compilation caches have separate reuse and cleanup controls. See Model Cache Management.

Tool benchmarks now show scenario verdicts and progress with tool-eval-bench v2.8.0. Resume remains suite-level, and unknown totals are reported without an invented percentage. Measurement results are retained when export or publication fails, so finalization can be retried without repeating measurements. See Performance Benchmarking, Tool Call Benchmarking, and Benchmark API.

SGLang dense FP8 tuning

sparkrun tune sglang --mode fp8 tunes dense block-FP8 kernels using explicit per-rank weight shapes. Compatible saved configurations are installed into future Docker serving containers. See SGLang FP8 tuning for shape selection and launch behavior.

Hybrid model sizing and vLLM options

Memory estimates now distinguish per-token attention KV cache from fixed per-sequence recurrent state for NemotronH and Qwen3-Next/Qwen3.5-style models. Missing recurrent-state metadata still produces a lower-bound estimate. See VRAM estimation.

vLLM recipes can use more mapped defaults for Mamba cache dtypes, hybrid prefix caching, parser plugins, and media options. Draft models in speculative_config are prepared from either YAML mappings or JSON strings, honoring CLI field overrides. See vLLM.

Breaking Python and plugin API changes

• Initialize applications with sparkrun.application.initialize and reuse the returned context. Frontends own confirmation, rendering, and readiness policy; detached API launches do not inherit CLI log-following and wait behavior.
• Put runtime settings such as port in RunOptions.overrides. Benchmark integrations use BenchmarkOptions.integrations, with public progress and decision callbacks.
• Public results, lifecycle parameters, and daemon payloads use workload_id; the earlier public cluster_id spellings have no aliases. Run/build options accept only recipe positionally; other fields are keyword-only.
• Readiness can attach to an existing workload by ID. New helpers expose runtime protocols, recorded-job endpoint discovery, CX7 planning, SSH diagnosis, and systemd export. Experimental lifecycle/setup/cache surfaces are identified in the Python reference.
• Plugins declare SPARKRUN_PLUGIN_API_VERSION = 1. Installed integrations use the shared sparkrun.plugins entry-point group and explicit selection.
• Import supported extension contracts from sparkrun.plugins. Equal repeated registrations are harmless; changed definitions raise PluginConflictError, even when registered by the same owner.
• Builders implement prepare(..., builder_context=None) in place of the old image-preparation hook. Arena and Kubernetes Python modules moved under sparkrun.plugins.

See Python API, Plugin Development, and Builders for the updated contracts.