Skip to content

Builders

Builders prepare container images or native host environments. They run during Phase 2 of the launch lifecycle, or independently through sparkrun build to prepare assets before starting inference.

The default docker-pull preparation leaves the image reference unchanged because distribution owns pulls. --rebuild forces a fresh pull, including this default path. The eugr builder is the main “real” builder, used for the eugr/spark-vllm-docker family of images.

BuilderDescriptionWhen used
docker-pullDefault preparation leaves the image reference unchanged; distribution pulls it. --rebuild forces a fresh registry pull.All recipes without an explicit builder field.
eugrResolves eugr/spark-vllm-docker images. Pulls a prebuilt image by default; runs build-and-copy.sh only when build_args request a wheels or custom build. The builder does not handle mods — those are resolved and expanded into pre_exec entries by Phase 1, builder-agnostic (see Recipes › Mods for the full mods spec).Recipes with builder: eugr, recipes using a recognized eugr :latest image (sentinel images, auto-detected), or recipes with build_args. The presence of mods alone also triggers it for backward compatibility with the original eugr-centric workflow.

The bundled uv-venv builder (alias venv) pairs with executor: local. It installs dependencies with uv and writes an activation file that the local executor automatically sources. It requires builder.uv_venv and executor.local; see Feature Flags.

builder: uv-venv
executor: local
builder_config:
python: "3.12"
requirements_file: requirements.txt
torch_backend: auto
verify_only: true

This fragment expects an already-prepared environment matching the requested dependencies. verify_only: true refuses to install missing or changed dependencies; omit it or use false for initial provisioning. Reuse checks the environment’s Python executable and dependency-hash marker, not a full audit of installed packages.

Requirements can also come from an inline requirements list or a pyproject file. File paths resolve on the controller relative to the recipe, and their contents contribute to the environment hash. venv_path and env_file can override generated locations; cuda_home supplies a CUDA toolkit path. An offline launch requires uv to exist and restricts dependency acquisition to local files/caches. See build for preparation-only behavior.

sparkrun mirrors build-and-copy.sh’s own rule: a build is requested when build_args contains any token that isn’t a known pull-compatible no-op.

  • Pull-compatible (no build): --tf5, --pre-tf, --pre-transformers. These are deprecated no-ops — tf5 and non-tf5 now produce identical images.
  • Everything else forces a build: --use-wheels (the standard nightly wheels build), plus custom-build flags like --rebuild-vllm, --rebuild-flashinfer, --vllm-ref, --exp-mxfp4, --force-*-download, --apply-*-pr. Flag values count too, which is harmless — the flag itself already forced the build.

So an ordinary recipe with no build_args, or one carrying only the legacy tf5 flags, is always a pull.

The eugr builder recognizes four :latest image refs as sentinels:

  • ghcr.io/spark-arena/dgx-vllm-eugr-nightly:latest
  • ghcr.io/spark-arena/dgx-vllm-eugr-nightly-tf5:latest
  • the Docker Hub eugr/spark-vllm image, in short and fully-qualified form

All four resolve to the same single non-tf5 build. What they resolve to depends on whether a build was requested:

build_argsSentinel resolves to
No build requested (the normal case)The authoritative GHCR nightly, pulled
A build requested (--use-wheels, custom flags)A local build target, built from upstream wheels

A non-sentinel eugr image that isn’t pullable is reused when already present locally; otherwise sparkrun substitutes its own nightly and pulls, since eugr will not build without an explicit flag.

See vLLM › Sentinel images for the user-facing explanation.

--rebuild (or builder_config.rebuild in a recipe) means “don’t reuse what’s already here”:

  • On the pull path, a registry image can’t be rebuilt, so it is re-pulled fresh — bypassing a stale local or head-node copy of the same tag.
  • On the build path, it forces a from-scratch build even when a matching image is present and the build cache would otherwise skip it.

Pull-first substitution is ON by default. Power users can disable it via defaults.builders.eugr.use_sentinel_image in ~/.config/sparkrun/config.yaml:

defaults:
builders:
eugr:
use_sentinel_image: false # legacy: use images verbatim, build if missing

Recipes that set build_args request a build regardless of this toggle — it only affects the implicit sentinel mapping and the missing-image fallback.

The eugr builder captures all combined stdout/stderr from build-and-copy.sh in memory so a build failure can always report the last ~80 lines of context plus the inferred phase. By default it does not write that output to disk — when the build succeeds the log is discarded, keeping ~/.cache/sparkrun/ lean for typical day-to-day runs.

When you’re actively debugging a misbehaving build and want a full persistent record, enable defaults.builders.eugr.save_build_logs:

defaults:
builders:
eugr:
save_build_logs: true # persist the full build-and-copy.sh log to disk

With this enabled:

  • Each rebuild writes its combined output to <cache_dir>/eugr-builds/<safe_image>-<YYYYMMDD-HHMMSS>.log (and -remote-<host>.log for delegated builds).
  • The build log path is logged at INFO+ so you can tail -f it while the build runs.
  • On failure, the RuntimeError and the ERROR-level summary both point at the persisted log file.

When save_build_logs is false (the default) and a build fails, the error message instead nudges the user to enable the setting before re-running so the next attempt has a full record on disk. The in-memory error tail is unaffected either way — phase and last-lines context are always surfaced regardless of disk persistence.

Everything below applies only on the build path, which since the July 2026 pull-first switch is the exception. These are the safety nets that activate when a build does happen:

  • --cleanup always appended. sparkrun appends --cleanup to every build-and-copy.sh invocation so the upstream script starts from a clean ./wheels/ directory and an empty set of .*-commit markers each time. This eliminates wheel-state drift across runs (stale commit files, half-restored backups after a failed download, dash-vs-underscore wheel-name collisions). The flag is added to an effective copy of build_args only — the cache identity stored under build_args remains tied to the recipe’s canonical args so subsequent runs still hit the build cache.
  • In-memory log capture (always on). Combined stdout/stderr from build-and-copy.sh is captured in memory so the last ~80 lines plus an inferred phase can be surfaced on failure regardless of verbosity. At -v (INFO+) the output also streams to the terminal in real time. Optional on-disk persistence is gated by defaults.builders.eugr.save_build_logs (default false) — see Persisting build logs below.
  • Phase-aware error context. On a non-zero build exit, sparkrun parses the captured log against phase markers (FlashInfer build command, vLLM build command, Building runner image with command, FlashInfer build failed, Error: No wheel files found, etc.) and includes the inferred phase in the RuntimeError message — e.g. phase=flashinfer-build (failed) — so the failure point is obvious without rummaging through scrollback.
  • Post-build flashinfer smoke test. After a successful build, sparkrun runs docker run --rm --entrypoint= <image> python3 -c "import flashinfer; import flashinfer_cubin; import flashinfer_jit_cache". If any import fails, the broken tag is removed via docker rmi (so the next launch retries from scratch instead of hitting a cached broken image) and a RuntimeError is raised with the captured ImportError — telling the user which sub-package is missing.
  • Build cache. When the upstream wheel hashes haven’t changed and the locally tagged image still exists with the same image ID, the rebuild is skipped entirely. Cache entries are stored under <cache_dir>/eugr-build-cache.json. The cache key is host-qualified for delegated builds so per-host artifacts are tracked independently.

Builders extend BuilderPlugin (from sparkrun.plugins), an SAF Plugin registered under the sparkrun.builder extension point:

MethodPurpose
prepare()Ensure the container image is available. Returns the final image name.
validate_recipe()Validate builder-specific recipe fields. Returns a list of issue strings.
version_info_commands()Return shell commands to capture version info from inside the container.
process_version_info()Process raw command output into flat key-value pairs for runtime info.
collect_container_labels()Inspect container OCI labels via docker inspect.

Custom builders are scanned from plugin modules. Installed integrations use module targets in the shared sparkrun.plugins entry-point group and explicit selection; see Plugin Development. The internal sparkrun.builder extension identifier is not a package entry-point group.

Builders may read soft defaults from the user’s ~/.config/sparkrun/config.yaml under defaults.builders.<builder-name>:

from sparkrun.plugins import BuilderPlugin
class MyBuilder(BuilderPlugin):
builder_name = "my-builder"
def prepare(self, image, recipe, hosts, config=None, dry_run=False,
transfer_mode="local", ssh_kwargs=None, builder_context=None):
opts = config.get_defaults_builder(self.builder_name) if config else {}
verbose = bool(opts.get("verbose", False))
# Prepare your image/environment here, honoring dry_run.
return image

SparkrunConfig.get_defaults_builder(name) returns the matching dict, or {} when the section is absent or malformed. Recipe fields and explicit CLI overrides should always win over these defaults — treat them as the lowest-priority layer.

A symmetric get_defaults_runtime(name) exists for runtime plugins that want the same shape of user-configurable defaults under defaults.runtimes.<runtime-name>.