Builders

Builders prepare container images before sparkrun distributes them to target hosts and launches the runtime. They run during Phase 2 of the launch lifecycle.

Most recipes skip builder work entirely — the default docker-pull builder is a no-op since the distribution phase handles image pulling on each host. The eugr builder is the main “real” builder, used for the eugr/spark-vllm-docker family of images.

eugr is pull-first as of July 2026

Upstream eugr transitioned from local builds to published, stable container images. sparkrun’s eugr builder mirrors that: it normally just pulls the relevant image and does no local build at all.

A local build only happens when the recipe’s build_args explicitly ask for one — see When a build actually happens. The build machinery documented further down still exists and is still correct, but it is now the exception rather than the common path.

Built-in builders

Builder	Description	When used
docker-pull	Default. Does nothing — relies on the distribution phase to pull images.	All recipes without an explicit builder field.
eugr	Resolves eugr/spark-vllm-docker images. Pulls a prebuilt image by default; runs build-and-copy.sh only when build_args request a wheels or custom build. The builder does not handle mods — those are resolved and expanded into pre_exec entries by Phase 1, builder-agnostic (see Recipes › Mods for the full mods spec).	Recipes with builder: eugr, recipes using a recognized eugr :latest image (sentinel images, auto-detected), or recipes with build_args. The presence of mods alone also triggers it for backward compatibility with the original eugr-centric workflow.

When a build actually happens

sparkrun mirrors build-and-copy.sh’s own rule: a build is requested when build_args contains any token that isn’t a known pull-compatible no-op.

• Pull-compatible (no build): --tf5, --pre-tf, --pre-transformers. These are deprecated no-ops — tf5 and non-tf5 now produce identical images.
• Everything else forces a build: --use-wheels (the standard nightly wheels build), plus custom-build flags like --rebuild-vllm, --rebuild-flashinfer, --vllm-ref, --exp-mxfp4, --force-*-download, --apply-*-pr. Flag values count too, which is harmless — the flag itself already forced the build.

So an ordinary recipe with no build_args, or one carrying only the legacy tf5 flags, is always a pull.

Sentinel container images

The eugr builder recognizes four :latest image refs as sentinels:

• ghcr.io/spark-arena/dgx-vllm-eugr-nightly:latest
• ghcr.io/spark-arena/dgx-vllm-eugr-nightly-tf5:latest
• the Docker Hub eugr/spark-vllm image, in short and fully-qualified form

All four resolve to the same single non-tf5 build. What they resolve to depends on whether a build was requested:

build_args	Sentinel resolves to
No build requested (the normal case)	The authoritative GHCR nightly, pulled
A build requested (--use-wheels, custom flags)	A local build target, built from upstream wheels

A non-sentinel eugr image that isn’t pullable is reused when already present locally; otherwise sparkrun substitutes its own nightly and pulls, since eugr will not build without an explicit flag.

See vLLM › Sentinel images for the user-facing explanation.

Forcing a fresh image

--rebuild (or builder_config.rebuild in a recipe) means “don’t reuse what’s already here”:

• On the pull path, a registry image can’t be rebuilt, so it is re-pulled fresh — bypassing a stale local or head-node copy of the same tag.
• On the build path, it forces a from-scratch build even when a matching image is present and the build cache would otherwise skip it.

Opting out of pull-first substitution

Pull-first substitution is ON by default. Power users can disable it via defaults.builders.eugr.use_sentinel_image in ~/.config/sparkrun/config.yaml:

defaults:
  builders:
    eugr:
      use_sentinel_image: false   # legacy: use images verbatim, build if missing

Caution

This is the legacy build escape hatch, not a “pull instead” switch. With it set to false, sparkrun stops substituting its own nightly: images are used verbatim, and one that isn’t present is built via build-and-copy.sh the old way. Leave it at the default unless you specifically want that behavior.

Recipes that set build_args request a build regardless of this toggle — it only affects the implicit sentinel mapping and the missing-image fallback.

Persisting build logs

The eugr builder captures all combined stdout/stderr from build-and-copy.sh in memory so a build failure can always report the last ~80 lines of context plus the inferred phase. By default it does not write that output to disk — when the build succeeds the log is discarded, keeping ~/.cache/sparkrun/ lean for typical day-to-day runs.

When you’re actively debugging a misbehaving build and want a full persistent record, enable defaults.builders.eugr.save_build_logs:

defaults:
  builders:
    eugr:
      save_build_logs: true   # persist the full build-and-copy.sh log to disk

With this enabled:

• Each rebuild writes its combined output to <cache_dir>/eugr-builds/<safe_image>-<YYYYMMDD-HHMMSS>.log (and -remote-<host>.log for delegated builds).
• The build log path is logged at INFO+ so you can tail -f it while the build runs.
• On failure, the RuntimeError and the ERROR-level summary both point at the persisted log file.

When save_build_logs is false (the default) and a build fails, the error message instead nudges the user to enable the setting before re-running so the next attempt has a full record on disk. The in-memory error tail is unaffected either way — phase and last-lines context are always surfaced regardless of disk persistence.

Build hygiene and observability

Everything below applies only on the build path, which since the July 2026 pull-first switch is the exception. These are the safety nets that activate when a build does happen:

• --cleanup always appended. sparkrun appends --cleanup to every build-and-copy.sh invocation so the upstream script starts from a clean ./wheels/ directory and an empty set of .*-commit markers each time. This eliminates wheel-state drift across runs (stale commit files, half-restored backups after a failed download, dash-vs-underscore wheel-name collisions). The flag is added to an effective copy of build_args only — the cache identity stored under build_args remains tied to the recipe’s canonical args so subsequent runs still hit the build cache.
• In-memory log capture (always on). Combined stdout/stderr from build-and-copy.sh is captured in memory so the last ~80 lines plus an inferred phase can be surfaced on failure regardless of verbosity. At -v (INFO+) the output also streams to the terminal in real time. Optional on-disk persistence is gated by defaults.builders.eugr.save_build_logs (default false) — see Persisting build logs below.
• Phase-aware error context. On a non-zero build exit, sparkrun parses the captured log against phase markers (FlashInfer build command, vLLM build command, Building runner image with command, FlashInfer build failed, Error: No wheel files found, etc.) and includes the inferred phase in the RuntimeError message — e.g. phase=flashinfer-build (failed) — so the failure point is obvious without rummaging through scrollback.
• Post-build flashinfer smoke test. After a successful build, sparkrun runs docker run --rm --entrypoint= <image> python3 -c "import flashinfer; import flashinfer_cubin; import flashinfer_jit_cache". If any import fails, the broken tag is removed via docker rmi (so the next launch retries from scratch instead of hitting a cached broken image) and a RuntimeError is raised with the captured ImportError — telling the user which sub-package is missing.
• Build cache. When the upstream wheel hashes haven’t changed and the locally tagged image still exists with the same image ID, the rebuild is skipped entirely. Cache entries are stored under <cache_dir>/eugr-build-cache.json. The cache key is host-qualified for delegated builds so per-host artifacts are tracked independently.

Builder plugin interface

Builders extend BuilderPlugin (from sparkrun.builders.base), an SAF Plugin registered under the sparkrun.builder extension point:

Method	Purpose
prepare_image()	Ensure the container image is available. Returns the final image name.
validate_recipe()	Validate builder-specific recipe fields. Returns a list of issue strings.
version_info_commands()	Return shell commands to capture version info from inside the container.
process_version_info()	Process raw command output into flat key-value pairs for runtime info.
collect_container_labels()	Inspect container OCI labels via docker inspect.

Custom builders are discovered via Python entry points in pyproject.toml, following the SAF plugin pattern.

Reading user config defaults

Builders may read soft defaults from the user’s ~/.config/sparkrun/config.yaml under defaults.builders.<builder-name>:

class MyBuilder(BuilderPlugin):
    builder_name = "my-builder"

    def prepare_image(self, image, recipe, hosts, config=None, ...):
        opts = config.get_defaults_builder(self.builder_name) if config else {}
        verbose = bool(opts.get("verbose", False))
        ...

SparkrunConfig.get_defaults_builder(name) returns the matching dict, or {} when the section is absent or malformed. Recipe fields and explicit CLI overrides should always win over these defaults — treat them as the lowest-priority layer.

A symmetric get_defaults_runtime(name) exists for runtime plugins that want the same shape of user-configurable defaults under defaults.runtimes.<runtime-name>.