Skip to content

Python API

sparkrun.api provides console-free operations, typed results/errors, and structured streams and opt-in diagnostic bundles. Frontends own confirmation, rendering, capture policy, and readiness policy. 0.4.0 breaks earlier Python contracts; see the migration map for changed options and imports.

from sparkrun.application import initialize
from sparkrun import api
sctx = initialize()
options = api.RunOptions(
recipe="qwen3-1.7b-vllm",
cluster="mylab",
overrides={"tensor_parallel": 2, "port": 9001},
)
plan = api.plan(options, sctx=sctx)
print(plan.host_list, plan.notes) # Render the preview in your frontend.
# Obtain any required authorization before execution.
result = api.run(options, sctx=sctx, plan=plan)
print(result.workload_id, result.serve_port, result.rc)

Planning can perform transport preparation, hardware discovery, and occupancy queries. It does not launch a workload. Reusing the live RunPlan avoids repeating placement and preserves the inputs shown to the user, but does not freeze remote cluster state. A serialized preview is not a substitute for the retained plan.

Hosts may be explicit, resolved through a cluster, or supplied by application defaults. Runtime settings such as port belong in overrides. Detached API calls return without the CLI’s log attachment/readiness policy; follow=False is the library default and follow only controls legacy foreground attachment. Use api.logs() or an explicit plugin iterator to observe a detached deployment. The daemon provides an operation workflow with readiness and prompts for remote applications. See Daemon API for authenticated HTTP/SSE clients; its operation names are independent of the Python library functions. --foreground CLI behavior is a separate contract.

from sparkrun.application import initialize
from sparkrun import api
sctx = initialize(config_path="/path/to/config.yaml")
recipes = api.search_recipes("qwen", sctx=sctx)
clusters = api.list_clusters(sctx=sctx)

Choose one application/profile and canonical config file before initializing plugins. Repeated initialization keeps that binding; switching it within the process raises an error. api.default_sctx() initializes the same binding but creates a fresh context view. Passing sctx shares cached managers. A fatal bootstrap failure requires repairing the cause and restarting the process. Cluster-specific SSH settings remain local to the operation.

api.resolve_sctx(sctx) reuses a supplied context or resolves the application’s default one. api.features, api.plugins, api.model_cache, and api.setup.run_setup_steps accept sctx= as well. Occupancy and monitoring calls (status, status_report, open_telemetry, open_live_monitor, live_monitor) accept omitted hosts and resolve them from the selected cluster or defaults. A named or default cluster that cannot be loaded raises SparkrunError; it does not silently discard that cluster’s SSH, executor, transport, or offline settings.

Custom frontends can supply an ApplicationProfile to initialize. Its stable application identity, command/package, update sources, selected integrations, and config/cache namespaces determine policy. The controller identity derives from a private seed and canonical config directory; it is not a user-selectable controller name. Moving/copying the directory changes that identity. Child processes inherit the selected profile/config. See the pinned application profile guide.

Use sparkrun.application.child_environment() when launching a child process that must retain the selected application binding.

RunOptions and BuildOptions accept only recipe positionally; pass every other option by keyword. They share host/cluster targeting, overrides, trust, executor configuration, transfer/cache settings, and offline. A build’s resolved mode is available as BuildPlan.offline. Executor keys such as entrypoint in overrides are routed to executor configuration, including for benchmark, proxy-load, and realized-export paths.

Import these returned types directly from sparkrun.api:

TypesUsed by
ClusterStatus, HostOccupancy, RunningWorkload, GpuOccupancystatus() occupancy and monitor frames
ClusterDefinitioncluster= arguments and run/build plans
SchedulingRequest, SchedulingResultschedule()

These are aliases of the core classes; additional fields are non-breaking. status_report() returns a display aggregate whose grouping may change; use status() for occupancy. Treat RunResult.launch_result as opaque: check it only for None, and pass the whole result to lifecycle helpers. Dry runs, ensure reuse, and some native handlers return no launch handle. A telemetry session’s supported operations are snapshot() and close(). Realized recipe documents are the export contract; their pins are diagnostic records.

Pin a compatible Sparkrun version when using api.workloads, api.model_cache, api.systemd_export, the lifecycle helpers below, or api.setup planning/CX7/SSH diagnosis helpers. These may change in a 0.4.x release. The setup runners, SSH bootstrap/mesh, identity checks/rotation, and RDMA diagnostics retain their separately documented contracts.

TaskPublic surface
Plan, launch, stopplan, run, stop, stop_all, RunOptions, RunPlan, RunResult
Prepare assets without inferenceBuildOptions, plan_build, build; materialize resolves container launch specs read-only
Browse/preview recipescatalog_recipes, get_recipe_details, search_recipes, Catalog* payloads
Resolve/import/retainresolve_catalog_recipe, import_recipe, retain_catalog_recipe, cleanup_catalog_imports
Registry/cluster cataloglist_registries, configure_registry, refresh_registries, list_clusters, catalog_cluster_capacity
Placement/occupancyschedule, status, status_report, find_running_intent
Workload metadata and logslist_jobs, open_log_target, logs; Log Streaming API
Diagnostic bundlescollect_diagnostics, capture_diagnostics, DiagnosticsResult; Diagnostics API
Plugin and capability policyapi.plugins.enable/disable, api.features.enable/disable/reset
Monitoringlive_monitor, open_live_monitor, open_telemetry
Setupapi.setup: probes/plans, apply/undo, manifests, SSH audits and rotation
Model cachesparkrun.api.model_cache: resolve_manifest, inspect_model, import_manifest, import_model, repair_model
Model-cache retentionsparkrun.api.model_cache: inventory_cache, adopt_cache, prune_cache, delete_model, pin_revision, reconcile_claim
Gatewaysapi.proxy: lifecycle, model queries, aliases and capability-dependent management
Runtime dialectsrecipe_protocols(recipe) returns RecipeProtocols without planning or host probes
Readiness and endpoint discoverywait_ready, post_launch, discovery_snapshot (experimental)
Export a systemd serviceapi.systemd_export (experimental)
Benchmarkingbenchmark, resume_benchmark, BenchmarkOptions, BenchmarkResult

Browse and preview return display payloads. Resolve to a live recipe before launch; import stages an external recipe, while retain preserves an import for later use. Keep source provenance: a path to a staged import does not automatically make it a trusted local recipe. Refreshing registries or importing is separate from a read-only catalog query. Listing does not clean up abandoned imports; a new import or explicit cleanup_catalog_imports() performs that sweep. See the pinned catalog guide for query/pagination and retain/cleanup examples.

Use api.setup to probe hosts and select the hardware/executor plan, then render findings before applying steps. Probe/readiness and execution share the step model; probing never authorizes mutation. Apply/undo return per-step outcomes and preserve recovery manifests. Obtain confirmation and per-host sudo credentials in the frontend. See SSH Setup and the pinned headless setup examples.

The following experimental helpers expose the decisions used by the setup CLI:

HelperContract
plan_cx7(hosts, *, cluster_name=..., cluster_manager=..., topology="auto", ...)Probe and return a read-only Cx7Plan; no configuration is applied. Accepts interface/port selection, saved topology, and event/choice callbacks. Three hosts with two physical ports each are ring candidates.
apply_cx7(plan, *, ssh_kwargs=..., dry_run=..., passwords=..., ...)Apply the plan and return per-host results plus topology/interface persistence outcomes. The frontend supplies sudo credentials.
save_cx7_topology(plan, *, failed=..., dry_run=..., ...)Record a complete, successfully applied topology; refuse partial application.
diagnose_ssh(hosts, user, *, local_user=..., on_event=...)Return SshDiagnosticReport and per-host diagnoses. Requires OpenSSH ControlMaster support and may prompt for a password on the controlling terminal; it is not an unattended daemon operation.

These helpers, including find_local_public_key and validate_cx7_options, are Python-only and do not add daemon operations. Check Cx7ApplyResult.failed and persistence results before reporting success.

sparkrun.api.systemd_export separates planning, rendering, and installation:

import sparkrun.api.systemd_export as systemd_export
service_plan = systemd_export.plan(
systemd_export.SystemdExportOptions(
recipe="my-recipe.yaml", cluster="mylab", dry_run=True,
),
sctx=sctx,
)
print(service_plan.recipe_yaml, service_plan.cluster_yaml, service_plan.unit_path)

Select exactly one of recipe or job (a recorded workload ID). Use locate_host or the mutating bootstrap_host, then render to produce SystemdExportArtifacts. The separate install_files, install_unit, and start functions perform installation; render_uninstall and uninstall handle removal. The caller owns confirmation and sudo credentials.

Export preserves the source cluster’s full host list and configuration. Placement runs when the service starts. Controller-local env_file values are resolved at export and written into the head node’s private cluster file. See export systemd.

model_cache.import_manifest(request, manifest, ...) imports supplied inventory without contacting the origin. import_model accepts that same inventory or can resolve it once with explicit consent:

from sparkrun.api import model_cache
from sparkrun.models.artifacts import ModelArtifactRequest
request = ModelArtifactRequest("org/model", revision="COMMIT")
reports = model_cache.import_model(
request,
cache_dir="/models/hf",
fetch_manifest=True,
)
for report in reports:
print(report.host, report.status, report.complete)

This example authorizes origin resolution on the controller and imports into its local cache. Pass hosts, ssh_kwargs, or session for remote targets. Resolution uses one immutable inventory for every target and may download a weight index, never model weights. Omit revision only when you intend to resolve the origin’s current default revision.

ParameterContract
manifestOptional ModelArtifactManifest; when supplied, skips origin resolution and the confirmation callback
fetch_manifestDefaults to False; must be True when no manifest is supplied
offlineDefaults to False; True requires a supplied manifest even with fetch consent
confirm_manifestOptional callback receiving the resolved manifest before target caches are opened; returning False raises ModelArtifactError and leaves target manifests unchanged
tokenExplicit origin credentials override local HF configuration; used only for resolution
checksumsHash selected existing files during import; defaults to False

Both calls return tuple[CacheValidationReport, ...]. An incomplete cache can return normally after the valid manifest is saved; inspect complete and status before proceeding. Invalid supplied metadata and origin failures never fall back to inferred local inventory. Import does not repair weights or enroll snapshots in retention. The frontend owns consent and presentation; see manifest import.

Cache management operates once per detected HF cache location, grouping hosts that share a cache. Inventory is read-only, and prune_cache defaults to a preview with a seven-day minimum age:

from sparkrun.api import model_cache
inventory = model_cache.inventory_cache(cache_dir="/models/hf")
preview = model_cache.prune_cache(
cache_dir="/models/hf",
dry_run=True,
min_age_days=7,
max_bytes=536870912000,
)
for report in preview:
print(report.location, report.planned_bytes, report.target_met)

Reports are typed CacheReport/CacheEntry objects that also support mapping access. report.location is a CacheLocation with key, kind (network, local, or unknown), source, hosts, per-host paths, and verified. The top-level host is the execution representative. Mutations verify shared access before operating once per location. Use stable reason_code values for automation, and inspect budget and estimate-kind fields before interpreting bytes.

The default host is localhost; pass hosts, ssh_kwargs, or an existing session for remote caches. Applying cleanup requires dry_run=False. These Python calls do not select a default cluster automatically. max_bytes budgets private tracked allocations, including blobs used only by a selected snapshot; min_free_bytes targets filesystem free space. Check target_met in every cache report even if the call returns normally. Pruning preserves pins, active claims, unknown content, busy caches, active downloads, and blobs still referenced elsewhere.

pin_revision(request, ..., pinned=True) takes a ModelArtifactRequest with a full immutable commit; pinned=False removes its retention pin. reconcile_claim(request, ..., claim=..., confirm_inactive=True) releases an exact claim after the caller has verified all of its workloads and launches stopped. It does not delete model files or discover inactivity. See Model Cache Management for tracking, protection, and host requirements.

Both APIs preview by default and return tuple[CacheReport, ...]:

from sparkrun.api import model_cache
from sparkrun.models.artifacts import ModelArtifactRequest
request = ModelArtifactRequest("org/model", revision="COMMIT")
adoption = model_cache.adopt_cache(
request, cache_dir="/models/hf", dry_run=True, pinned=True,
)
deletion = model_cache.delete_model(
request, cache_dir="/models/hf", dry_run=True,
)

adopt_cache accepts either a request or all_models=True, never both. Its options include offline=False, last_used="now" (or "mtime"), pinned=False, token, and an endpoint for all-model adoption. It validates present files and writes usage records with dry_run=False; it never downloads weights or creates launch manifests. Offline adoption requires HF blob-linked entries.

delete_model deletes one requested revision or all revisions when request.revision is unset. Applying with dry_run=False first previews every cache and returns refusal reports without deleting anything if any target is blocked. Pins, age, and untracked status do not protect explicit deletion. force=True overrides workload claims, but never active downloads or publication locks. Saved manifests and blobs referenced elsewhere are preserved.

The frontend owns confirmation. Pass the confirmed revision set as allowed_revisions when applying deletion to reject revisions that appeared after its preview. The same guard is available on daemon model_cache.delete. An empty collection permits no revisions; omission uses the model/revision selection at execution time. Inspect target_met, reason_code, and actions such as would_adopt / adopted or would_delete / deleted; a normal return is not evidence that a mutation occurred.

api.open_live_monitor opens a reusable LiveMonitorSession combining telemetry and workload occupancy. Keep the session open while your UI polls it, then close it or use a context manager:

from sparkrun import api
with api.open_live_monitor(cluster="mylab", interval=2) as monitor:
frame = monitor.frame()
for activity in frame.hosts:
points = monitor.history(activity.host)
print(activity.host, len(points))
series = monitor.history_snapshot()

frame() is a nonblocking latest snapshot. history(host) returns a copied list of (recorded_at, MonitorSample) tuples, oldest first. The timestamp is the controller’s wall-clock record time, suitable for a shared chart axis; each sample’s backend timestamp may use a different clock or format.

History accrues independently of frame() calls. Omit history_points or pass None to use monitor.history_points from configuration (default 120, about four minutes at the default two-second interval). An explicit history_points argument overrides configuration; values below one are floored to one. The session’s history_points property reports the effective capacity.

Missing samples and repeated reads of the same sample object are skipped, so a stalled stream does not extend history with stale readings. The list can be empty just after opening; an unknown host raises KeyError. Recorded data remains readable after collection stops. History is session-local, not durable storage or part of MonitorFrame.

history_snapshot() returns all selected hosts in order as a list[HostHistory], with each host included once. Each HostHistory has host and an oldest-first tuple of HistoryPoint(recorded_at, sample) values. These DTOs support wire serialization and back the daemon’s initial monitor history event.

MonitorSample includes cpu_name, gpu_count, net_send_bytes_s, and net_recv_bytes_s alongside CPU, memory, GPU, power, and temperature metrics. These fields are strings; unavailable values are empty. Network rates are bytes per second in the API, while the TUI displays adaptive bits-per-second units. Backend support determines which metrics are populated.

status_interval controls occupancy polling separately (default max(interval * 2, 5) seconds). api.live_monitor accepts the same history-window option and yields frames, closing its session when the generator ends. Use open_live_monitor when your frontend needs direct access to historical points.

Use workload_id for public API parameters and result fields: RunPlan, RunResult, StopResult, JobInfo, BenchmarkResult, and IntentMatch all use it. RunOptions.workload_id_override, AmbiguousWorkload.workload_ids, and discovery’s workload_ids use the same naming. The old public cluster_id spellings have no aliases.

Core occupancy/executor types such as RunningWorkload and LaunchResult, stored job metadata, benchmark export metadata, and recipe templates still use cluster_id internally. The daemon projects workload identities as workload_id. Keep this separate from a saved host-cluster name and from a daemon operation’s job_id.

preview = api.stop(workload_id=result.workload_id, dry_run=True, sctx=sctx)
print(preview.workload_id, preview.dry_run, preview.containers_removed)

stop and log resolution share target rules. An explicit workload ID wins over a recipe selector; hosts come from explicit hosts, then recorded job hosts, then an explicit cluster. Recipe selectors discover matching launch intent within the selected cluster/host scope. Where metadata exists, the recorded cluster supplies connection identity, even if the application’s default cluster changed. Ambiguous recipe matches raise AmbiguousWorkload.

A stop preview can contact hosts for discovery but performs no teardown and reports zero removed containers. api.stop_all(hosts=...) or api.stop_all(cluster=...) requires an explicit target; it never chooses the default cluster on the caller’s behalf. The daemon’s workloads.stop_all follows the same rule.

Readiness, discovery, and external launchers

Section titled “Readiness, discovery, and external launchers”

These lifecycle helpers are experimental:

CallBehavior
wait_ready(result, timeout=..., sctx=...)Observe readiness for a launch from this process; requires a launch handle.
wait_ready(workload_id=..., timeout=..., sctx=...)Wait on a recorded workload, including one launched by another process. Uses its recorded recipe, hosts, runtime, port, and connection. Unknown workloads raise JobNotFound.
post_launch(result, readiness=..., sctx=...)Run post-serve hooks and honor stop_after_post; return PostLaunchResult with .stopped. Reuses the launch’s trust verdict.
discovery_snapshot(hosts=..., cluster=..., workload_ids=..., sctx=...)Observe endpoints with coverage information, using the target cluster’s transport and SSH identity.

Pass exactly one of result or workload_id to wait_ready. timeout=None or 0 adds no caller budget; configured readiness deadlines and operation cancellation still apply. Negative timeouts raise ValueError.

discovery_snapshot(metadata_only=True, workload_ids=..., sctx=...) considers recorded jobs without a cluster status sweep or SSH. It still probes HTTP endpoints and returns only healthy ones with mode="metadata"; do not combine this mode with hosts or cluster. Endpoints include their recorded recipe_name. Ordinary discovery uses the default cluster/default hosts when omitted, and falls back to metadata-only coverage only when no host source exists.

api.recipe_protocols(recipe, sctx=...) returns immutable RecipeProtocols tuples without planning, SSH, or Hub lookups. protocols defaults to ("openai",) if the runtime declares none; capabilities describes the runtime’s native capabilities, not the recipe’s declared capabilities:.

Integrations starting processes outside api.run can call replace_intent_deployments(plan, strict=True, include_current=True, sctx=...) after preparation and immediately before starting processes. It replaces only matching intent with overlapping placement and returns stopped workload IDs; strict mode rejects incomplete discovery or unconfirmed teardown. fresh_workload_id(plan) returns a copied plan with a new placement token and canonical workload ID. Normal api.run callers use its existing replacement barrier. To revive recorded recipe state without re-resolution, use Recipe.from_state(job.metadata["recipe_state"]) from sparkrun.plugins.

The experimental sparkrun.api.workloads surface manages recorded ownership and lifecycle state. inventory(owner=..., sctx=...) includes unfinished activations after caller loss as well as saved workloads. Its result contains workloads, providers, and complete:

  • At most 256 workload rows are returned. complete=False means execution was in flight or the result was truncated; missing rows cannot prove absence.
  • complete=True describes complete recorded inventory, not a live health probe.
  • ensure_ready(binding, owner=..., ...) can adopt or wake an existing exact revision even when the recipe source has since changed. Receipt and ownership checks still apply. Recovery or launch resolves the source and validates the requested revision before using it.

The daemon’s workloads.inventory and workloads.ensure_ready expose the same behavior. See Daemon API.

api.proxy.sync() returns ProxySyncResult, including warnings: tuple[str, ...]. These are safe routing diagnostics from a successful import; they do not mean the sync failed. Discovery status exposes a separate warnings list alongside last_error. Handle both when rendering status. ProxyUpdateFailed preserves the gateway’s safe rejection diagnostic; failed discovery reports unexpected exceptions by class name only.

Use api.open_log_target(workload_id, ...) to resolve reusable sources, then read, follow, page, or export their output with target.dump(path, ...). api.logs(workload_id, ...) is the convenience iterator. Public lifecycle APIs and daemon requests/results use workload_id, including RunResult.workload_id. Keep hosts as the full topology and select one source with host, container, or source_id. See Log Streaming API for cancellation, finite export, native Kubernetes, and resource changes.

Diagnostic capture wraps operations without adding fields to RunOptions:

from sparkrun import api
from sparkrun.application import initialize
sctx = initialize()
with api.capture_diagnostics("run.ndjson", sctx=sctx) as capture:
result = api.run(api.RunOptions(recipe="my-recipe.yaml", cluster="lab"), sctx=sctx)
readiness = api.wait_ready(result, sctx=sctx)
if not readiness.ready:
raise RuntimeError("Workload did not become ready")
print(capture.result.partial, capture.result.output_path)

Choose a new output path for each capture. The bundle includes operation-attributed DEBUG logs, feedback, and admitted plugin snapshots, with shared redaction and limits. Nested captures share the outer sink. A returned failure value must be raised or reported with capture.emit_error(phase, error) to mark the outcome. api.collect_diagnostics() performs standalone configuration/host/workload collection. See Diagnostics API for the complete contract and benchmark capture.

RunOptions.trust is boolean and defaults to False. False retains automatic trust for local recipes and trusted registries; otherwise gated execution raises TrustRejected. True supplies explicit authorization. The library never prompts, even with a TTY. A frontend must obtain the user’s authorization before calling the API. trust=None is not a prompting mode.

Call api.untrusted_hooks(plan.recipe, sctx=sctx) to obtain the hooks requiring authorization: a mapping of pre_exec, mods, post_exec, and post_commands to command tuples (mod names for mods). An empty mapping means no hook consent is needed. One verdict covers the whole launch and is decided before preparation or eviction. Dry runs report the required trust without executing hooks. Benchmark frontends can authorize through the recipe_trust decision callback; without one the decision defaults to refusal.

Registry-loaded recipes must retain matching source provenance for automatic trust. Reload stale recipes or obtain explicit authorization. Plugin source trust and recipe-registry trust are separate decisions. See Security.

Operational failures use api.SparkrunError subclasses such as InsufficientCapacity, RecipeNotFound, HostsUnreachable, TrustRejected, JobNotFound, AmbiguousWorkload, and IntegrationUnavailable. InsufficientCapacity retains the requested slots and observed host/status detail. Pure model/plan validation may raise ValueError or TypeError; interrupts propagate. Check public RunResult.rc; plugin launches need not supply a private launch handle.

Benchmark results, callback decisions, and retryable finalization failures have separate contracts: see Benchmark API. For operation events and subscribers see Feedback.

Earlier input/import0.4.0 replacement
RunOptions(port=9001)RunOptions(recipe="my-recipe", overrides={"port": 9001})
RunOptions.diagnostics_pathWrap operations with api.capture_diagnostics; CLI --collect-diagnostics remains
Public cluster_id, cluster_ids, cluster_id_overrideworkload_id, workload_ids, workload_id_override; no aliases
Positional run/build fields after recipeKeyword-only arguments
post_launch() dictionaryPostLaunchResult.stopped
Private runtime protocol queriesapi.recipe_protocols(recipe)
Private recipe-state deserializationRecipe.from_state(state) from sparkrun.plugins
Plugin contracts imported from defining modulesSupported exports in sparkrun.plugins; operational calls remain in sparkrun.api / sparkrun.application
BenchmarkOptions.arenaintegrations={"arena": {}} to select it, {} for none
Private benchmark prompt/completion hooksprogress_callback and decision_callback
sparkrun.arenasparkrun.plugins.sparkarena
sparkrun.api.k8ssparkrun.plugins.k8s.api
sparkrun.orchestration.executors.k8ssparkrun.plugins.k8s.executor
Builder image-only preparation hookprepare(..., builder_context=None)
Per-family installed entry-point groupsModule targets in sparkrun.plugins with explicit selection

Use public imports instead of leading-underscore modules. Pin compatible core and integration releases together. This Python boundary does not retire saved recipe, configuration, or benchmark formats by implication. The full pinned migration guide records detailed contract changes.