Daemon API

The daemon exposes authenticated JSON operations and SSE streams over HTTP protocol 1 (/v1). This reference covers the API bundled with sparkrun: 86 core operations and 8 experimental Kubernetes operations. Availability depends on plugin loading policy, capability flags, and the caller’s permissions. Use the running daemon’s GET /v1/ops response to discover its supported API.

See Daemon Mode for startup, secure connections, configuration, gateway discovery, and recovery. The daemon CLI is a reference client.

Requests and jobs

Send a JSON object to either POST /v1/ops/{name} or the operation’s advertised http_path. Both use the same authentication, schema validation, idempotency, and admission path. For example, workloads.run has alias POST /v1/workloads/run. Query aliases also use POST, with {} when there are no arguments.

Kind	HTTP response	Where to read the result
query	200 with the declared result	Response body; no durable job is created
job	202 with job_id and events_url	GET /v1/jobs/{id} → result after success
stream	Dedicated SSE endpoint	Stream events; no automatic POST alias

A job acknowledgement looks like:

{
  "job_id": "JOB_ID",
  "events_url": "/v1/jobs/JOB_ID/events"
}

A query is bounded inline execution and may include administrative mutations. Its permission and description determine what it does. Only operations declaring cacheable: true use the short query cache.

HTTP 202 confirms acceptance. It does not mean that the operation succeeded. A workload’s workload_id identifies a deployment; a daemon job_id identifies one invocation. Benchmark IDs are separate again.

For a cluster-create request with an admin token:

from pathlib import Path
from sparkrun.daemon.client import Client

client = Client(
    url="https://controller:8443",
    token=Path("/private/daemon.token").read_text().strip(),
    ca_cert="/private/ca.pem",
)
info = client.info()  # Checks the HTTP protocol; includes the credential's manager.
operations = client.request("GET", "/v1/ops")
create = next(op for op in operations if op["name"] == "cluster.create")
submission = client.request(
    "POST",
    create["http_path"],
    {"name": "lab", "hosts": ["spark-a", "spark-b"]},
    headers={"Idempotency-Key": "create-lab-001"},
)
job = client.request("GET", f"/v1/jobs/{submission['job_id']}")

Poll the job until terminal, or follow its event stream and then fetch the job. Use result on success and inspect error on failure. A run plan is a display preview, not a resumable execution handle; the live worker retains its resolved plan through approval, readiness, and post-launch hooks.

States, prompts, and cancellation

Nonterminal states are queued, starting, running, waiting_input, and cancelling. Terminal states are succeeded, failed, cancelled, and lost.

For a pending prompt, obtain the user’s answer and send exactly prompt_id and value to POST /v1/jobs/{id}/input. Approval/decision values are booleans; secret values are strings and require separate secret-prompt permission. Use the ID supplied by the job’s current prompt:

{"prompt_id": "PROMPT_ID_FROM_JOB", "value": true}

Prompt metadata is durable, while secret answers remain in worker memory and are redacted from subsequent events/results. The default prompt timeout is 300 seconds. Do not place credentials in operation arguments or log messages.

DELETE /v1/jobs/{id} requests cancellation. It does not delete history or roll back remote effects. Disconnecting a client does not cancel a job. A lost worker is never automatically rerun; inspect and reconcile uncertain effects before starting another mutation.

Idempotency

Set Idempotency-Key on job submissions. The key is scoped to the authenticated principal: the same operation, version, arguments, and manager return the same job through either URL. A conflicting submission returns 409. A later intentional action needs a new key.

Idempotency is bounded by retention. Terminal results/events default to seven days; idempotency tombstones remain for 30 days, and active jobs retain their keys beyond that window. After a lost response, inspect the existing job or retry with the same key. Never treat a disconnect, expired key, or retryable error as authorization to submit an uncertain mutation with a fresh key.

Authentication and identity

TCP requires a bearer token, including loopback. /v1/health is the only public route. Same-user Unix peers without a bearer token have local admin and secret permissions; an explicit bearer selects its own scope and manager on Unix sockets. Remote TCP requires TLS unless the server explicitly permits plaintext.

Permissions are hierarchical: read → operate → admin. Secret-prompt permission is independent. Non-admin principals see only their own jobs and must retain the operation’s permission to inspect or answer them. Tokens are issued, listed, and revoked locally through sparkrun daemon token; there is no HTTP token-admin API. X-Sparkrun-Client is only an advisory display label.

Identity	Meaning
Authenticated principal	Owns daemon jobs and scopes retry keys
Credential manager	Stable workload management identity, reported by /v1/info
Operation owner	Plugin declaration owner from its loader; core for host operations
Workload owner	Manager recorded for the deployed workload

The default workload manager is daemon. A locally issued daemon token create NAME --manager automation credential grants manager automation; operation arguments cannot select it. Non-default manager tokens issued by the CLI/integration helper share principal manager:<name>, preserving access to retained jobs and retry keys across credential replacement. Accepted jobs keep their admitted identity after token revocation.

workloads.run and workloads.ensure_ready use the manager for new workload ownership. Adoption of an existing healthy endpoint does not transfer ownership. Guarded stop/sleep/wake/reconcile require the matching owner, exact workload ID, and recipe revision. Another manager can observe or route to a healthy endpoint. workloads.stop_matching and workloads.stop_all remain broad operator controls. This is an automation guard within a single-account permission model, not tenant isolation. Manager identity is separate from permission scope and does not narrow the permissions granted to a credential.

Recipe-hook trust remains separate: TCP requests need both server opt-in and explicit trust: true. See Security & Trust and Daemon Mode.

Discovery and schemas

GET /v1/ops returns a list filtered by permissions and feature flags. Each entry contains name, owner, local_name, version, kind, scope, description, resources, http_path, request_schema, and result_schema. Use these advertised names and paths rather than reconstructing plugin URLs. All operations in this reference currently have operation version 1, separate from the HTTP protocol version.

/openapi.json describes all registered operations, including higher-scope and feature-disabled declarations. It is not a filtered capability list. The Swagger UI at /docs supports bearer authorization. An alias’s OpenAPI response schema for a job describes its acknowledgement; x-sparkrun-operation.result points to its final result schema.

OpenAPI also describes Job, Prompt, Error, Event, and Info envelopes. Successful job results discriminate by operation name and version. Query and worker results are encoded, redacted, and validated against their declared schemas before being cached or stored. Invalid results produce ResultContractError with code invalid_operation_result. Stored job results have a 2 MiB encoded-size limit; request safety limits are distinct from result constraints.

Omit optional request fields unless the schema explicitly permits null. Requests reject unknown top-level fields. The operation entries below list documented top-level arguments and whether they are required. null means the published schema accepts it; it does not guarantee identical normalization by every underlying API. Consult discovery/OpenAPI for nested properties, enums, bounds, and full result schemas. Results may be objects, arrays, or booleans; there is no universal success envelope shared by queries and jobs.

Reservation policies describe worker execution claims:

Policy	Meaning
auto	Infer destinations; unresolved mutations use a conservative controller-wide claim
controller	Explicit controller-wide reservation
none	Skip the worker reservation; handlers may still take domain locks or public-API reservations

These policies do not bypass authorization. Host-owned proxy.serve and proxy.watch are persistent service jobs, admitted against daemon.max_services (default 2), separately from daemon.max_jobs (default 4). Manage their ownership through public proxy/discovery controls. These internal executors have no public operation catalog entry, HTTP alias, or submission endpoint.

Infrastructure routes

These routes manage the daemon, jobs, streams, and reservations; they are separate from generated operation aliases.

Method	Path	Permission	Behavior / input
GET	/v1/health	public	Liveness and protocol; not worker readiness.
GET	/v1/info	read	Profile/config binding, runtime identity, restart-required state.
GET	/v1/ops	read	Authorized, feature-filtered operation metadata and request/result schemas.
POST	/v1/ops/{name}	selected operation	Generic dispatcher; JSON arguments, query 200 or job 202.
GET	/v1/jobs	read + job visibility	Optional state, limit (default 200; clamped 1–1000); bounded list, no cursor.
GET	/v1/jobs/{id}	job owner/admin + operation scope	Durable state, result/error, public prompt metadata; no raw arguments, worker nonce or runtime lease.
DELETE	/v1/jobs/{id}	job access + operate	Requests cancellation; does not delete history or promise remote rollback.
POST	/v1/jobs/{id}/input	job access + operation scope; secret if needed	Exactly prompt_id, value; answer a live worker prompt.
GET	/v1/jobs/{id}/artifacts/diagnostics	job owner/admin + operation scope	Finished job’s NDJSON artifact; expires with job history.
GET	/v1/jobs/{id}/events	job access	Durable SSE; Last-Event-ID; expired cursor → 410, later expiry → gap event.
GET	/v1/events	read + job visibility	SSE snapshots of visible jobs every 2 seconds; no inventory stream or replay cursor.
GET	/v1/streams/logs	read	SSE; optional workload_id, recipe, comma-separated hosts, cluster, host, container, source_id, scope=head/all, nonnegative tail, follow=true/false. Disconnect cancels readers.
GET	/v1/streams/monitor	read	SSE; optional comma-separated hosts, cluster, interval (1–60 seconds). Initial in-memory history backfill, then live frame events.
GET	/v1/reservations	admin	Execution claims and verified owner liveness; dead claims remain until reconciled.
POST	/v1/reservations/{owner}/reconcile	admin	{"acknowledge_effects":true}; releases only a verified dead owner’s claims.
GET	/v1/snapshots/{name}	read	Advisory snapshot of discovery.snapshot; no other selector is accepted.
POST	/v1/maintenance	admin	{"drain":true/false}; admission control, not worker cancellation.
POST	/v1/shutdown	admin	{}; stops API process, workers continue.
GET	/openapi.json	authenticated	Full registered operation schemas, including feature-disabled or higher-scope declarations; not the filtered capability view.
GET	/docs	read	Swagger UI; same authorization boundary.

Streams

Log streams resolve saved Docker or native Kubernetes workloads. hosts specifies the full topology; use host, container, or opaque source_id to select sources. Unknown/invalid query parameters are rejected before streaming. Records include source identity and native resource metadata where applicable. Reader failures become SSE errors; disconnect cancels finite and following readers. Reads never remove saved workload metadata or release cache claims. See Log Streaming API for ordering, source changes, and cancellation. There is no daemon paging route or server-side log target ID.

Job feedback at /v1/jobs/{id}/events supports durable replay with Last-Event-ID. An expired cursor returns 410; expiry after connection produces a gap event. Fetch current job state, then resume from available history. Results too large for an event must be fetched from the job endpoint.

/v1/events sends snapshots of visible jobs every two seconds, without a replay cursor. Logs and monitoring have their own streams and are not job feedback. No combined log/monitor or MCP route is mounted.

Monitor history and live frames

A monitor subscription first receives a history event containing the shared session’s recent telemetry. It can be empty when collection has just started:

{
  "type": "history",
  "queried_at": 1792250000.0,
  "series": [
    {"host": "spark-a", "points": []}
  ]
}

Each populated points entry contains recorded_at (controller wall-clock seconds) and sample (a telemetry sample), ordered oldest first. Samples include hardware identity and network rate fields alongside CPU, memory, and GPU metrics. Network rates are in bytes per second; unavailable sample values are empty strings.

Live events then use type: "frame", queried_at, and hosts with current telemetry and workload occupancy. Dispatch on type: history uses series, so it must not be interpreted as a frame or an empty host inventory. These types are fields in the SSE data JSON, not named SSE events; an EventSource client reads them from its normal message handler.

The history capacity comes from monitor.history_points in the daemon’s configuration (default 120 samples per host). It is not an HTTP query parameter. Repeated reads of the same sample do not add history points. Late subscribers can reuse an existing session’s history, including across overlapping host selections with matching settings. A new subscription after an executor or transport configuration change opens fresh sessions rather than reusing stale connections. Sessions linger for five seconds after their final subscriber leaves.

History is neither persisted nor replayed by Last-Event-ID; reconnecting only receives whatever remains in a current shared session. See the Python monitoring API for typed history values and configuration overrides.

Diagnostic bundles

Submit diagnostics.collect through POST /v1/diagnostics/collect or its generic operation URL. It is a read-scoped job, with optional hosts, cluster, workload_id, log_tail, max_bytes, and dry_run. Clients cannot provide server output paths or privileged-probe credentials. Its result includes section/host status and an artifact descriptor (name, media_type, size_bytes, url).

workloads.run, benchmarks.run, and benchmarks.resume also accept collect_diagnostics: true. This captures the entire adapter, including readiness and hooks, while preserving the operation’s ordinary result/error schema. An artifact event announces the bundle; its URL is also derivable from the job ID:

GET /v1/jobs/{job_id}/artifacts/diagnostics

Download after the job finishes, including failed or cancelled jobs that produced a bundle. Access follows job ownership and operation permissions; retention follows job history. Lost workers may leave incomplete files: inspect bundle_complete, partial, and truncated. The reference CLI prints results and events; use an HTTP client to download the artifact. See Diagnostics API for capture scope, redaction, size limits, and plugin diagnostic sections.

Choosing an operation

Names use domain.action, with underscores inside multiword actions. Python library names are independent of the daemon operation vocabulary. Use canonical names such as workloads.run; retired flat daemon names have no aliases.

Surface	Meaning
/v1/jobs	Durable daemon invocation history, prompts, results and events.
workloads.list	Saved deployment metadata; does not probe current health.
workloads.inventory	Recorded provider lifecycle/ownership/action inventory, including unfinished activations and a completeness flag.
cluster.observe / cluster.report	Raw host occupancy and coverage / grouped, metadata-enriched display report.
workloads.plan / workloads.run	Resolve placement / launch and optionally await readiness and post-launch hooks.
workloads.ensure_ready	Adopt, recover or launch with durable effect receipts and exact revision identity.
workloads.stop	Exact-ID/revision teardown of an integration-owned workload.
workloads.stop_matching / workloads.stop_all	Explicit selector-based teardown / host-wide teardown; distinct from ownership-guarded automation.
workloads.materialize	Resolve container launch units without launching.
recipes.plan_build / recipes.build	Plan / prepare native or image environments and stage assets without starting inference.
recipes.validate / recipes.realize	Validate a recipe / export a placed, pinned realization.
catalog.search / catalog.recipes	Simple recipe summaries / paginated catalog browsing and facets.
catalog.registries.*	Named admin actions: add, remove, enable, disable, trust, untrust; no action-dispatch field.
benchmarks.run / benchmarks.resume	Execute a benchmark / resume by benchmark ID, separate from the daemon job ID.
discovery.snapshot	Scoped endpoint observations with original timestamp, host coverage and errors.
model_cache.list / model_cache.prune	Cache-local usage/protection inventory / preview or apply LRU cleanup.
model_cache.pin / model_cache.reconcile	Retain an immutable revision / explicitly release an inactive workload claim.
model_cache.adopt / model_cache.delete	Enroll existing downloads in retention tracking / explicitly remove a model or revision.

cluster.observe and cluster.report can omit hosts to use the selected cluster or configured defaults. workloads.stop_all requires explicit hosts or cluster; teardown never chooses a default target. Build operations accept offline, including during recipe loading.

discovery.snapshot accepts workload_ids and metadata_only. The latter skips host resolution and SSH, considers recorded jobs, and still HTTP-probes their endpoints; it cannot be combined with hosts or cluster. Returned endpoints include recipe_name. Metadata-only or partial coverage does not prove that an unobserved workload has stopped.

The Python CX7 planning/application, SSH diagnosis, and systemd-export helpers are not daemon operations. Use only operations advertised by /v1/ops.

Registry add requires name and URL and never grants trust. trust requires acknowledge_trust: true. Each registry action rejects irrelevant fields. Advisory snapshots use /v1/snapshots/discovery.snapshot, with the same coverage-bearing result as the corresponding operation. Plugin endpoint projections retain their own namespaces and routing-policy fields.

Workload inventory and revision changes

workloads.inventory includes unfinished activations after caller loss and returns at most 256 rows. Its complete field is false if execution is in flight or the inventory exceeds that limit. Treat missing rows as absence only when complete is true; even a complete inventory describes recorded state, not live health.

workloads.ensure_ready can adopt or wake an existing exact revision after its source recipe changes. Receipt and ownership checks still apply. Recovery and new launches resolve the recipe again and require the requested revision to match.

Proxy discovery status also returns warnings separately from last_error. Warnings describe routing issues following a successful import and do not fail the sweep or delay its next normal interval. Safe gateway rejection messages appear in last_error; unexpected exception details remain class-name-only.

Model-cache manifest imports

POST /v1/model_cache/import is an operate-scoped job. Supply a manifest for an import without origin access. Omitting it requires explicit fetch_manifest: true, for example:

{
  "model": "org/model",
  "revision": "COMMIT",
  "cache_dir": "/models/hf",
  "hosts": ["spark-a", "spark-b"],
  "fetch_manifest": true
}

Replace COMMIT with the revision you intend to check. The daemon resolves metadata once using server-side HF credentials, then imports the same immutable inventory on all targets without an interactive prompt. It may download a weight index, never model weights. offline: true requires a supplied manifest even with fetch consent. Invalid manifests and origin failures never fall back to inferred local file inventories.

To review metadata first, call model_cache.resolve and submit its result as the manifest in a later import. Omitting revision resolves the origin’s current default revision. Incomplete imports can save the manifest and complete the job successfully while returning incomplete validation reports; inspect their complete and status fields. Import does not repair weights or refresh retention usage. Tokens and Python confirmation callbacks are not request fields.

Model-cache retention

model_cache.list requires read, model_cache.pin requires operate, and model_cache.prune, model_cache.reconcile, model_cache.adopt, and model_cache.delete require admin. All six are jobs and return the usual 202 acknowledgement before their final reports are available.

Supply cache_dir as the targets’ HF home and hosts as the resolved host list. Unlike the CLI, these operations do not accept cluster or resolve a default cluster; omitting hosts targets localhost. Cache mutations verify shared access and operate once per detected cache location.

Prune defaults to dry_run: true and min_age_days: 7. Set dry_run: false to apply a reviewed policy. Without a byte target it selects every eligible revision older than the minimum age; max_bytes budgets private tracked allocations, including blobs used only by the selected snapshot, while min_free_bytes targets filesystem free space. Both are per-cache targets. Protection rules are preserved even when a target cannot be met.

Pinning defaults to pinned: true; use false to remove the pin. Pin and reconcile require a full immutable revision. Reconciliation additionally requires an exact claim and confirm_inactive: true, supplied only after inspecting every host covered by that claim and stopping all associated workloads and launches. It removes the protection record without deleting model files.

model_cache.adopt takes either model (optionally revision) or all_models: true. It previews by default; dry_run: false writes usage records without downloading or moving files. offline: true requires HF blob-linked entries instead of contacting the origin. last_used is now by default or mtime; pinned: true protects adopted revisions from pruning. Adoption does not create a launch manifest, and use by other tools does not refresh retention age.

model_cache.delete requires model; optional revision narrows the selection to a commit or ref. It also defaults to dry_run: true. Clients must obtain their own confirmation before submitting dry_run: false; there is no CLI-style confirmation prompt. Pins, age, and tracked status do not block deletion. force: true overrides workload claims, but never active downloads or publication locks. Apply preflights all targets and returns refusal reports if any is blocked; later failures can still leave partial progress. Saved manifests are retained.

Pass allowed_revisions as the array of immutable revisions from the confirmed preview when applying deletion. A revision that appeared afterward is refused. [] authorizes no revisions; omitting this field applies to the model/revision selection at execution time. This guard does not bypass claims, locks, or other deletion checks.

For example, preview adoption with an admin credential:

{
  "cache_dir": "/models/hf",
  "hosts": ["spark-a", "spark-b"],
  "model": "org/model",
  "dry_run": true,
  "pinned": true
}

Send this body to POST /v1/model_cache/adopt, then read the job’s final result.

Final results are arrays of cache reports with cache_root, entries, dry_run, planned_bytes, reclaimed_bytes, free_bytes, and target_met. Prune adds free_bytes_after. Entries describe tracking, usage, pins, claims, eligibility, protection reason_code / reason, and any action, including would_adopt, adopted, would_delete, and deleted. location carries key, kind, source, hosts, per-host paths, and verified; host is the execution representative. Count a shared location once, not once per participating host. A successful job can still report target_met: false; clients must inspect it. Cleanup can remove exclusive blobs, while blobs referenced elsewhere are kept. See Model Cache Management.

Core operations

benchmarks

benchmarks.resume — job / operate

Resume a saved benchmark by benchmark ID, separate from its daemon job ID.

POST /v1/benchmarks/resume

Reservation: auto. Operation version: 1.

Argument	Type	Required
benchmark_id	string	Yes
dry_run	boolean	No
integrations	object	No
export_files	boolean	No
output_file	string	No
timeout	integer	No
api_key_env	string	No
collect_diagnostics	boolean	No

benchmarks.run — job / operate

Launch or reuse inference and execute the selected benchmark framework.

POST /v1/benchmarks/run

Reservation: auto. Operation version: 1.

Argument	Type	Required
recipe	string	Yes
category	string	No
framework	string	No
profile	string	No
bench_args	object	No
hosts	string[]	No
cluster	string	No
overrides	object	No
resume	string	No
skip_run	boolean	No
no_stop	boolean	No
timeout	integer	No
api_key_env	string	No
integrations	object	No
output_file	string	No
export_files	boolean	No
solo	boolean	No
dry_run	boolean	No
scheduler	string	No
rootful	boolean	No
trust	boolean	No
offline	boolean	No
sync_tuning	boolean	No
extra_docker_opts	string[]	No
collect_diagnostics	boolean	No

catalog

catalog.capacity — job / read

Inspect a named cluster’s hardware, occupancy, and available capacity for recipe placement.

POST /v1/catalog/capacity

Reservation: auto. Operation version: 1.

Argument	Type	Required
cluster	string	Yes

catalog.cleanup — job / operate

Remove abandoned staged recipe imports while preserving retained imports.

POST /v1/catalog/cleanup

Reservation: auto. Operation version: 1.

No arguments. Send {}.

catalog.clusters — query / read

List saved cluster definitions for catalog selection without probing hosts.

POST /v1/catalog/clusters

Reservation: auto. Operation version: 1.

No arguments. Send {}.

catalog.import — job / operate

Stage supplied recipe content for preview and resolution, preserving its imported provenance.

POST /v1/catalog/import

Reservation: auto. Operation version: 1.

Argument	Type	Required
content	string	Yes

catalog.recipe — job / read

Inspect one recipe’s configuration, metadata, and validation details, optionally checking its expected revision.

POST /v1/catalog/recipe

Reservation: auto. Operation version: 1.

Argument	Type	Required
reference	string	Yes
overrides	object	No
expected_revision	string	No

catalog.recipes — query / read

Browse cached recipe summaries with filters, pagination, and facets; does not refresh registries.

POST /v1/catalog/recipes

Reservation: auto. Operation version: 1.

Argument	Type	Required
query	string	No
registry	string	No
runtime	string	No
local_only	boolean	No
offset	integer	No
limit	integer	No
filters	object	No

catalog.refresh — job / operate

Refresh enabled recipe registries from their remote sources.

POST /v1/catalog/refresh

Reservation: auto. Operation version: 1.

No arguments. Send {}.

catalog.registries — query / read

List configured recipe registries and their enabled, stemless, and trust settings.

POST /v1/catalog/registries

Reservation: auto. Operation version: 1.

No arguments. Send {}.

catalog.registries.add — job / admin

Add a named recipe registry from a URL; adding it does not grant trust.

POST /v1/catalog/registries/add

Reservation: auto. Operation version: 1.

Argument	Type	Required
name	string	Yes
url	string	Yes
subpath	string	No

catalog.registries.disable — job / admin

Disable a registry while retaining its saved configuration.

POST /v1/catalog/registries/disable

Reservation: auto. Operation version: 1.

Argument	Type	Required
name	string	Yes

catalog.registries.enable — job / admin

Enable a configured recipe registry for discovery and resolution.

POST /v1/catalog/registries/enable

Reservation: auto. Operation version: 1.

Argument	Type	Required
name	string	Yes

catalog.registries.remove — job / admin

Remove a registry from configuration so it no longer participates in recipe discovery.

POST /v1/catalog/registries/remove

Reservation: auto. Operation version: 1.

Argument	Type	Required
name	string	Yes

catalog.registries.trust — job / admin

Grant a registry standing trust for recipe execution after explicit acknowledgement.

POST /v1/catalog/registries/trust

Reservation: auto. Operation version: 1.

Argument	Type	Required
name	string	Yes
acknowledge_trust	boolean	Yes

catalog.registries.untrust — job / admin

Revoke a registry’s standing trust for recipe execution.

POST /v1/catalog/registries/untrust

Reservation: auto. Operation version: 1.

Argument	Type	Required
name	string	Yes

catalog.retain — job / operate

Keep a staged imported recipe so later cleanup does not remove it.

POST /v1/catalog/retain

Reservation: auto. Operation version: 1.

Argument	Type	Required
reference	string	Yes

catalog.search — query / read

Search recipe summaries with simple filters; catalog.recipes provides paginated browsing.

POST /v1/catalog/search

Reservation: auto. Operation version: 1.

Argument	Type	Required
query	string	No
registry	string	No
runtime	string	No
include_hidden	boolean	No
include_local	boolean	No
unique_names	boolean	No

cluster

cluster.create — job / admin

Save a new named cluster with its hosts, description, and optional SSH user.

POST /v1/cluster/create

Reservation: auto. Operation version: 1.

Argument	Type	Required
name	string	Yes
hosts	string[]	Yes
description	string	No
user	string	No

cluster.default — job / admin

Select the default saved cluster, or clear the selection with a null name.

POST /v1/cluster/default

Reservation: auto. Operation version: 1.

Argument	Type	Required
name	string/null	No

cluster.delete — job / admin

Delete a saved cluster definition without stopping its running workloads.

POST /v1/cluster/delete

Reservation: auto. Operation version: 1.

Argument	Type	Required
name	string	Yes

cluster.observe — job / read

Observe raw per-host occupancy and coverage errors.

POST /v1/cluster/observe

Reservation: auto. Operation version: 1.

Argument	Type	Required
hosts	string[]	No
cluster	string	No
executor	string	No

cluster.report — job / read

Observe occupancy and enrich grouped workload/idle-host results with saved metadata.

POST /v1/cluster/report

Reservation: auto. Operation version: 1.

Argument	Type	Required
hosts	string[]	No
cluster	string	No
executor	string	No

cluster.update — job / admin

Change a saved cluster’s hosts, description, or SSH user.

POST /v1/cluster/update

Reservation: auto. Operation version: 1.

Argument	Type	Required
name	string	Yes
hosts	string[]	No
description	string	No
user	string	No

diagnostics

diagnostics.collect — job / read

Collect a bounded diagnostic bundle for configuration, hosts, and optional workload logs.

POST /v1/diagnostics/collect

Reservation: auto. Operation version: 1.

Argument	Type	Required
hosts	string[]	No
cluster	string	No
workload_id	string	No
log_tail	integer	No
max_bytes	integer	No
dry_run	boolean	No

discovery

discovery.snapshot — job / read

Observe healthy workload endpoints with coverage and errors; metadata-only mode probes recorded endpoints without an SSH sweep.

POST /v1/discovery/snapshot

Reservation: auto. Operation version: 1.

Argument	Type	Required
hosts	string[]	No
cluster	string	No
workload_ids	string[]	No
metadata_only	boolean	No

model_cache

model_cache.adopt — job / admin

Preview or enroll existing untracked model snapshots in usage tracking without downloading or moving files.

POST /v1/model_cache/adopt

Reservation: auto. Operation version: 1.

Argument	Type	Required
model	string	No
revision	string	No
endpoint	string	No
cache_dir	string	Yes
hosts	string[]	No
all_models	boolean	No
dry_run	boolean	No
offline	boolean	No
last_used	string	No
pinned	boolean	No

model_cache.delete — job / admin

Preview or explicitly delete a model or revision, preserving blobs still used by other snapshots.

POST /v1/model_cache/delete

Reservation: auto. Operation version: 1.

Argument	Type	Required
model	string	Yes
revision	string	No
cache_dir	string	Yes
hosts	string[]	No
dry_run	boolean	No
force	boolean	No
allowed_revisions	string[]	No

model_cache.import — job / operate

Import a supplied manifest or explicitly resolve one from the model origin, then report target-cache completeness without downloading weights.

POST /v1/model_cache/import

Reservation: auto. Operation version: 1.

Argument	Type	Required
model	string	Yes
revision	string	No
endpoint	string	No
projector	string	No
file_selection	string	No
manifest	object	No
cache_dir	string	Yes
hosts	string[]	No
checksums	boolean	No
fetch_manifest	boolean	No
offline	boolean	No

model_cache.inspect — job / read

Check selected model files against their manifest and report missing, invalid, or unreadable cache content.

POST /v1/model_cache/inspect

Reservation: auto. Operation version: 1.

Argument	Type	Required
model	string	Yes
revision	string	No
endpoint	string	No
projector	string	No
file_selection	string	No
cache_dir	string	Yes
hosts	string[]	No
checksums	boolean	No

model_cache.list — job / read

Inventory tracked model-cache revisions, byte accounting, usage, pins, and active claims.

POST /v1/model_cache/list

Reservation: auto. Operation version: 1.

Argument	Type	Required
cache_dir	string	Yes
hosts	string[]	No

model_cache.pin — job / operate

Add or remove a retention pin for an exact model revision on the selected hosts.

POST /v1/model_cache/pin

Reservation: auto. Operation version: 1.

Argument	Type	Required
model	string	Yes
revision	string	Yes
endpoint	string	No
cache_dir	string	Yes
hosts	string[]	No
pinned	boolean	No

model_cache.prune — job / admin

Preview or apply age- and budget-based cleanup of eligible model-cache revisions, preserving protected content.

POST /v1/model_cache/prune

Reservation: auto. Operation version: 1.

Argument	Type	Required
cache_dir	string	Yes
hosts	string[]	No
dry_run	boolean	No
min_age_days	number	No
max_bytes	integer	No
min_free_bytes	integer	No

model_cache.reconcile — job / admin

Release an exact stale cache claim after the caller confirms its workloads are inactive; does not delete model files.

POST /v1/model_cache/reconcile

Reservation: auto. Operation version: 1.

Argument	Type	Required
model	string	Yes
revision	string	Yes
endpoint	string	No
cache_dir	string	Yes
hosts	string[]	No
claim	string	Yes
confirm_inactive	boolean	Yes

model_cache.repair — job / operate

Acquire or copy missing and invalid model files, then validate the repaired caches.

POST /v1/model_cache/repair

Reservation: auto. Operation version: 1.

Argument	Type	Required
model	string	Yes
revision	string	No
endpoint	string	No
projector	string	No
file_selection	string	No
cache_dir	string	Yes
hosts	string[]	No
local_cache_dir	string	No
transfer_mode	string	No
checksums	boolean	No
offline	boolean	No
dry_run	boolean	No

model_cache.resolve — job / read

Resolve a model and revision into an immutable manifest of the selected files.

POST /v1/model_cache/resolve

Reservation: auto. Operation version: 1.

Argument	Type	Required
model	string	Yes
revision	string	No
endpoint	string	No
projector	string	No
file_selection	string	No
cache_dir	string	No

proxy

proxy.add_alias — job / admin

Save an additional public model name pointing at a gateway target.

POST /v1/proxy/add_alias

Reservation: none. Operation version: 1.

Argument	Type	Required
alias	string	Yes
target	string	Yes

proxy.discovery.ensure — query / admin

Ensure the current gateway generation has a dedicated discovery service and return its daemon job ID.

POST /v1/proxy/discovery/ensure

Reservation: none. Operation version: 1.

Argument	Type	Required
generation	string	Yes

proxy.discovery.status — query / admin

Read the gateway discovery service’s ownership, scope, and health state.

POST /v1/proxy/discovery/status

Reservation: none. Operation version: 1.

No arguments. Send {}.

proxy.discovery.stop — query / admin

Stop the dedicated gateway discovery service without stopping inference workloads.

POST /v1/proxy/discovery/stop

Reservation: none. Operation version: 1.

No arguments. Send {}.

proxy.install_service — job / admin

Install the persistent proxy systemd service, optionally starting it immediately.

POST /v1/proxy/install_service

Reservation: none. Operation version: 1.

Argument	Type	Required
scope	string	No
cluster	string	No
now	boolean	No
dry_run	boolean	No

proxy.list_aliases — query / read

List saved gateway model aliases without launching a gateway.

POST /v1/proxy/list_aliases

Reservation: none. Operation version: 1.

No arguments. Send {}.

proxy.list_gateways — query / read

List registered gateway providers and their availability.

POST /v1/proxy/list_gateways

Reservation: none. Operation version: 1.

No arguments. Send {}.

proxy.models — job / read

Query the running gateway for its registered models.

POST /v1/proxy/models

Reservation: none. Operation version: 1.

No arguments. Send {}.

proxy.register_loaded_model — job / operate

Record a loaded recipe and its overrides for gateway discovery; does not launch inference.

POST /v1/proxy/register_loaded_model

Reservation: none. Operation version: 1.

Argument	Type	Required
recipe	string	Yes
overrides	object	No
cluster	string	No

proxy.remove_alias — job / admin

Remove a saved gateway model alias.

POST /v1/proxy/remove_alias

Reservation: none. Operation version: 1.

Argument	Type	Required
alias	string	Yes

proxy.resolve_gateway — query / read

Resolve an explicit or configured gateway provider and validate its availability.

POST /v1/proxy/resolve_gateway

Reservation: none. Operation version: 1.

Argument	Type	Required
name	string	No

proxy.service_status — job / admin

Inspect the proxy systemd unit and optionally retrieve recent journal output.

POST /v1/proxy/service_status

Reservation: none. Operation version: 1.

Argument	Type	Required
scope	string	No
journal_lines	integer	No

proxy.start — job / operate

Discover endpoints, prepare the selected gateway, and start its service and configured discovery backend.

POST /v1/proxy/start

Reservation: none. Operation version: 1.

Argument	Type	Required
host	string	No
port	integer	No
gateway	string	No
host_filter	string[]	No
cluster	string	No
clear_cluster	boolean	No
auto_discover	boolean	No
discovery_backend	string	No
discover_interval	integer	No
discover_removal_grace_sweeps	integer	No
restart	boolean	No
persist	boolean	No
dry_run	boolean	No

proxy.status — job / read

Inspect the running gateway and its discovery status.

POST /v1/proxy/status

Reservation: none. Operation version: 1.

No arguments. Send {}.

proxy.stop — job / operate

Stop the gateway and its discovery service while leaving inference workloads running.

POST /v1/proxy/stop

Reservation: none. Operation version: 1.

Argument	Type	Required
dry_run	boolean	No

proxy.sync — job / operate

Refresh the gateway’s model configuration from discovered endpoints, aliases, and optional host filters.

POST /v1/proxy/sync

Reservation: none. Operation version: 1.

Argument	Type	Required
aliases	object	No
host_filter	string[]	No

proxy.uninstall_service — job / admin

Remove the installed proxy systemd service, or preview its removal.

POST /v1/proxy/uninstall_service

Reservation: none. Operation version: 1.

Argument	Type	Required
scope	string	No
dry_run	boolean	No

proxy.unregister_loaded_model — job / operate

Remove a recipe from saved gateway load tracking; does not itself stop inference.

POST /v1/proxy/unregister_loaded_model

Reservation: none. Operation version: 1.

Argument	Type	Required
recipe	string	Yes

recipes

recipes.build — job / operate

Prepare images or native environments and stage recipe assets without starting inference.

POST /v1/recipes/build

Reservation: auto. Operation version: 1.

Argument	Type	Required
recipe	string	Yes
hosts	string[]	No
cluster	string	No
overrides	object	No
dry_run	boolean	No
trust	boolean	No
rebuild	boolean	No
offline	boolean	No
executor	string	No
executor_config	object	No
cache_dir	string	No
local_cache_dir	string	No
transfer_mode	string	No
transfer_interface	string	No
topology	string	No
sync_tuning	boolean	No
preserve_model_perms	boolean	No
skip_model_fan_out	boolean	No
builder_context	object	No

recipes.plan_build — job / read

Plan environment preparation and asset staging without reserving inference ranks.

POST /v1/recipes/plan_build

Reservation: auto. Operation version: 1.

Argument	Type	Required
recipe	string	Yes
hosts	string[]	No
cluster	string	No
overrides	object	No
dry_run	boolean	No
trust	boolean	No
rebuild	boolean	No
offline	boolean	No
executor	string	No
executor_config	object	No
cache_dir	string	No
local_cache_dir	string	No
transfer_mode	string	No
transfer_interface	string	No
topology	string	No
sync_tuning	boolean	No
preserve_model_perms	boolean	No
skip_model_fan_out	boolean	No
builder_context	object	No

recipes.realize — job / operate

Plan against the selected cluster and export a pinned, resolved recipe.

POST /v1/recipes/realize

Reservation: auto. Operation version: 1.

Argument	Type	Required
recipe	string	Yes
hosts	string[]	No
cluster	string	No
overrides	object	No
strategy_options	object	No
solo	boolean	No
dry_run	boolean	No
trust	boolean	No
ensure	boolean	No
scheduler	string	No
transfer_mode	string	No
cache_dir	string	No
offline	boolean	No
runtime_cache	boolean	No
auto_port	boolean	No
ray_port	integer	No
dashboard_port	integer	No
dashboard	boolean	No
init_port	integer	No
executor	string	No
executor_config	object	No
rootful	boolean	No
workload_id_override	string	No
transfer_interface	string	No
local_cache_dir	string	No
sync_tuning	boolean	No
extra_docker_opts	string[]	No
topology	string	No
preserve_model_perms	boolean	No
skip_model_fan_out	boolean	No
recipe_ref	string	No

recipes.validate — job / read

Validate a recipe and its overrides, returning structured issues without launching it.

POST /v1/recipes/validate

Reservation: auto. Operation version: 1.

Argument	Type	Required
recipe	string	Yes
overrides	object	No

setup

setup.ensure_local_key — job / admin

Find a usable local SSH key or generate one when requested.

POST /v1/setup/ensure_local_key

Reservation: auto. Operation version: 1.

Argument	Type	Required
preferred	string	No
generate	boolean	No

setup.mesh_ssh_keys — job / admin

Distribute public keys to establish SSH access among the selected hosts, or preview the changes.

POST /v1/setup/mesh_ssh_keys

Reservation: auto. Operation version: 1.

Argument	Type	Required
hosts	string[]	Yes
user	string	No
key	string	No
timeout	integer	No
dry_run	boolean	No

setup.plan — job / read

Probe hosts and produce a hardware-aware setup plan without applying it.

POST /v1/setup/plan

Reservation: auto. Operation version: 1.

Argument	Type	Required
hosts	string[]	No
cluster	string	No

setup.probe_hosts — job / read

Collect host reachability, hardware, and setup check results without applying fixes.

POST /v1/setup/probe_hosts

Reservation: auto. Operation version: 1.

Argument	Type	Required
hosts	string[]	No
cluster	string	No

setup.probe_ssh_access — job / read

Check whether the selected hosts accept SSH using the requested user and key.

POST /v1/setup/probe_ssh_access

Reservation: auto. Operation version: 1.

Argument	Type	Required
hosts	string[]	Yes
user	string	No
key	string	No
timeout	integer	No
max_workers	integer	No

setup.rdma_test — job / operate

Run the selected RDMA diagnostic suite and return per-link or collective performance results.

POST /v1/setup/rdma_test

Reservation: auto. Operation version: 1.

Argument	Type	Required
hosts	string[]	No
cluster	string	No
suite	string	No
parallel_pairs	integer	No
full	boolean	No
image	string	No
duration	integer	No
queue_pairs	integer	No
msg_size	string	No
nccl_binary	string	No
gid_index	integer	No
link_type	string	No
force_container	boolean	No
keep_containers	boolean	No
dry_run	boolean	No

setup.run_steps — job / admin

Apply selected setup steps with progress, credential prompts, and recovery manifests, or preview them.

POST /v1/setup/run_steps

Reservation: auto. Operation version: 1.

Argument	Type	Required
hosts	string[]	No
cluster	string	No
dry_run	boolean	No
only_steps	string[]	No

setup.undo — job / admin

Undo eligible setup steps recorded in a named cluster’s recovery manifest.

POST /v1/setup/undo

Reservation: auto. Operation version: 1.

Argument	Type	Required
cluster	string	Yes
dry_run	boolean	No
only_steps	string[]	No

tailscale

tailscale.down — job / admin

Disconnect selected hosts from Tailscale, optionally removing the installation.

POST /v1/tailscale/down

Reservation: auto. Operation version: 1.

Argument	Type	Required
hosts	string[]	No
cluster	string	No
remove	boolean	No
dry_run	boolean	No
ask_sudo	boolean	No

tailscale.expose — job / admin

Configure Tailscale exposure for a cluster endpoint or gateway.

POST /v1/tailscale/expose

Reservation: auto. Operation version: 1.

Argument	Type	Required
cluster	string	No
proxy	boolean	No
head_host	string	No
port	integer	No
set_proxy_host	boolean	No
ask_sudo	boolean	No

tailscale.join — job / admin

Install Tailscale on selected hosts and initiate joining the tailnet with the requested settings.

POST /v1/tailscale/join

Reservation: auto. Operation version: 1.

Argument	Type	Required
hosts	string[]	No
cluster	string	No
tag	string	No
ephemeral	boolean	No
enable_ssh	boolean	No
hostname	string	No
dry_run	boolean	No
ask_sudo	boolean	No

tailscale.status — job / read

Inspect Tailscale installation, connection, and address information on selected hosts.

POST /v1/tailscale/status

Reservation: auto. Operation version: 1.

Argument	Type	Required
hosts	string[]	No
cluster	string	No

workloads

workloads.endpoints — job / read

Observe credential-free workload endpoints for the requested binding and report live coverage or metadata-only evidence.

POST /v1/workloads/endpoints

Reservation: none. Operation version: 1.

Argument	Type	Required
recipe_revision	string	No
cluster_candidates	string[]	No
workload_id	string	No

workloads.ensure_ready — job / operate

Adopt, recover, or launch an exact recipe revision and return a verified ready endpoint with ownership and effect receipts.

POST /v1/workloads/ensure_ready

Reservation: controller. Operation version: 1.

Argument	Type	Required
recipe	string	Yes
recipe_revision	string	No
cluster_candidates	string[]	No
overrides	object	No
timeout	number	No

workloads.find_intent — job / read

Find an observed running deployment matching a launch intent.

POST /v1/workloads/find_intent

Reservation: auto. Operation version: 1.

Argument	Type	Required
intent_id	string	Yes
hosts	string[]	Yes
cluster	string	No

workloads.inventory — job / read

List recorded workloads and unfinished activations with ownership, lifecycle actions, and a completeness flag.

POST /v1/workloads/inventory

Reservation: none. Operation version: 1.

No arguments. Send {}.

workloads.list — query / read

List saved workload metadata without probing live health.

POST /v1/workloads/list

Reservation: auto. Operation version: 1.

Argument	Type	Required
limit	integer	No

workloads.materialize — job / operate

Resolve container launch units from a plan without starting them.

POST /v1/workloads/materialize

Reservation: auto. Operation version: 1.

Argument	Type	Required
recipe	string	Yes
hosts	string[]	No
cluster	string	No
overrides	object	No
strategy_options	object	No
solo	boolean	No
dry_run	boolean	No
trust	boolean	No
ensure	boolean	No
scheduler	string	No
transfer_mode	string	No
cache_dir	string	No
offline	boolean	No
runtime_cache	boolean	No
auto_port	boolean	No
ray_port	integer	No
dashboard_port	integer	No
dashboard	boolean	No
init_port	integer	No
executor	string	No
executor_config	object	No
rootful	boolean	No
workload_id_override	string	No
transfer_interface	string	No
local_cache_dir	string	No
sync_tuning	boolean	No
extra_docker_opts	string[]	No
topology	string	No
preserve_model_perms	boolean	No
skip_model_fan_out	boolean	No
recipe_ref	string	No

workloads.plan — job / read

Resolve placement and launch settings without launching a workload.

POST /v1/workloads/plan

Reservation: auto. Operation version: 1.

Argument	Type	Required
recipe	string	Yes
hosts	string[]	No
cluster	string	No
overrides	object	No
strategy_options	object	No
solo	boolean	No
dry_run	boolean	No
trust	boolean	No
ensure	boolean	No
scheduler	string	No
transfer_mode	string	No
cache_dir	string	No
offline	boolean	No
runtime_cache	boolean	No
auto_port	boolean	No
ray_port	integer	No
dashboard_port	integer	No
dashboard	boolean	No
init_port	integer	No
executor	string	No
executor_config	object	No
rootful	boolean	No
workload_id_override	string	No
transfer_interface	string	No
local_cache_dir	string	No
sync_tuning	boolean	No
extra_docker_opts	string[]	No
topology	string	No
preserve_model_perms	boolean	No
skip_model_fan_out	boolean	No
recipe_ref	string	No

workloads.plugins — job / read

List plugin inventory and identify enabled, loaded plugins that provide workload lifecycle controls.

POST /v1/workloads/plugins

Reservation: none. Operation version: 1.

No arguments. Send {}.

workloads.reconcile — job / admin

Reconcile uncertain effects for an exact workload after verifying its revision, ownership, and placement evidence.

POST /v1/workloads/reconcile

Reservation: controller. Operation version: 1.

Argument	Type	Required
recipe	string	Yes
recipe_revision	string	Yes
cluster_candidates	string[]	No
overrides	object	No
workload_id	string	Yes
acknowledge_effects	boolean	Yes

workloads.run — job / operate

Plan and launch a workload, with optional readiness and post-launch hooks.

POST /v1/workloads/run

Reservation: auto. Operation version: 1.

Argument	Type	Required
recipe	string	Yes
hosts	string[]	No
cluster	string	No
overrides	object	No
strategy_options	object	No
solo	boolean	No
dry_run	boolean	No
trust	boolean	No
ensure	boolean	No
scheduler	string	No
transfer_mode	string	No
cache_dir	string	No
offline	boolean	No
runtime_cache	boolean	No
auto_port	boolean	No
ray_port	integer	No
dashboard_port	integer	No
dashboard	boolean	No
init_port	integer	No
executor	string	No
executor_config	object	No
rootful	boolean	No
workload_id_override	string	No
transfer_interface	string	No
local_cache_dir	string	No
sync_tuning	boolean	No
extra_docker_opts	string[]	No
topology	string	No
preserve_model_perms	boolean	No
skip_model_fan_out	boolean	No
recipe_ref	string	No
collect_diagnostics	boolean	No
confirm	boolean	No
wait_ready	boolean	No
post_launch	boolean	No

workloads.sleep — job / operate

Ask the owning lifecycle provider to put an exact workload to sleep when that action is supported.

POST /v1/workloads/sleep

Reservation: controller. Operation version: 1.

Argument	Type	Required
recipe	string	Yes
recipe_revision	string	Yes
cluster_candidates	string[]	No
overrides	object	No
workload_id	string	Yes

workloads.status — job / read

Ask the lifecycle provider for the current state of an exact workload and record the observation.

POST /v1/workloads/status

Reservation: none. Operation version: 1.

Argument	Type	Required
recipe	string	Yes
recipe_revision	string	Yes
cluster_candidates	string[]	No
overrides	object	No
workload_id	string	Yes

workloads.stop — job / operate

Stop an exact workload and recipe revision with manager ownership checks and durable effect tracking.

POST /v1/workloads/stop

Reservation: controller. Operation version: 1.

Argument	Type	Required
recipe	string	Yes
recipe_revision	string	Yes
cluster_candidates	string[]	No
overrides	object	No
workload_id	string	Yes

workloads.stop_all — job / operate

Discover and stop all workloads on the given hosts or cluster (one is required; a teardown never defaults its target).

POST /v1/workloads/stop_all

Reservation: auto. Operation version: 1.

Argument	Type	Required
hosts	string[]	No
cluster	string	No
dry_run	boolean	No

Provide either hosts or cluster; this operation does not fall back to the default cluster.

workloads.stop_matching — job / operate

Stop workloads selected by ID, recipe or hosts; no integration-owner guard.

POST /v1/workloads/stop_matching

Reservation: auto. Operation version: 1.

Argument	Type	Required
workload_id	string	No
recipe	string	No
hosts	string[]	No
cluster	string	No
overrides	object	No

workloads.wake — job / operate

Ask the owning lifecycle provider to wake an exact workload when that action is supported.

POST /v1/workloads/wake

Reservation: controller. Operation version: 1.

Argument	Type	Required
recipe	string	Yes
recipe_revision	string	Yes
cluster_candidates	string[]	No
overrides	object	No
workload_id	string	Yes

Bundled Kubernetes plugin (experimental)

These Kubernetes operations are experimental. Their schemas and behavior may change; pin compatible versions when building clients.

Requires the loaded k8s plugin and the requested capability flags.

operations

plugins.k8s.cluster_info — job / operate

Inspect the selected Kubernetes cluster and optionally pin its kubectl version to the server release.

POST /v1/plugins/k8s/ops/cluster_info

Reservation: auto. Operation version: 1.

Argument	Type	Required
kubeconfig	string	No
context	string	No
namespace	string	No
pin	boolean	No

plugins.k8s.configure_service_account — job / admin

Prepare a Kubernetes service account and access configuration, optionally creating its namespace and writing kubeconfig.

POST /v1/plugins/k8s/ops/configure_service_account

Reservation: auto. Operation version: 1.

Argument	Type	Required
name	string	No
namespace	string	No
kubeconfig	string	No
context	string	No
create_namespace	boolean	No
token_duration	string	No
write_kubeconfig	boolean	No
dry_run	boolean	No

plugins.k8s.ensure_kubectl — job / admin

Find and validate kubectl, optionally downloading the requested version.

POST /v1/plugins/k8s/ops/ensure_kubectl

Reservation: auto. Operation version: 1.

Argument	Type	Required
version	string	No
context	string	No
download	boolean	No

plugins.k8s.jobset_status — job / operate

Read a named JobSet’s identity, namespace, and status without exposing its full pod specification.

POST /v1/plugins/k8s/ops/jobset_status

Reservation: auto. Operation version: 1.

Argument	Type	Required
name	string	Yes
namespace	string	No
kubeconfig	string	No
context	string	No

plugins.k8s.kueue_status — job / operate

Check whether Kueue and JobSet custom resource definitions are installed on the selected cluster.

POST /v1/plugins/k8s/ops/kueue_status

Reservation: auto. Operation version: 1.

Argument	Type	Required
kubeconfig	string	No
context	string	No

plugins.k8s.list_nodes — job / operate

List Kubernetes nodes, optionally filtering by label selector or GPU availability.

POST /v1/plugins/k8s/ops/list_nodes

Reservation: auto. Operation version: 1.

Argument	Type	Required
kubeconfig	string	No
context	string	No
selector	string	No
gpu_only	boolean	No

plugins.k8s.setup_kueue — job / admin

Check or install Kueue and JobSet components, with optional version selection and dry-run preview.

POST /v1/plugins/k8s/ops/setup_kueue

Reservation: auto. Operation version: 1.

Argument	Type	Required
install	boolean	No
kueue_version	string	No
jobset_version	string	No
namespace	string	No
kubeconfig	string	No
context	string	No
dry_run	boolean	No

plugins.k8s.stop_jobset — job / operate

Delete a named Kubernetes JobSet and its managed workload resources.

POST /v1/plugins/k8s/ops/stop_jobset

Reservation: auto. Operation version: 1.

Argument	Type	Required
name	string	Yes
namespace	string	No
kubeconfig	string	No
context	string	No

Results and errors

Family	Final result / meaning
Catalog	Bounded recipe listings/facets, registry or cluster arrays, recipe details with stable source_kind, normalized overrides, revision, plugin items and issues; refresh/capacity/import mutations have their own schemas.
Planning/execution	Explicit Plan/Run/Build/Benchmark DTOs. Workload workload_id is distinct from the daemon job_id. A serialized plan is a preview, not a resumable execution handle.
Generic workloads	ensure_ready: ready state, exact workload ID, credential-free endpoint, original observation timestamp, adoption flag and shared workload row. Guarded stop: offline state/IDs; selector/all teardown retain their separate Stop/StopAll DTOs. Inventory: recorded state/providers/actions and complete. Control: the same workload row after provider observation/action. Reconcile: ID/state/receipt count.
Discovery	discovery.snapshot and workloads.endpoints: sweep ID/time, live/metadata mode, hosts/coverage/errors/completeness, safe endpoints. A partial or metadata-only sweep is not authoritative absence.
Proxy	State/models/start/stop/sync/alias/service DTOs; status includes discovery owner/backend/job/scope/health. proxy.discovery.ensure returns an existing/new watch job ID, not that watch’s result.
Setup/network/cache/Kubernetes	Operation-specific result schemas; some are maps/arrays or booleans, not a universal envelope. Check /v1/ops rather than guessing from operation name.

Query results and successful job results do not have the same HTTP envelope. Transport errors use detail and usually a typed error record; job failures are stored in the job’s error. Terminal job states are succeeded, failed, cancelled, lost; nonterminal states are queued, starting, running, waiting_input, cancelling. Lost/uncertain effects require reconciliation; retryable does not mean that replaying a mutation with a fresh key is safe.

Common HTTP failures include:

Status	Meaning
401	Missing, expired, or invalid authentication
403	Insufficient permission, disabled feature, or rejected trust
404	Unknown operation, resource, or snapshot selector
408 / 413	Request-body timeout / configured body limit exceeded
409	Conflicting retry key, resource/capacity conflict, or admission/maintenance conflict
410	Requested event history is no longer retained
422	Invalid JSON object, unknown field, or schema/type violation
502 / 504	Hosts unreachable / operation timeout
500	Unexpected failure; inspect private daemon diagnostics

Transport errors include detail and usually an error object with type, retryable, and attrs. Authentication and permission errors may contain only detail; clients must not assume a uniform error envelope. A job accepted with 202 can later fail, with that failure recorded in the job’s error.

Extending the API

Plugins register framework-free DaemonOp declarations through sparkrun.plugins. The loader binds ownership before importing the module, covering both top-level registration and register(v). Declare a local name, request/result schemas, execution kind, permission, feature gate, and reservation policy. Handlers receive decoded arguments, an operation-local context, and a prompt callback.

For a plugin whose loader-assigned ID is example:

from sparkrun.plugins import DaemonOp, register_plugin_daemon_op

SPARKRUN_PLUGIN_API_VERSION = 1


def load_status():
    from .handlers import status
    return status


def register(v):
    register_plugin_daemon_op(DaemonOp(
        name="status",
        handler_loader=load_status,
        kind="query",
        scope="read",
        request_schema={"type": "object", "properties": {}, "additionalProperties": False},
        result_schema={
            "type": "object",
            "properties": {"ready": {"type": "boolean"}},
            "required": ["ready"],
            "additionalProperties": False,
        },
        description="Report readiness without changing state.",
    ))

This registers plugins.example.status, with alias POST /v1/plugins/example/ops/status. The lazy .handlers.status function must provide the declared result. Keep FastAPI, Pydantic, and uvicorn imports out of scanned registration modules. See Plugin Development for loading and packaging.

Local names use lowercase identifier segments separated by dots: workloads.activate becomes plugins.<plugin-id>.workloads.activate and /v1/plugins/<plugin-id>/ops/workloads/activate. Two plugins can declare the same local name. plugins is reserved as a local root. Streams use dedicated endpoints and report http_path: null.

The stable ID comes from the managed manifest, installed entry point, or bundled/directory discovery identity. ASCII case and runs of _, ., and - normalize to lowercase hyphens: oci_relay becomes oci-relay. IDs start with a letter, end with a letter/digit, and accept at most 100 input characters. Paths, traversal-like names, core, and plugins are rejected. Versions/generations are excluded. Duplicate selected IDs fail before import; existing store paths and configuration keys retain their names. sparkrun plugins list --json exposes plugin_id.

An explicit declaration owner must match the loader’s identity. Registration outside a loader context fails. Embedded applications can use the public application_daemon_registration() context, which assigns app-<application-profile-id>. Conflicting declarations are rejected and failed imports/hooks roll back registrations. These rules do not sandbox trusted plugin code.

Schema vocabulary

Plugin request and result schemas are validated at registration. Supported validation keywords are type (including a list of types), properties, required, boolean/schema additionalProperties, schema items, anyOf, enum, minimum, maximum, minLength, maxLength, minItems, maxItems, and Python-regex pattern. Sibling constraints still apply alongside anyOf. Annotations title, description, default, examples, deprecated, readOnly, writeOnly, and $comment are accepted without changing validation. Unsupported keywords such as $ref, const, oneOf, allOf, and format are rejected; use inline supported schemas and a one-value enum for constants.

Plugin JSON adapters

Adapters decode a bounded JSON envelope into a declared operation and arguments. A plugin identified as example can register relative path compat/status:

from sparkrun.plugins import DaemonRoute, register_plugin_daemon_route


def load_decoder():
    return lambda envelope: ("status", {})


def register_adapter():  # Call from register(v), after declaring status.
    register_plugin_daemon_route(DaemonRoute(
        path="compat/status",
        decode_loader=load_decoder,
        operations=("status",),
        request_schema={"type": "object", "properties": {}, "additionalProperties": False},
    ))

Its URL is POST /v1/plugins/example/routes/compat/status. Local target names resolve within the plugin. Use core:cluster.create for an explicit core target or plugins.other.status for another plugin. The selected target still enforces its schema, permissions, feature gate, secret/trust policy, and job behavior. The decoder must return one of its declared target spellings.

Paths must be literal and relative: no traversal, parameters, wildcards, encoded separators, queries, or fragments. Adapters cannot mount arbitrary ASGI handlers. Targets must exist at startup and path collisions fail startup. The separate routes/ branch cannot shadow generated ops/ aliases. Changing declarations requires a daemon restart. The bundled operations documented here currently register no custom adapters.

Plugin feedback

LiL, OCI Relay, and Sparkarena contribute effects through enclosing operations without independent daemon endpoints.

Structured Feedback appears in job SSE as versioned type: "feedback" events, with a top-level message and activity, stage, host, counter, and metric fields. Operation-scoped legacy type: "log" messages also carry message. These events describe operation progress, such as image transfer or restore; container output uses the log stream.

The source daemon API reference contains the corresponding operation inventory. The running daemon’s discovery and OpenAPI schemas remain authoritative for its installed versions and plugins.