Daemon Mode

The daemon exposes sparkrun.api through authenticated JSON and SSE. It keeps catalog queries and monitor sessions warm and runs each mutation or long operation in a fresh, independently supervised process. One job retains its context and live plan through approval, execution, readiness, and post-launch hooks.

The local daemon supports HTTP/TCP on Linux, macOS, and Windows. This is a single-user controller, bound to one OS account, application profile, and canonical config file. It controls Linux/Spark targets over SSH; local Linux setup and GPU workloads retain their existing platform requirements. Gateway integrations use shared workload lifecycle and discovery APIs. See Daemon API for HTTP routes, operation arguments, result contracts, and plugin registration.

Starting and connecting

The daemon feature is enabled by default on alpha, disabled on stable/beta. To opt in explicitly:

sparkrun setup features enable daemon
sparkrun daemon start
sparkrun daemon status
sparkrun daemon call catalog.clusters
sparkrun daemon stop

Use sparkrun daemon --config /absolute/config.yaml ... for another binding. serve runs in the foreground; restart reconnects existing workers. stop stops the API only. stop --cancel-jobs first drains admission and waits for cancellation; if it times out, the API remains available for recovery.

On macOS and Windows, the default listener is authenticated loopback TCP on an OS-assigned port. Linux retains its private Unix socket by default; use --no-unix for the same TCP-only mode. Same-UID Unix peers without a bearer token have local admin and credential-prompt permissions. An explicit bearer token uses its own permissions and workload manager on Unix sockets too. CLI discovery reads the binding’s protected local connection file when no Unix listener is present. daemon status and recovery commands do not load external plugins.

TCP is bearer-authenticated, including loopback:

sparkrun daemon serve --no-unix --tcp 127.0.0.1:0
# Optional discovery file for another local client:
sparkrun daemon serve --no-unix --tcp 127.0.0.1:0 --conn-file /private/path/daemon.json
sparkrun daemon token create automation --scope operate --expires-in 3600
sparkrun daemon token list
sparkrun daemon token revoke TOKEN_ID

TCP serving automatically writes a private connection file for local CLI discovery. --conn-file additionally publishes a file for clients and refuses to overwrite an existing one. Files contain URL, token, protocol, config binding, and process identity. Credentials use mode 0600 on POSIX and protected owner-only DACLs on Windows; Windows storage must support ACLs, and reparse points are rejected. A credential is valid only while its API process lives; normal shutdown removes it. After a crash, desktop clients use core’s native process identity checks to verify stale state before reclaiming it. Local tokens rotate on restart while keeping the same account principal, preserving job ownership and idempotency. Manually created tokens survive API restart until revoked/expired; only hashes are stored server-side.

Remote TCP requires a certificate and matching key by default:

sparkrun daemon serve --tcp 0.0.0.0:8443 \
  --tls-cert /private/server.crt --tls-key /private/server.key

To explicitly allow remote TCP without TLS, pass --allow-insecure:

sparkrun daemon serve --no-unix --tcp 0.0.0.0:8080 --allow-insecure
# IPv6 wildcard bind:
sparkrun daemon serve --no-unix --tcp '[::]:8080' --allow-insecure

Bearer authentication is still required; tokens and request/response content are transmitted unencrypted. The flag permits plaintext but does not disable TLS when both certificate and key are provided, or accept an incomplete TLS configuration. daemon restart preserves the explicit opt-in with the other listener settings. There is no config/environment setting that implicitly permits remote plaintext. When connecting remotely to a wildcard bind, use the host’s reachable address.

A Linux Unix listener remains plaintext and peer-authenticated. TLS and remote binds also keep a separate bearer-authenticated loopback HTTP listener for local CLI recovery, independent of the remote address and certificate names/trust stores. All listeners share one application lifecycle. Remote plaintext binds without --allow-insecure are refused. Proxy headers do not affect transport or auth policy.

Explicit client endpoints

Applications can use the HTTP/SSE API or sparkrun.daemon.client.Client directly. The CLI is also a reference client for debugging those requests. Put shared connection options before call, status, or jobs:

sparkrun daemon --url https://controller:8443 --token-file /private/daemon.token status
sparkrun daemon --url https://controller:8443 --token-file /private/daemon.token \
  call catalog.clusters
sparkrun daemon --url https://controller:8443 --token-file /private/daemon.token \
  jobs events JOB_ID --after 12
sparkrun daemon --socket /private/path/api.sock jobs list

--url takes an HTTP(S) origin, including an optional port; URL credentials, path prefixes, queries, and fragments are rejected. --socket selects a local Unix socket, not a socket on another machine. Windows clients use TCP. Endpoint options are mutually exclusive with each other and with --config. Without an explicit endpoint, existing local discovery applies.

Supply either --token TOKEN or --token-file PATH. The file contains just the bearer token, with an optional trailing newline, not a connection JSON document. If neither is provided, --url uses SPARKRUN_DAEMON_TOKEN when set. Explicit credentials take precedence over that environment variable. The variable does not affect local discovery or Unix connections. Token files avoid putting a credential in command arguments or shell history. Create a token on the daemon host with daemon token create, then provision it to the client.

HTTPS uses certificate and hostname verification. Use --ca-cert /path/ca.pem with --url https://... for a private CA or trusted self-signed certificate; there is no option to disable verification. Explicit http://... addresses use plaintext; a non-loopback server requires serve --allow-insecure as described above. The Python reference client supports the same CA selection through Client(url=..., token=..., ca_cert=...).

Explicit clients check the server protocol without loading local plugins, requiring the local daemon feature, comparing local config bindings, or checking the server’s PID on the client host. Unreachable endpoints, authentication failures, and protocol mismatches exit with an error; they never fall back to a local daemon or resubmit an operation. call --wait and all jobs subcommands retain their normal prompt, event replay, and cancellation behavior.

serve, start, stop, restart, token, systemd, and resources remain local commands and reject client connection options.

Operations, jobs, and prompts

Authenticated GET /v1/ops publishes operation names, permissions, execution classes, declaration ownership (owner, local_name), HTTP alias paths (http_path), and closed request/result schemas. /openapi.json and /docs describe the wire API. /v1/health reveals only health/protocol; /v1/info requires auth.

sparkrun daemon call cluster.create \
  --data '{"name":"lab","hosts":["spark-a","spark-b"]}' --wait
sparkrun daemon call workloads.run \
  --data '{"recipe":"registry/recipe","cluster":"lab","confirm":true}' \
  --idempotency-key launch-lab-001 --wait
sparkrun daemon jobs list
sparkrun daemon jobs show JOB_ID
sparkrun daemon jobs events JOB_ID --after 12
sparkrun daemon jobs answer JOB_ID
sparkrun daemon jobs cancel JOB_ID

POST /v1/ops/{name} returns a result directly for bounded inline queries, or HTTP 202 with job_id and events_url for jobs. Use Idempotency-Key for retries of the same request; different arguments with the same principal/key return 409. Inspect the existing job after an uncertain response. Clients must not fall back to local execution or submit a new mutation merely because a connection failed.

A run preview is display-only. An approved run executes the same resolved Python plan, not a reconstruction of its JSON preview. Jobs requiring input enter waiting_input; answers go to the authenticated live worker. Approval/decision answers are booleans; secret answers are strings. Prompts time out after 300 seconds by default and fail closed. Secret values stay in worker memory and are redacted from subsequent wire events/results; prompt metadata and answered state are durable. Do not put credentials in request arguments, recipe overrides, or freeform log messages. Named secret fields are rejected/redacted; arbitrary text is not a general-purpose secret detector.

Permissions are hierarchical (read, operate, admin) with independent --secret permission for credential prompts. Non-admin tokens access only their own jobs and need the operation’s scope to inspect or answer them. Token revocation also ends existing streams. TCP recipe-hook trust requires server opt-in via serve --allow-request-trust (or daemon.allow_request_trust) and a request with explicit trust: true; same-UID Unix callers can make that decision locally.

Terminal states are succeeded, failed, cancelled, and lost. Cancellation means execution ended, not that remote effects were rolled back. A worker lost after authorization is never rerun automatically. After reconciling remote effects:

sparkrun daemon resources list
sparkrun daemon resources reconcile OWNER_ID --acknowledge-effects

Core resource reservations also cover standalone mutating API/CLI operations. They serialize shared controller/documents and overlapping host sets. Unresolved destinations use a conservative controller-wide reservation. Host keys normalize user prefixes and case; different SSH aliases for the same machine are not yet resolved through DNS/SSH config, so use consistent destination names. Dead-owner reservations remain until explicit reconciliation.

HTTP aliases

Core queries and jobs have convenient POST /v1/... URLs, with dots in the operation name replaced by slashes. For example, create a cluster with POST /v1/cluster/create and the same JSON body you would send to POST /v1/ops/cluster.create:

HTTP alias	Operation	Response
POST /v1/cluster/create	cluster.create	202, job acknowledgement
POST /v1/cluster/update	cluster.update	202, job acknowledgement
POST /v1/cluster/delete	cluster.delete	202, job acknowledgement
POST /v1/catalog/clusters	catalog.clusters	200, cluster listing
POST /v1/workloads/run	workloads.run	202, job acknowledgement
POST /v1/plugins/k8s/ops/list_nodes	plugins.k8s.list_nodes	202, job acknowledgement

Aliases accept a JSON object, including {} for queries without arguments. Query means inline execution and can include administrative mutations; consult the operation’s scope and description. Only explicitly cacheable queries use the short query cache. Queries use POST too; GET /v1/catalog/clusters is not a query alias. The HTTP workflow below shows cluster creation using the reference client. The generic /v1/ops/{name} endpoint and sparkrun daemon call remain available.

Both URLs use the same dispatcher on TCP and Unix sockets, including authentication, permissions, feature gates, validation, and job handling. Retrying a job through either URL with the same principal, arguments, and Idempotency-Key returns the same job. Job inspection, cancellation, prompts, and event streams use the existing /v1/jobs/... endpoints; log tails and monitoring keep their dedicated stream endpoints.

Use each operation’s http_path from GET /v1/ops when building a client. Discovery filters operations by the caller’s permissions and enabled features. OpenAPI describes all registered operations, with stable operation IDs, request schemas, and permission metadata; those permissions and feature gates still apply to calls. For jobs, the HTTP 202 response schema describes the acknowledgement. x-sparkrun-operation.result references the final result schema, fetched through GET /v1/jobs/{id} after completion. Queries return their result directly. The Swagger UI at /docs includes an Authorize control for bearer tokens.

Plugin namespaces

Plugins declare local operation names under a stable, loader-assigned identity. For example, the Kubernetes integration exposes plugins.k8s.list_nodes through POST /v1/plugins/k8s/ops/list_nodes:

sparkrun daemon call plugins.k8s.list_nodes --data '{"context":"lab"}' --wait

The plugin must be loaded and its feature gates enabled. Core names and URLs remain in their own domains, such as cluster.create and /v1/cluster/create. Plugin versions and installation generations do not become part of the URL.

Custom JSON adapters use /v1/plugins/<plugin-id>/routes/<relative-path> and select declared operations through the same dispatcher. Namespaces preserve permission checks, feature gates, job ownership, and retry behavior. They are contribution ownership rules for trusted plugins, not code isolation. Changing plugins or their declarations requires a daemon restart.

Use the canonical name and http_path returned by discovery. The Daemon API extension guide covers registration, schemas, identity rules, and adapters.

Workload ownership and recovery

Core workloads.* operations manage deployment lifecycle for automation clients, including SparkRoute. Use workloads.run to launch, workloads.ensure_ready to adopt, recover, or launch an exact recipe revision, and workloads.inventory to inspect recorded ownership, provider state, and permitted actions.

Inventory includes unfinished activation records after a caller is lost. Its complete flag is false during in-flight execution or when the 256-row response limit omits records; only a complete response can support absence decisions. An exact recorded workload can be adopted or woken even after its source recipe changes. Recovery or a new launch still requires the requested recipe revision.

The credential grants a stable workload manager. Ordinary calls use daemon; SparkRoute uses sparkroute. /v1/info reports the current manager, and accepted jobs retain it through execution and API restart. Request arguments and X-Sparkrun-Client cannot select or override that identity.

For a separately managed gateway, issue its credential locally and provision the secret into a private token file:

sparkrun daemon token create gateway --scope admin --manager sparkroute

Manager identity and token scope are separate. Managed SparkRoute credentials need admin scope for catalog administration and do not receive secret-prompt permission. Non-default manager credentials issued through the CLI share a stable principal, so replacement credentials can resume retained jobs and retry keys.

Adopting a healthy workload does not transfer its owner. Automatic workloads.stop, workloads.sleep, workloads.wake, and workloads.reconcile require the matching manager, workload ID, and recipe revision. Another manager can observe and route to the endpoint. Explicit operator controls workloads.stop_matching and workloads.stop_all retain their broader scope. These guards prevent accidental automation cleanup within one OS account; they are not tenant isolation.

Placement, post-launch effects, and lifecycle receipts live separately from daemon job retention. A failed or lost job does not authorize another launch or replay of uncertain hooks. Inspect the workload, release verified-dead execution reservations with daemon resources reconcile, then reconcile with the workload’s manager credential:

sparkrun daemon --url https://controller:8443 --token-file /private/sparkroute.token \
  call workloads.reconcile --data \
  '{"recipe":"@registry/recipe","recipe_revision":"REVISION","workload_id":"WORKLOAD_ID","acknowledge_effects":true}' --wait

Reconciliation checks complete placement evidence and matching live identity; it does not rerun hooks. ColdSnap supplies a workload lifecycle provider, with status/sleep/wake accessed through these core operations.

Streams, restart, and storage

• GET /v1/jobs/{id}/events: durable SSE, sequence IDs, Last-Event-ID replay; HTTP 410 when the cursor precedes retained history. Fetch the job, then resume from the available history. Results over the event limit are fetched separately.
• GET /v1/events: bounded job-state snapshots for reconnecting clients.
• GET /v1/streams/logs: cancellable finite/follow logs for Docker and native Kubernetes workloads; bounded buffering with gap events. Select with workload_id, then optional host, container, or source_id filters. hosts remains the full workload topology. See Log Streaming API.
• GET /v1/streams/monitor: reference-counted monitor sessions, latest frame, five-second teardown linger. Disconnect/cancellation closes quiet reads too. A history event first supplies the shared session’s recent per-host samples; live frame events follow. Set monitor.history_points in config.yaml to change the per-host capacity (default 120). This is an in-memory window, not durable replay. New subscriptions use fresh sessions when the cluster’s executor or transport configuration changes. See monitor stream details.

Jobs, events, prompt metadata, and idempotency records live in <config-root>/daemon/<binding>/jobs.sqlite3 (WAL). Local discovery and any Unix socket use the private runtime directory, with a short private OS temporary path on POSIX if Unix socket length requires it. Directories are private and unsafe ancestors/symlinks are rejected. Unknown exceptions produce a generic wire error and private stack locations, without exception values or local-variable dumps.

Storage has not been migrated wholesale to SQLite:

State	Storage	Callers
Workload placement, post-launch effect and lifecycle receipts	<config-root>/workload-receipts.sqlite3	Shared workload APIs and plugin adapters
Durable operation jobs, events, results, prompts, retry keys	<config-root>/daemon/<binding>/jobs.sqlite3	Daemon and its workers
Resource reservations and owner identities	<config-root>/operations.sqlite3	Shared by daemon and standalone mutating API/CLI operations
Workload metadata (recipe, hosts, deployment identity)	<cache-root>/jobs/*.yaml	Existing shared workload APIs, including daemon calls
Model-cache usage, pins, and workload claims	HF_HOME/.sparkrun-model-artifacts/usage/ on each cache host	Preparation, confirmed stop, and model-cache management APIs
Configuration, cluster definitions, benchmark state	Existing YAML/JSON files	Existing core APIs

A daemon job_id identifies an operation invocation. A workload workload_id identifies the deployment produced or affected by an operation. For example, a daemon workloads.run job may finish successfully while the deployed inference service keeps running; its result includes the workload identity. daemon jobs list shows operation history, whereas daemon call workloads.list and Python sparkrun.api.list_jobs() list saved workload metadata. Ordinary standalone CLI operations do not create daemon job-history rows. The daemon reuses core workload storage; its durable operation journal is additional state, not a replacement workload catalog. Audited queries return directly without creating job rows.

API restart adopts workers through native process creation identities and a nonce challenge over bounded loopback TCP. Linux retains boot ID + /proc start time; macOS/Windows use process creation time. PID-only checks are never sufficient. Jobs write events/results directly while the API is down. Host reboot, worker crash, and replacing a Python environment do not preserve an in-memory operation.

POST /v1/shutdown requires admin authorization and stops only the API. CLI and desktop use it on every OS. Cancellation first requests cooperative cleanup; unresponsive workers become lost after forced termination. POSIX process mode tracks verified descendants, including nested sessions; arbitrary orphaned or malicious children are outside that tree guarantee. Linux systemd uses independent per-worker cgroups. Windows workers own named Job Objects: stopping the API does not close their containment handles, while worker crashes and forced cancellation terminate contained children. Successful jobs release crash cleanup so intentional persistent services can outlive the workflow. Windows Job Object assignment must succeed before execution; an incompatible outer supervisor fails closed.

Diagnostic capture

Collect a bounded bundle independently of a launch:

sparkrun daemon call diagnostics.collect '{"cluster":"mylab"}' --wait

For a saved deployment, supply workload_id to use its recorded placement and include workload logs. The HTTP alias is POST /v1/diagnostics/collect.

Run and benchmark jobs can capture their whole lifecycle:

sparkrun daemon call workloads.run \
  '{"recipe":"@registry/recipe","cluster":"mylab","collect_diagnostics":true}' --wait

benchmarks.run and benchmarks.resume accept the same opt-in argument. Capture includes operation-attributed DEBUG logs, feedback, and bounded plugin sections while retaining normal console thresholds. Benchmark-owned log tails are saved before workload cleanup. Capture failures are reported separately from workload success/failure.

The result or an artifact event supplies the bundle URL. An authenticated HTTP client downloads GET /v1/jobs/{job_id}/artifacts/diagnostics after completion, including a failed/cancelled job that produced a bundle. Downloads follow job ownership and operation scope and expire with job history. The CLI client prints results/events; it does not download artifacts automatically. Clients cannot set server file paths or supply privileged-probe credentials. See Diagnostics API for limits, redaction, and completion checks, and Daemon API for the HTTP contract.

Services and software updates

Foreground serving, background start/stop/restart, and GUI-owned lifetimes are portable. Native macOS LaunchAgent and Windows service/autostart installers are a separate follow-up; they must preserve independently running workers across API service restarts. Linux has the following systemd installer:

sparkrun daemon systemd install --now
sparkrun daemon systemd status
sparkrun daemon restart
sparkrun daemon systemd uninstall

The installer validates a user unit, binds it to this interpreter/config, and refuses to replace unrelated units. API and per-job units have separate cgroups; restarting/stopping the API service does not kill workers. Service mode requires --worker-supervisor systemd; the installer sets it. User lingering is an explicit administrator/user choice; installation does not enable it. For a remote service, add the TCP and TLS flags (or explicit --allow-insecure) to its ExecStart override. Certificates and tokens are provisioned separately.

Daemon and worker runtime leases pin core identity and managed-plugin generation IDs. Plugin activation requires API restart; admission enters maintenance when runtime identity changes. In-use generations cannot be removed or pruned, and retention never exceeds plugins.retain_generations (default 2, current/previous). An update that would evict a leased generation waits for the user to drain/stop its owners. Unmanaged external directories cannot provide immutable-generation guarantees.

sparkrun update and setup update refuse to replace an interpreter installation while any daemon/worker lease in that installation is live, including another config binding. Stop/drain all such instances first, update, then restart. Normal plugin compatibility evaluation stays in plugin management/update; daemon startup does not add a compatibility audit. The existing default plugin-update behavior and --no-update-plugins remain intact.

Configuration and bounds

Settings are under daemon: in the application config. Restart the API after changing admission/retention settings.

Key	Default	Meaning
max_services	2	Separate capacity for gateway/discovery service jobs; use at least 2 for both
max_jobs	4	Concurrent ordinary jobs, including waiting prompts; services use their own budget
max_queued_jobs	128	Accepted queued jobs
max_query_threads	8	Audited query threads
max_queued_queries	32	Total admitted active/waiting queries
query_timeout	30	Cooperative query deadline, seconds
max_streams	64	SSE subscriptions
max_body_bytes	1048576	HTTP request-body limit
max_events	4096	Retained events per job
max_event_bytes	8388608	Retained event bytes per job
cancel_grace	15	Seconds before forced worker cancellation
worker_start_timeout	60	Seconds allowed for worker registration
prompt_timeout	300	Seconds allowed for a prompt answer
job_retention_days	7	Terminal result/event retention
retained_jobs	1000	Maximum retained terminal jobs
discovery_interval	0	Opt-in endpoint snapshots; 0 disables, minimum 5 seconds

Results have a 2 MiB cap and individual events a 256 KiB cap. Log subscription queues hold 128 lines and emit gaps on overflow. Query caches hold 128 entries for at most two seconds and invalidate on config revision/job events. Query cancellation does not release a thread slot until underlying work ends; blocking third-party calls still need their own timeout. Idempotency tombstones remain for 30 days and active jobs retain their keys beyond that window. Never blindly retry an older uncertain mutation after its retry window expires.

Anonymous usage telemetry

Daemon workloads.run and benchmarks.run operations retain the existing core API telemetry emitters and payloads. All usage events now include via_daemon: true when emitted inside daemon-managed operation execution, false for direct CLI/Python execution. Attribution follows nested API work, worker processes, query threads, and stream producers through operation context. It is not inferred from a client-supplied header, job name, or process-wide environment flag.

Each serving process emits one daemon_serve event after all its listeners are ready. This covers foreground serve, background start, restart, systemd, and desktop-owned serving. Starting an already-running daemon does not produce another event; rejected startup does not report successful serving. Multiple listeners share one event. Its fields are unix_socket, tcp, tls, remote_tcp, worker_supervisor, parent_managed, startup_ms, and the usual system/version/installation envelope. Serving is not an operation request, so this event has via_daemon: false. There are no periodic heartbeat events.

No addresses, ports, socket/config paths, bearer tokens, principal IDs, job IDs, or raw client headers are added to usage events. The durable SQLite job journal is not uploaded. Host/GPU monitoring streams remain separate from anonymous usage reporting.

Policy and installation identity come from the daemon host’s application profile and bound config. telemetry.enabled: false disables reporting unless an explicit environment override forces it on. Set SPARKRUN_NO_TELEMETRY=1 in the serving process environment to opt out; remote client environment settings are not transmitted to the server. Ordinary process workers inherit the environment; systemd workers now explicitly receive telemetry opt-out, endpoint, key, and timeout overrides, including application-profile variable names and aliases. For installed services, configure the daemon unit environment or bound config; an environment variable on the client running daemon start does not configure the systemd service. New workers read the config; already-running jobs retain their existing context.

Serving-event delivery runs outside the API event loop and is best-effort, using the existing telemetry timeout and failure handling. Run/benchmark events keep their existing emission points; a run event can precede readiness checks or post-launch hooks and is not a report of final daemon job success.

Optional clients and services

SPARKRUN_DAEMON=auto enables selective read-only CLI passthrough for recipe listing/search and JSON cluster status. Discovery failures can fall back before submission; submitted operations are not retried locally. Normal CLI behavior remains the default.

daemon.discovery_interval enables advisory endpoint snapshots at /v1/snapshots/discovery.snapshot. It defaults to zero to avoid an unused duplicate sweep. Gateway discovery uses its own scope, interval, and removal grace through proxy.discovery_backend: daemon. The old daemon.proxy_autodiscover setting no longer schedules a watcher.

The desktop application uses the daemon API for operations, reconnection, cancellation, and prompts. It can attach to an existing verified daemon or start a GUI-owned TCP controller. Closing a GUI-owned API leaves durable jobs running.

MCP support is disabled. Its adapter implementation is retained for possible future reuse, but no MCP command or service entry point is registered.

Reusable worker pools remain deferred. Warm queries and shared monitoring provide the initial reuse without retaining plugin state between independent jobs.

Gateway discovery and transport

LiteLLM and SparkRoute share discovery sessions, host coverage, removal grace, and ownership checks. Keep the standalone process backend, or select daemon discovery explicitly after enabling the daemon feature:

sparkrun proxy start --discovery-backend daemon --cluster lab
sparkrun proxy status
sparkrun daemon call proxy.discovery.status

Startup attaches to or starts the matching local controller; dry runs do neither. The session pins the effective host/cluster selection, SSH context, interval, and removal grace. A changed default cluster takes effect on the next explicit start. proxy.auto_discover: false disables periodic discovery with either backend.

The gateway runs in a persistent proxy.serve service job, with a separate proxy.watch service job for discovery. Configure at least two daemon.max_services slots for both. These jobs survive an API restart and do not consume ordinary max_jobs capacity. Stopping discovery alone does not stop the gateway; stopping the proxy stops its matching discovery session even if the gateway has exited. Use proxy lifecycle commands to manage these jobs rather than repeatedly submitting raw service operations. proxy.watch and proxy.serve are internal executors with no public alias or submission path.

Discovery status reports the backend, generation/job, scope, last attempt, last complete successful observation, last application, sanitized error, and routing warnings. Warnings from a successful import do not fail the sweep or delay its normal interval. running means the loop is active. Metadata-only or partial coverage cannot prove absence on an unobserved host, and a failed configuration update does not advance removal grace. An uncertain or dead owner requires explicit stop/restart.

For SparkRoute’s HTTP/SSE client, select daemon transport in proxy.yaml:

proxy:
  gateway: sparkroute
  discovery_backend: daemon
  discover_interval: 30
  discover_removal_grace_sweeps: 2

gateways:
  sparkroute:
    transport: daemon

gateways.sparkroute.transport defaults to bridge. With SparkRoute selected, daemon transport defaults discovery to daemon unless a backend was explicitly saved. These settings do not affect another selected gateway. Legacy locations are not read or migrated. The plugin checks that the host operations and gateway binary support the selected transport; use compatible host, plugin, and gateway versions. An uncertain mutation never triggers an automatic fallback to the bridge.

Managed startup creates a private connection descriptor and a sparkroute manager credential. It follows local TCP token rotation without starting Python for every request. SparkRoute’s lifecycle, admin catalog, and profile-refresh clients use the selected transport. Its plugin contributes endpoint/model-card projections; core workloads.* owns lifecycle and provider inventory.

Changing transport first drains active gateway work. Daemon cutover refuses active bridge workers; switching back checks daemon jobs admitted under manager sparkroute and revokes its credentials under the admission lock. Existing placement/provider records remain readable; uncertain effects need reconciliation.

For a separately managed gateway, see the connection descriptor in the source daemon guide.

HTTP workflow

The authenticated API publishes /v1/ops and /openapi.json. Discover schemas before constructing a request. The following uses the public reference client, a token file, and the cluster-create alias. Cluster creation requires an admin token; the request creates a saved cluster definition and returns a job acknowledgement:

from pathlib import Path
from sparkrun.daemon.client import Client

client = Client(
    url="https://controller:8443",
    token=Path("/private/daemon.token").read_text().strip(),
    ca_cert="/private/ca.pem",
)
# Inspect these schemas before submission.
operations = client.request("GET", "/v1/ops")

submission = client.request(
    "POST",
    "/v1/cluster/create",
    {"name": "lab", "hosts": ["spark-a", "spark-b"]},
    headers={"Idempotency-Key": "create-lab-001"},
)
job_id = submission["job_id"]
job = client.request("GET", f"/v1/jobs/{job_id}")

HTTP 202 means the operation was accepted. Poll the job until it reaches a terminal state, handling any pending prompt, then inspect its result or error.

See Daemon API for request, prompt, retry, and result contracts, and daemon CLI for command options. Durable feedback is described in Structured Feedback; it is separate from workload logs and monitor frames.