Managing Clusters

List all clusters

sparkrun cluster list

Shows all saved clusters with their hosts and whether they are the default.

Show cluster details

sparkrun cluster show mylab

Displays the cluster’s hosts, description, SSH user, and default status.

Inspect effective configuration

sparkrun cluster inspect mylab

Shows the resolved settings — transfer mode, transfer interface, topology, SSH user, cache directories — and checks whether those cache directories exist on each host. The fastest way to diagnose a transfer or permission problem without launching a job.

Cluster-level environment

A cluster can carry container environment variables that apply to every workload launched on it. They sit below recipe env and CLI overrides in precedence: CLI > recipe env > cluster env.

~/.config/sparkrun/clusters/mylab.yaml

env:
  HF_TOKEN: "${HF_TOKEN}"
env_file: /home/me/.sparkrun.env

${VAR} references resolve at launch time from env_file — read locally on the control machine and emitted per host, so the file never has to exist on the cluster nodes. Secrets therefore stay in your env file and never land in the cluster YAML.

Note

References resolve only against env_file — never against os.environ or recipe env — so this cannot be used to pull arbitrary host secrets into a container. A reference with no matching key, or with no env_file configured, is an error rather than an empty string.

This block is typically populated by sparkrun cluster import, which maps a legacy CONTAINER_* block into it. Import owns the block and rewrites it wholesale on re-sync.

Delete a cluster

sparkrun cluster delete mylab

Removes the cluster configuration. Use --force to skip the confirmation prompt. This does not affect running workloads.

Default cluster

Set a default

sparkrun cluster set-default mylab

Show the current default

sparkrun cluster default

Remove the default

sparkrun cluster unset-default

When a default cluster is set, workload commands (run, stop, logs, status) use it automatically unless --hosts or --cluster is specified.

Cluster status

sparkrun cluster status
sparkrun cluster status --cluster mylab

Shows sparkrun containers running on cluster hosts — container names, images, and status.

Cluster monitor

sparkrun cluster monitor              # Interactive TUI (default)
sparkrun cluster monitor --simple     # Plain text output
sparkrun cluster monitor --json       # JSON output for scripting

Live-monitors CPU, RAM, and GPU metrics across all hosts in the cluster. The TUI mode (default) uses Textual for a rich terminal interface with progress bars. Press q to quit.

Option	Description
--cluster	Target a specific cluster
--interval	Refresh interval in seconds (default: 5)
--simple	Plain text output instead of TUI
--json	JSON output for scripting

Check job status

sparkrun cluster check-job <recipe>

Check whether a specific recipe’s containers are still running across cluster hosts.