Skip to content

Managing Clusters

Use the all-node group as your default and add overlapping subgroups as needed. A single-node group is a normal cluster too. See Recommended Cluster Setups.

Terminal window
sparkrun cluster list

Shows all saved clusters with their hosts and whether they are the default.

Terminal window
sparkrun cluster show mylab

Displays the cluster’s hosts, description, SSH user, and default status.

Terminal window
sparkrun cluster inspect mylab

Shows the resolved settings — transfer mode, transfer interface, topology, SSH user, cache directories — and checks whether those cache directories exist on each host. The fastest way to diagnose a transfer or permission problem without launching a job.

Save offline: true on a cluster to make run, benchmark, and proxy load reuse existing artifacts by default:

Terminal window
sparkrun cluster update mylab --offline
sparkrun cluster update mylab --online

--online removes the saved offline default. Individual operations can override the cluster policy with --offline or --online. Copies within the cluster are allowed; required recipes, images, models, and tools must already be available. See Offline launches.

A cluster can carry container environment variables that apply to every workload launched on it. They sit below recipe env and CLI overrides in precedence: CLI > recipe env > cluster env.

~/.config/sparkrun/clusters/mylab.yaml
env:
HF_TOKEN: "${HF_TOKEN}"
env_file: /home/me/.sparkrun.env

${VAR} references resolve at launch time from env_file — read locally on the control machine and emitted per host, so the file never has to exist on the cluster nodes. Secrets therefore stay in your env file and never land in the cluster YAML.

This block is typically populated by sparkrun cluster import, which maps a legacy CONTAINER_* block into it. Import owns the block and rewrites it wholesale on re-sync.

Terminal window
sparkrun cluster delete mylab

Removes the cluster configuration. Use --force to skip the confirmation prompt. This does not affect running workloads.

Terminal window
sparkrun cluster set-default mylab
Terminal window
sparkrun cluster default
Terminal window
sparkrun cluster unset-default

When a default cluster is set, workload commands (run, stop, logs, status) use it automatically unless --hosts or --cluster is specified.

Terminal window
sparkrun cluster status
sparkrun cluster status --cluster mylab

Shows sparkrun containers running on cluster hosts — container names, images, and status.

Terminal window
sparkrun cluster monitor # Interactive TUI (default)
sparkrun cluster monitor --simple # Plain text output
sparkrun cluster monitor --json # JSON output for scripting

Live-monitors CPU, RAM, GPU, and network metrics across hosts. Select a host for recent sensor sparklines, or select a workload to open searchable, colorized logs and export retained history. See host metrics and monitor controls. Press q on the main screen to quit.

OptionDescription
--clusterTarget a specific cluster
--intervalRefresh interval in seconds (default: 2)
--simplePlain text output instead of TUI
--jsonJSON output for scripting
Terminal window
sparkrun cluster check-job <recipe>

Check whether a specific recipe’s containers are still running across cluster hosts.

Hardware inventory, per-host executor destinations, and live reservations determine placement. Ordinary workloads reserve accelerators exclusively; a low engine memory fraction does not automatically permit co-location. See Schedulers. Workload metadata is shared across direct and daemon calls, while daemon operation history is separate. Missing observations are not proof of idle capacity. Cluster model validation/selection/provider settings and runtime cache cleanup are covered in Model Cache Management.