Cluster Commands
Clusters are named logical groups of hosts. A group can contain one node, and groups can overlap. Save all nodes as the default and use smaller named groups when useful. Prefer cluster references over explicit host lists in everyday commands; see Recommended Cluster Setups.
The scheduler selects and orders eligible hosts for each job; the first selected host becomes its head node. See Schedulers & Placement for hardware constraints, reservations, and explicit layouts.
Create a cluster
Section titled “Create a cluster”sparkrun cluster create <name> --hosts <ip1>,<ip2>,... [options]| Option | Description |
|---|---|
--hosts / -H | Comma-separated host list |
--hosts-file | File with hosts (one per line) |
-d / --description | Cluster description |
--user / -u | SSH username for this cluster |
--cache-dir | HuggingFace cache directory on cluster hosts (useful when control machine and cluster have different paths, e.g. macOS → Linux) |
--transfer-mode | Resource transfer mode: auto, local, push, or delegated |
--transfer-interface | Network interface for transfers: auto, cx7, or mgmt |
--scheduler | Placement scheduler (default occupancy-sparse). See Schedulers & Placement. |
--max-gpu-mem-util | Fraction of each accelerator’s memory treated as usable for placement |
--executor | Pin the launch executor for this cluster |
--executor-opt / -o | Per-executor config, -o key=value (repeatable) |
--default | Set this cluster as the default after creating it |
--offline | Make launches use existing artifacts by default; see Offline defaults |
sparkrun cluster create mylab \ --hosts 10.24.11.13,10.24.11.14 \ --user dgxuser \ -d "2-node DGX Spark cluster"
# With transfer mode for external control machinessparkrun cluster create mylab \ --hosts 10.24.11.13,10.24.11.14 \ --transfer-mode pushSee Resource Transfer Modes for details on transfer mode options.
Update a cluster
Section titled “Update a cluster”sparkrun cluster update <name> [options]All options are the same as create. Only provided options are updated — omitted fields are left unchanged. Pass --user "" or --cache-dir "" to explicitly clear a value.
update additionally accepts:
| Option | Description |
|---|---|
--add-host | Add a host without restating the whole list (repeatable) |
--remove-host | Remove a host (repeatable) |
--topology | Set the cluster’s network topology |
--infer-hardware | Probe hosts and record detected accelerator metadata |
--clear-executor-config | Drop any stored per-executor config |
--offline / --online | Enable or remove the cluster’s offline launch default |
sparkrun cluster update mylab --hosts 10.24.11.13,10.24.11.14,10.24.11.15sparkrun cluster update mylab --add-host 10.24.11.15sparkrun cluster update mylab --transfer-mode delegatedsparkrun cluster update mylab --user newuser
# Adopt the 0.3.x default placement on a cluster created by sparkrun 0.2.xsparkrun cluster update mylab --scheduler occupancy-sparseSee Resource Transfer Modes for details on transfer mode options.
List clusters
Section titled “List clusters”sparkrun cluster listShows all saved clusters with their hosts and descriptions. The default cluster is marked with *.
Show cluster details
Section titled “Show cluster details”sparkrun cluster show <name>Displays name, description, user, cache directory, transfer mode, default status, and all hosts.
Delete a cluster
Section titled “Delete a cluster”sparkrun cluster delete <name>sparkrun cluster delete <name> --force # skip confirmation promptInspect effective configuration
Section titled “Inspect effective configuration”sparkrun cluster inspect [name]Shows the resolved cluster settings — transfer mode, transfer interface, topology, SSH user, and cache directories — and checks whether those cache directories actually exist on each remote host. Useful for diagnosing configuration, transfer, or permission problems without running a job.
Import an existing cluster
Section titled “Import an existing cluster”sparkrun cluster import svd <path-to-.env>Imports a spark-vllm-docker .env file into a sparkrun cluster, mapping
CLUSTER_NODES to hosts, ETH_IF to fabric interfaces, and CONTAINER_* to
env references. The env references are resolved from the file at launch time,
so secrets stay out of the cluster YAML. Re-running syncs in place. Aliased as
sparkrun cluster import eugr.
Set/unset default cluster
Section titled “Set/unset default cluster”sparkrun cluster set-default <name>sparkrun cluster unset-defaultsparkrun cluster default # show current defaultWhen a default cluster is set, commands like sparkrun run and sparkrun stop use it automatically without needing --cluster or --hosts.
Check cluster status
Section titled “Check cluster status”sparkrun cluster statussparkrun status # aliasShows running sparkrun workloads, pending operations (downloads/distributions in progress), and idle hosts across the cluster. See Cluster Status for full details.
Example Docker workload output (addresses and image tags are illustrative):
$ sparkrun cluster statusJob: qwen3.5-35b-a3b-fp8-sglang (tp=2) (2 container(s)) node_0 10.24.11.13 (ib: 192.168.11.13) Up 35 seconds scitrera/dgx-spark-sglang:0.5.9-dev1-329817e2-t5 node_1 10.24.11.14 (ib: 192.168.11.14) Up 9 seconds scitrera/dgx-spark-sglang:0.5.9-dev1-329817e2-t5 logs: sparkrun logs qwen3.5-35b-a3b-fp8-sglang --hosts 10.24.11.13,10.24.11.14 --tp 2 stop: sparkrun stop qwen3.5-35b-a3b-fp8-sglang --hosts 10.24.11.13,10.24.11.14 --tp 2
Idle hosts (no sparkrun containers): 10.24.11.16 10.24.11.17
Total: 2 container(s) across 4 host(s)Monitor cluster
Section titled “Monitor cluster”sparkrun cluster monitor [options]Live-monitor CPU, RAM, and GPU metrics across cluster hosts, with interactive
workload log viewing. By default this launches a TUI; pass --simple for
plain-text output or --json for telemetry samples. Press q on the main screen
or Ctrl-C to stop.

| Option | Description |
|---|---|
--hosts / -H | Comma-separated host list |
--hosts-file | File with hosts (one per line) |
--cluster | Use a saved cluster by name |
--interval / -i | Sampling interval in seconds (default: 2) |
--simple | Use plain-text output instead of TUI |
--json | Emit telemetry samples as JSON |
--dry-run / -n | Show what would be done |
sparkrun cluster monitor --cluster mylabsparkrun cluster monitor --cluster mylab --interval 5sparkrun cluster monitor --cluster mylab --simpleHost metrics and history
Section titled “Host metrics and history”Select a host to see its CPU and GPU identity, memory use, and recent sensor
history. The detail panel uses compact sparklines for CPU/GPU utilization,
temperatures, GPU power and clock, RAM, and network send/receive rates. The
nv-monitor backend also supplies application/cache memory breakdowns when available.
Unavailable metrics display - and are skipped in the history.
By default, the session retains up to 120 telemetry points per host, about four minutes at
the default two-second interval. Collection continues while you view another
host or workload logs; history starts when the monitor opens and is not persisted.
Network rates use adaptive bits-per-second units in the TUI; JSON sample fields
net_send_bytes_s and net_recv_bytes_s use bytes per second. The Bash backend
sums non-loopback interfaces.
Set the per-host history capacity in config.yaml:
monitor: history_points: 300 # About ten minutes at a two-second sampling intervalUse a positive integer; missing, invalid, or non-positive settings fall back to 120. Repeated reads of the same sample do not add points, so stalled telemetry does not appear as fresh flat-line readings. The window is bounded by sample count, not a guaranteed duration.
Applications can access the same session history through the Python monitoring API.
Interactive workload logs
Section titled “Interactive workload logs”Select a host with Enter to open its workload list, then choose a workload with Enter or a mouse click to stream its logs. The viewer colorizes known runtime output and provides a fallback for other runtimes.
| Key | Action in the log viewer |
|---|---|
| Space | Pause/follow, or jump back to the live tail when scrolled away |
1, 2, 3 | Select container, node, or whole-workload cluster scope where available |
/ | Search with a regular expression |
n, b | Next or previous search match |
h | Toggle search-term highlighting |
c | Toggle log colorization |
l | Toggle line wrapping |
d | Export logs to a local file |
Escape or q | Return from the log viewer |
Scroll upward to page older retained history. Cluster scope combines the selected
workload’s sources in arrival order while following. Searching operates on loaded
history, and paging depends on the logs still retained by the backend. Once the
search input is focused, typing / inserts a literal slash into your expression.
The export dialog lets you choose scope, path, and whether to overwrite an
existing file. It reads complete retained logs for that scope, not just the visible
buffer. Cancelled or failed exports discard partial output. Viewing, exporting,
or closing logs leaves inference running. For scripts, use
sparkrun logs or the Log Streaming API.