Cluster Status
Check running workloads
Section titled “Check running workloads”sparkrun cluster statussparkrun status # aliasThis shows workloads reported by enabled executors on your cluster hosts, grouped by job.
Output
Section titled “Output”The status command displays:
- Running jobs — containers grouped by workload ID with role (head/worker/solo), host, status, and container image
- IP mapping — when InfiniBand was detected during launch, shows the complementary IP alongside each host (e.g. management IP if you defined the cluster by IB IPs, or vice versa)
- Idle hosts — hosts that have no sparkrun containers running
- Pending operations — downloads or image distributions currently in progress that haven’t launched containers yet, with elapsed time and a note that they will consume VRAM once launched
- Copy-paste commands — ready-to-use
sparkrun logsandsparkrun stopcommands for each running job
Example output
Section titled “Example output”$ sparkrun cluster statusJob: qwen3.5-35b-a3b-fp8-sglang (tp=2) (2 container(s)) node_0 10.24.11.13 (ib: 192.168.11.13) Up 35 seconds scitrera/dgx-spark-sglang:0.5.9-dev1-329817e2-t5 node_1 10.24.11.14 (ib: 192.168.11.14) Up 9 seconds scitrera/dgx-spark-sglang:0.5.9-dev1-329817e2-t5 logs: sparkrun logs qwen3.5-35b-a3b-fp8-sglang --hosts 10.24.11.13,10.24.11.14 --tp 2 stop: sparkrun stop qwen3.5-35b-a3b-fp8-sglang --hosts 10.24.11.13,10.24.11.14 --tp 2
Idle hosts (no sparkrun containers): 10.24.11.16 10.24.11.17
Total: 2 container(s) across 4 host(s)Options
Section titled “Options”# Check a specific clustersparkrun cluster status --cluster mylab
# Check a saved subgroupsparkrun cluster status --cluster pair-aIf no options are given, uses the default cluster.
Interactive monitoring and logs
Section titled “Interactive monitoring and logs”Run sparkrun cluster monitor --cluster mylab for live host metrics and workload
occupancy. Select a host for sensor sparklines, including network rates, or select
a workload to view logs, search them, or export retained history. See
host metrics and monitor controls.
Placement, state, and cache policy
Section titled “Placement, state, and cache policy”Hardware inventory, per-host executor destinations, and live reservations determine placement. Ordinary workloads reserve accelerators exclusively; a low engine memory fraction does not automatically permit co-location. See Schedulers. Workload metadata is shared across direct and daemon calls, while daemon operation history is separate. Missing observations are not proof of idle capacity. Cluster model validation/selection/provider settings and runtime cache cleanup are covered in Model Cache Management.