SSH Setup
All multi-node orchestration in sparkrun relies on SSH. At minimum, you need passwordless SSH from your control machine to every cluster node.
Resource transfers use the effective local, push, delegated, or per-node pull policy. Controller-to-host SSH is required; node-to-node access depends on that policy. However, setting up a full SSH mesh (every host can reach every other host) is recommended.
Automatic setup with sparkrun
Section titled “Automatic setup with sparkrun”The easiest approach is sparkrun setup ssh, which creates a full mesh across your cluster hosts and the control machine (included automatically):
# Set up SSH mesh across your saved cluster + this machinesparkrun setup ssh --cluster mylab
# Or use your default clustersparkrun setup ssh
# Add extra hosts beyond the clustersparkrun setup ssh --cluster mylab --extra-hosts 10.0.0.99
# Exclude the control machine from the meshsparkrun setup ssh --cluster mylab --no-include-self
# Skip IP discovery phase (InfiniBand, CX7)sparkrun setup ssh --cluster mylab --no-discover-ipsYou will be prompted for passwords on first connection to each host. After that, every host in the mesh can SSH to every other host without passwords.
After installing keys, sparkrun verifies that pubkey authentication actually works by testing each host independently (without reusing the password-authenticated connection). If verification fails, it prints diagnostics and actionable remediation steps but continues with the inter-node mesh.
Manual SSH setup
Section titled “Manual SSH setup”If you prefer to set up SSH yourself:
# Generate a key if you don't have onessh-keygen -t ed25519
# Copy to each nodessh-copy-id 192.168.11.13ssh-copy-id 192.168.11.14SSH user configuration
Section titled “SSH user configuration”By default, sparkrun uses your current OS user for SSH. You can configure the user per-cluster:
# Set during cluster creationsparkrun cluster create mylab --hosts 192.168.11.13,192.168.11.14 --user dgxuser
# Update an existing clustersparkrun cluster update mylab --user dgxuserOr override per-command with --user.
Cross-user SSH (control machine ≠ cluster user)
Section titled “Cross-user SSH (control machine ≠ cluster user)”A common setup is running sparkrun from a control machine where your local user (e.g., me) differs from the cluster SSH user (e.g., dgxuser). sparkrun handles this automatically — the mesh script installs your local user’s public key into the remote user’s authorized_keys.
If cross-user SSH fails after the mesh completes, the most common cause is home directory permissions. OpenSSH’s StrictModes (enabled by default) rejects keys when the remote user’s home directory is group- or world-writable.
sparkrun automatically fixes this during setup (chmod go-w ~), but if you encounter issues, use --diagnose to pinpoint the problem:
sparkrun setup ssh --cluster mylab --diagnoseThis runs comprehensive diagnostics on each host:
- Tests pubkey authentication independently (without password)
- Checks home directory,
.ssh/, andauthorized_keyspermissions - Inspects
sshd_configforAuthorizedKeysFile,StrictModes,AllowUsers/AllowGroups - Verifies your local public key is present in the remote
authorized_keys - Prints pass/fail results with specific remediation steps
Advanced SSH configuration
Section titled “Advanced SSH configuration”For non-default ports, identity files, or jump hosts, use ~/.ssh/config:
Host spark1 HostName 192.168.11.13 User dgxuser
Host spark2 HostName 192.168.11.14 User dgxuserDocker group
Section titled “Docker group”The SSH user must be a member of the docker group on every cluster node:
sudo usermod -aG docker "$USER"Login identity location
Section titled “Login identity location”Bootstrap reuses configured/existing SSH identities when possible. New Sparkrun-
owned login keys live beside the active config file under ssh/sparkrun_ed25519,
not automatically under ~/.ssh/. Existing keys are not regenerated or overwritten.
These client login identities are different from the server host keys below.
SSH host identity audit and rotation
Section titled “SSH host identity audit and rotation”The read-only ssh_identity setup step checks SSH server public keys and
/etc/machine-id across cluster hosts. Both setup check (including --json)
and the wizard use probe_setup_hosts and the same step evaluation. The DGX
Spark and generic NVIDIA plans include it for Docker and local executors on
multi-host clusters. It is independent of mesh readiness, so an already working
SSH mesh does not hide duplicate identities. The setup.steps.ssh_identity
feature flag and per-host setup constraints control its probes and findings.
setup ssh also checks its cluster targets after successful mesh setup.
Duplicates produce warnings identifying the affected hosts and a suggested
rotation command. Unreachable hosts or missing identity data make the audit
incomplete; they are never reported as verified unique. Findings are advisories
and never trigger rotation automatically. For a complete audit with duplicates,
the wizard offers the same confirmed rotation and mesh refresh as
ssh-rotate-keys, then reprobes readiness. Machine-ID duplication adds
machine-ID regeneration to that offer, with a reboot reminder. Declining leaves
the warnings visible. The wizard’s --yes accepts the safe default (no rotation)
and prints remediation without prompting; deliberate unattended rotation uses
setup ssh-rotate-keys --yes. setup check always remains read-only.
The comparison covers only the selected, reachable machines. Addresses sharing one Linux boot ID are treated as aliases of the same running machine.
sparkrun setup ssh-rotate-keys --cluster mylab --dry-runsparkrun setup ssh-rotate-keys --cluster mylabsparkrun setup ssh-rotate-keys --cluster mylab --machine-idRotation prompts for confirmation; --yes supplies explicit noninteractive
confirmation. Sudo may still require a password. --hosts, --hosts-file, and
--user use the usual setup target selection. The control machine is a rotation
target only when selected as one of the hosts. The command rotates the standard
RSA, ECDSA, and Ed25519 server host keys. It preserves SSH login private keys
and refreshes the same passwordless mesh used by setup ssh for the selected
hosts, using their configured SSH user, key, and options.
--machine-id also generates a fresh machine ID and updates a regular D-Bus
machine-ID copy. A D-Bus symlink to /etc/machine-id follows that change. Plan a
reboot afterward: services retain their existing OS identity until restarted,
and machine-ID-derived network identifiers may change. Sparkrun does not reboot
hosts. Key rotation alone reloads SSH; it does not require a reboot.
Targets must be Linux machines with Python 3, OpenSSH, sudo, and an active,
reloadable systemd SSH service. Existing host trust must verify before any keys
change. Custom SSH server key paths, host certificates, HostKeyAgent, and
matching wildcard/negated known_hosts entries require administrator-managed
rotation. The controller needs OpenSSH, but no local bash or sudo.
The console-free API exposes check_ssh_host_identities(...) and
rotate_ssh_host_keys(...) in sparkrun.api.setup. It returns structured
identity/duplicate reports and per-host rotation outcomes. The frontend obtains
confirmation and supplies optional SudoPasswords callbacks. The audit sends
public keys and hashes of machine/boot IDs; it does not export private keys or
raw machine IDs.
Trust handoff and interrupted rotation
Section titled “Trust handoff and interrupted rotation”New keys are generated in a root-only staging directory before activation. Their
public halves are received through the existing authenticated connection and
saved under <cache_dir>/ssh-rotations/<operation>/recovery.json. The command
validates SSH configuration, reloads the service, and verifies a fresh connection
pinned to the staged keys. It then updates the controller’s effective
UserKnownHostsFile files and the selected peers’ ~/.ssh/known_hosts, including
per-host loopback entries and unique discovered network addresses. Other clients
must update their own trust records. This handoff does not use ssh-keyscan or
disable host-key verification.
Failures exit nonzero and identify each affected host. A failed activation or
verification attempts to restore that host’s original identities. Successfully
rotated hosts remain rotated. Old private keys are deleted after verification and
controller trust update; a failed rollback or interruption retains them under
/etc/ssh/.sparkrun-key-rotation. A subsequent rotation refuses to overwrite
that recovery directory.
Use the printed recovery record to inspect the operation token, target names,
old/new public keys, and completion states. For an interrupted activation, its
*-recovery.known_hosts file contains only the authenticated old and staged
keys; *-new.known_hosts contains only the staged keys. These can be supplied as
OpenSSH’s UserKnownHostsFile with StrictHostKeyChecking=yes and
ControlPath=none to establish a recovery connection. Verify the machine’s
current identity before updating trust or removing recovery files. The remote
state.json maps original files to the root-only old/ backups; keep those
backups until the outcome is resolved. Do not bypass a host-key mismatch with
StrictHostKeyChecking=no.