Skip to content

SSH Setup

All multi-node orchestration in sparkrun relies on SSH. At minimum, you need passwordless SSH from your control machine to every cluster node.

Resource transfers use the effective local, push, delegated, or per-node pull policy. Controller-to-host SSH is required; node-to-node access depends on that policy. However, setting up a full SSH mesh (every host can reach every other host) is recommended.

The easiest approach is sparkrun setup ssh, which creates a full mesh across your cluster hosts and the control machine (included automatically):

Terminal window
# Set up SSH mesh across your saved cluster + this machine
sparkrun setup ssh --cluster mylab
# Or use your default cluster
sparkrun setup ssh
# Add extra hosts beyond the cluster
sparkrun setup ssh --cluster mylab --extra-hosts 10.0.0.99
# Exclude the control machine from the mesh
sparkrun setup ssh --cluster mylab --no-include-self
# Skip IP discovery phase (InfiniBand, CX7)
sparkrun setup ssh --cluster mylab --no-discover-ips

You will be prompted for passwords on first connection to each host. After that, every host in the mesh can SSH to every other host without passwords.

After installing keys, sparkrun verifies that pubkey authentication actually works by testing each host independently (without reusing the password-authenticated connection). If verification fails, it prints diagnostics and actionable remediation steps but continues with the inter-node mesh.

If you prefer to set up SSH yourself:

Terminal window
# Generate a key if you don't have one
ssh-keygen -t ed25519
# Copy to each node
ssh-copy-id 192.168.11.13
ssh-copy-id 192.168.11.14

By default, sparkrun uses your current OS user for SSH. You can configure the user per-cluster:

Terminal window
# Set during cluster creation
sparkrun cluster create mylab --hosts 192.168.11.13,192.168.11.14 --user dgxuser
# Update an existing cluster
sparkrun cluster update mylab --user dgxuser

Or override per-command with --user.

Cross-user SSH (control machine ≠ cluster user)

Section titled “Cross-user SSH (control machine ≠ cluster user)”

A common setup is running sparkrun from a control machine where your local user (e.g., me) differs from the cluster SSH user (e.g., dgxuser). sparkrun handles this automatically — the mesh script installs your local user’s public key into the remote user’s authorized_keys.

If cross-user SSH fails after the mesh completes, the most common cause is home directory permissions. OpenSSH’s StrictModes (enabled by default) rejects keys when the remote user’s home directory is group- or world-writable.

sparkrun automatically fixes this during setup (chmod go-w ~), but if you encounter issues, use --diagnose to pinpoint the problem:

Terminal window
sparkrun setup ssh --cluster mylab --diagnose

This runs comprehensive diagnostics on each host:

  • Tests pubkey authentication independently (without password)
  • Checks home directory, .ssh/, and authorized_keys permissions
  • Inspects sshd_config for AuthorizedKeysFile, StrictModes, AllowUsers/AllowGroups
  • Verifies your local public key is present in the remote authorized_keys
  • Prints pass/fail results with specific remediation steps

For non-default ports, identity files, or jump hosts, use ~/.ssh/config:

Host spark1
HostName 192.168.11.13
User dgxuser
Host spark2
HostName 192.168.11.14
User dgxuser

The SSH user must be a member of the docker group on every cluster node:

Terminal window
sudo usermod -aG docker "$USER"

Bootstrap reuses configured/existing SSH identities when possible. New Sparkrun- owned login keys live beside the active config file under ssh/sparkrun_ed25519, not automatically under ~/.ssh/. Existing keys are not regenerated or overwritten. These client login identities are different from the server host keys below.

The read-only ssh_identity setup step checks SSH server public keys and /etc/machine-id across cluster hosts. Both setup check (including --json) and the wizard use probe_setup_hosts and the same step evaluation. The DGX Spark and generic NVIDIA plans include it for Docker and local executors on multi-host clusters. It is independent of mesh readiness, so an already working SSH mesh does not hide duplicate identities. The setup.steps.ssh_identity feature flag and per-host setup constraints control its probes and findings. setup ssh also checks its cluster targets after successful mesh setup. Duplicates produce warnings identifying the affected hosts and a suggested rotation command. Unreachable hosts or missing identity data make the audit incomplete; they are never reported as verified unique. Findings are advisories and never trigger rotation automatically. For a complete audit with duplicates, the wizard offers the same confirmed rotation and mesh refresh as ssh-rotate-keys, then reprobes readiness. Machine-ID duplication adds machine-ID regeneration to that offer, with a reboot reminder. Declining leaves the warnings visible. The wizard’s --yes accepts the safe default (no rotation) and prints remediation without prompting; deliberate unattended rotation uses setup ssh-rotate-keys --yes. setup check always remains read-only.

The comparison covers only the selected, reachable machines. Addresses sharing one Linux boot ID are treated as aliases of the same running machine.

Terminal window
sparkrun setup ssh-rotate-keys --cluster mylab --dry-run
sparkrun setup ssh-rotate-keys --cluster mylab
sparkrun setup ssh-rotate-keys --cluster mylab --machine-id

Rotation prompts for confirmation; --yes supplies explicit noninteractive confirmation. Sudo may still require a password. --hosts, --hosts-file, and --user use the usual setup target selection. The control machine is a rotation target only when selected as one of the hosts. The command rotates the standard RSA, ECDSA, and Ed25519 server host keys. It preserves SSH login private keys and refreshes the same passwordless mesh used by setup ssh for the selected hosts, using their configured SSH user, key, and options.

--machine-id also generates a fresh machine ID and updates a regular D-Bus machine-ID copy. A D-Bus symlink to /etc/machine-id follows that change. Plan a reboot afterward: services retain their existing OS identity until restarted, and machine-ID-derived network identifiers may change. Sparkrun does not reboot hosts. Key rotation alone reloads SSH; it does not require a reboot.

Targets must be Linux machines with Python 3, OpenSSH, sudo, and an active, reloadable systemd SSH service. Existing host trust must verify before any keys change. Custom SSH server key paths, host certificates, HostKeyAgent, and matching wildcard/negated known_hosts entries require administrator-managed rotation. The controller needs OpenSSH, but no local bash or sudo.

The console-free API exposes check_ssh_host_identities(...) and rotate_ssh_host_keys(...) in sparkrun.api.setup. It returns structured identity/duplicate reports and per-host rotation outcomes. The frontend obtains confirmation and supplies optional SudoPasswords callbacks. The audit sends public keys and hashes of machine/boot IDs; it does not export private keys or raw machine IDs.

New keys are generated in a root-only staging directory before activation. Their public halves are received through the existing authenticated connection and saved under <cache_dir>/ssh-rotations/<operation>/recovery.json. The command validates SSH configuration, reloads the service, and verifies a fresh connection pinned to the staged keys. It then updates the controller’s effective UserKnownHostsFile files and the selected peers’ ~/.ssh/known_hosts, including per-host loopback entries and unique discovered network addresses. Other clients must update their own trust records. This handoff does not use ssh-keyscan or disable host-key verification.

Failures exit nonzero and identify each affected host. A failed activation or verification attempts to restore that host’s original identities. Successfully rotated hosts remain rotated. Old private keys are deleted after verification and controller trust update; a failed rollback or interruption retains them under /etc/ssh/.sparkrun-key-rotation. A subsequent rotation refuses to overwrite that recovery directory.

Use the printed recovery record to inspect the operation token, target names, old/new public keys, and completion states. For an interrupted activation, its *-recovery.known_hosts file contains only the authenticated old and staged keys; *-new.known_hosts contains only the staged keys. These can be supplied as OpenSSH’s UserKnownHostsFile with StrictHostKeyChecking=yes and ControlPath=none to establish a recovery connection. Verify the machine’s current identity before updating trust or removing recovery files. The remote state.json maps original files to the root-only old/ backups; keep those backups until the outcome is resolved. Do not bypass a host-key mismatch with StrictHostKeyChecking=no.