Multi-Node Tensor Parallelism

This tutorial shows how to run models that exceed a single DGX Spark’s 128 GB unified memory by splitting them across multiple nodes with tensor parallelism.

Prerequisites

• 2 or more DGX Sparks with SSH access
• A cluster created (via sparkrun cluster create or the setup wizard)
• SSH mesh configured (sparkrun setup ssh)
• ConnectX-7 networking configured (sparkrun setup cx7) — recommended for fast NCCL communication

If you haven’t done these yet, see Quick Start for the one-time setup steps.

Why multi-node?

Each DGX Spark has a single GPU with 128 GB of unified memory. Models that exceed this limit — or that benefit from more compute — can be split across multiple Sparks using tensor parallelism (TP).

TP splits model layers across GPUs. With sparkrun, each DGX Spark contributes one GPU, so --tp 2 means 2 hosts, --tp 4 means 4 hosts, and so on.

1. Check VRAM requirements

Use sparkrun show to see whether a model fits on a single node:

sparkrun show qwen3-30b-a3b-vllm

The VRAM estimate tells you the expected memory usage at the default tensor parallelism. To see the estimate at a different TP level, use the -o override:

# Compare VRAM at TP=1 vs TP=2
sparkrun show qwen3-30b-a3b-vllm
sparkrun show qwen3-30b-a3b-vllm -o tensor_parallel=2

If the estimate at TP=1 exceeds 128 GB, you need multi-node.

Tip

Some recipes set a min_nodes value that enforces a minimum TP. sparkrun will tell you if a recipe requires more nodes than you’ve specified.

2. Launch with tensor parallelism

Run with --tp 2 to spread the model across two Sparks:

sparkrun run qwen3-30b-a3b-vllm --tp 2

This uses your default cluster. You can also specify hosts explicitly:

sparkrun run qwen3-30b-a3b-vllm --tp 2 --hosts 10.24.11.13,10.24.11.14

3. What happens behind the scenes

When you launch a multi-node job, sparkrun:

1. Detects InfiniBand interfaces on each host for fast inter-node communication
2. Configures NCCL environment variables to use the high-speed network
3. Syncs the container image to all hosts (skipped if already present)
4. Syncs the model to all hosts (uses InfiniBand IPs for fast rsync when available)
5. Starts containers on all placed hosts — the first host of the placement becomes the head node, the others are workers
6. Establishes distributed communication between nodes (via Ray, native distributed init, or RPC depending on the runtime)

The head node runs the inference API endpoint. Worker nodes participate in tensor-parallel computation but don’t serve HTTP traffic.

Which host is the head?

The head is the first host of the scheduled placement, not automatically the first host in your cluster definition. With the default occupancy-sparse scheduler, placement is chosen from the least-loaded hosts, so the head can differ between launches as the cluster fills up. Run sparkrun status to see which host actually took the head role — its output also gives you the ready-to-use logs and stop commands.

4. Verify the deployment

Check that all nodes are running:

sparkrun status

You should see containers on each host — one head and one or more workers — all showing as running.

5. Monitor your cluster

Watch live resource usage across all nodes:

sparkrun cluster monitor

This opens a TUI (terminal UI) showing CPU, RAM, and GPU utilization on each host. For a simpler non-interactive view:

sparkrun cluster monitor --simple

6. Test the endpoint

The API is served on the head node. sparkrun status tells you which host that is:

curl http://<head-node-ip>:8000/v1/models

Usage is identical to single-node — the tensor parallelism is transparent to API clients.

7. Stop the workload

When stopping a multi-node job, match the --tp you used at launch:

sparkrun stop qwen3-30b-a3b-vllm --tp 2

Caution

If you launched with --tp 2, you must also pass --tp 2 to stop and logs. This is because the --tp value affects the cluster ID that sparkrun uses to identify the job’s containers.

Tips

• Head node selection: The head is the first host of the scheduled placement, which is not necessarily the first host in your cluster definition — see the note below.
• InfiniBand matters: CX-7 networking provides dramatically faster model syncing and NCCL communication. If multi-node jobs are slow to start or inference latency is high, make sure CX-7 is configured (sparkrun setup cx7).
• Node selection: If your cluster has more hosts than the requested TP level, sparkrun uses only as many as the job needs. Which ones depends on the scheduler: the default occupancy-sparse picks the least-loaded hosts, while greedy always takes the first N in the definition.

Next steps

• Measure performance: Compare single-node vs multi-node throughput with Benchmarking Models
• Serve multiple models: Use the Proxy Gateway to expose models through a unified API