Executors
An executor is the layer that actually launches your workload on a host.
| Selector | Status | Feature flag | What it does |
|---|---|---|---|
docker | Stable | executor.docker (on) | docker run per host. The default and the production-supported path. |
local | Experimental | executor.local (off) | Native subprocess via setsid. No container. PID/log files under ~/.cache/sparkrun/local/. |
Recipes select an executor via the top-level executor: field; per-executor
configuration lives under executor_config:. See
Recipe format › Executor fields for the
full field reference.
When to use each
Section titled “When to use each”- Docker (default) — every shipped DGX Spark recipe, every multi-node cluster runtime that’s been validated end-to-end, every benchmark profile. Stay on Docker unless you have a specific reason to leave.
- Local — bare-metal experimentation when you don’t want a container in
the loop, on a supported Linux target. Portable controllers do not qualify Apple GPU execution. It uses the host environment or an explicitly prepared environment. Multi-host native
cluster runtimes (
vllmwith distributed/mp, andsglang) work; Ray-based runtimes do not.
Resolution chain
Section titled “Resolution chain”Which executor is picked from the first layer that names one, highest priority first:
- CLI override (
-o executor=local). - Recipe (
executor:). - Cluster pin (
sparkrun cluster create … --executor local). - Runtime default (
runtime.default_executor()). SparkrunConfig.default_executorinconfig.yaml.- The baseline default —
docker, unless it has been disabled, in which case the sole remaining enabled executor is used.
Its configuration (executor_config:) is layered separately, again highest
priority first: CLI → recipe → runtime default → per-executor runtime
adjustments → SparkrunConfig.executor_config → per-executor defaults (Docker
ships DOCKER_DEFAULTS; Local ships none) → dataclass field defaults.
If a layer names an executor that is unknown or gated off, resolution raises
ExecutorUnavailableError naming the flag to enable. It never falls back to
Docker.
Docker launch defaults
Section titled “Docker launch defaults”Normal sparkrun Docker launches are non-privileged and include
no-new-privileges. Seccomp policy, Linux capabilities, and locked-memory limits
are separate settings.
Seccomp profiles
Section titled “Seccomp profiles”In 0.4.0, sparkrun defaults to seccomp=io-uring: a bundled, pinned copy of
Moby’s default profile
with an allow rule for io_uring_setup, io_uring_enter, and io_uring_register.
The default deny action, architecture mappings, and other rules remain intact.
Atlas uses this shared policy.
Select a profile through executor_config.security_opt in application config,
a cluster, a recipe, or API overrides:
| Setting | Behavior |
|---|---|
seccomp=io-uring | sparkrun’s default profile with the three io_uring syscalls allowed |
seccomp=builtin | Docker’s own default profile |
seccomp=/path/to/profile.json | A custom profile loaded from the controller |
seccomp=unconfined | Explicitly disables seccomp filtering |
These selections are interpreted by sparkrun; io-uring is not a Docker built-in
profile name. If no seccomp option is supplied, including an empty security-options
list, sparkrun selects io-uring. Explicitly privileged containers remain subject
to Docker’s privileged-container behavior.
For a custom policy:
executor_config: security_opt: - no-new-privileges - seccomp=/home/operator/policies/inference.jsonThe path is local to the controller running sparkrun. ~ expands to that user’s
home, and relative paths resolve against its working directory. sparkrun validates
and snapshots the policy before replacing a workload, then carries its contents
in each node’s launch script. The nodes do not need a copy at the same path.
Docker validates target-specific policy support when creating the container.
Custom profiles are used as supplied; include any io_uring permissions your
workload needs. Preview generation reads and validates the local file without
writing to remote hosts.
Capabilities and locked memory
Section titled “Capabilities and locked memory”The shared Docker defaults do not add IPC_LOCK or SYS_NICE. A runtime or
explicit executor_config.cap_add can add capabilities;
Atlas adds both.
Normal non-privileged launches receive --ulimit memlock=-1:-1, removing the
soft and hard locked-memory limits. That does not grant IPC_LOCK: a resource
limit and a Linux capability control different things. Recipes and executor
overrides can supply their own ulimit settings.
Recipe example
Section titled “Recipe example”model: Qwen/Qwen3-1.7Bruntime: vllmexecutor: local
executor_config: working_dir: /opt/qwen3 log_dir: /var/log/sparkrun env_file: /etc/sparkrun.env command_prefix: nice -n 10 # prepend to the serve command gpus: "device=0,2" # → CUDA_VISIBLE_DEVICES=0,2
defaults: port: 8000Limitations of the Local executor
Section titled “Limitations of the Local executor”- No images, no volumes, no Ray strategy.
- GPU visibility honors
gpus: allandgpus: device=0,1; anything else logs a warning and leaves visibility to the workload. - Process-group lifecycle is hand-coded (
setsid+kill -- -<pgid>); there is no supervisor, no restart-on-crash.
Status is cross-executor
Section titled “Status is cross-executor”Docker containers and native local workloads are invisible to each other’s
introspection, so sparkrun queries every enabled executor on a cluster and
merges the results. A native workload therefore shows up in sparkrun status
and sparkrun cluster monitor alongside containers, and sparkrun stop --all
tears down both.
For the contributor-facing executor reference (ExecutorConfig.from_chain
field-by-field, EXT_EXECUTOR SAF discovery), see
docs/EXECUTORS.md.
Resolved destinations and native recovery
Section titled “Resolved destinations and native recovery”Per-host executor configuration and destination constraints are part of the plan,
including filesystem/model access and ownership. Native workloads need verified
process identity and recovery records; stale metadata is not authority to adopt,
kill, or delete an unrelated process. See Execution Flow.
Kubernetes requires the k8s plugin to be loaded plus the requested child
capability and lives under sparkrun.plugins.k8s.