Skip to content

Executors

An executor is the layer that actually launches your workload on a host.

SelectorStatusFeature flagWhat it does
dockerStableexecutor.docker (on)docker run per host. The default and the production-supported path.
localExperimentalexecutor.local (off)Native subprocess via setsid. No container. PID/log files under ~/.cache/sparkrun/local/.

Recipes select an executor via the top-level executor: field; per-executor configuration lives under executor_config:. See Recipe format › Executor fields for the full field reference.

  • Docker (default) — every shipped DGX Spark recipe, every multi-node cluster runtime that’s been validated end-to-end, every benchmark profile. Stay on Docker unless you have a specific reason to leave.
  • Local — bare-metal experimentation when you don’t want a container in the loop, on a supported Linux target. Portable controllers do not qualify Apple GPU execution. It uses the host environment or an explicitly prepared environment. Multi-host native cluster runtimes (vllm with distributed/mp, and sglang) work; Ray-based runtimes do not.

Which executor is picked from the first layer that names one, highest priority first:

  1. CLI override (-o executor=local).
  2. Recipe (executor:).
  3. Cluster pin (sparkrun cluster create … --executor local).
  4. Runtime default (runtime.default_executor()).
  5. SparkrunConfig.default_executor in config.yaml.
  6. The baseline default — docker, unless it has been disabled, in which case the sole remaining enabled executor is used.

Its configuration (executor_config:) is layered separately, again highest priority first: CLI → recipe → runtime default → per-executor runtime adjustments → SparkrunConfig.executor_config → per-executor defaults (Docker ships DOCKER_DEFAULTS; Local ships none) → dataclass field defaults.

If a layer names an executor that is unknown or gated off, resolution raises ExecutorUnavailableError naming the flag to enable. It never falls back to Docker.

Normal sparkrun Docker launches are non-privileged and include no-new-privileges. Seccomp policy, Linux capabilities, and locked-memory limits are separate settings.

In 0.4.0, sparkrun defaults to seccomp=io-uring: a bundled, pinned copy of Moby’s default profile with an allow rule for io_uring_setup, io_uring_enter, and io_uring_register. The default deny action, architecture mappings, and other rules remain intact. Atlas uses this shared policy.

Select a profile through executor_config.security_opt in application config, a cluster, a recipe, or API overrides:

SettingBehavior
seccomp=io-uringsparkrun’s default profile with the three io_uring syscalls allowed
seccomp=builtinDocker’s own default profile
seccomp=/path/to/profile.jsonA custom profile loaded from the controller
seccomp=unconfinedExplicitly disables seccomp filtering

These selections are interpreted by sparkrun; io-uring is not a Docker built-in profile name. If no seccomp option is supplied, including an empty security-options list, sparkrun selects io-uring. Explicitly privileged containers remain subject to Docker’s privileged-container behavior.

For a custom policy:

executor_config:
security_opt:
- no-new-privileges
- seccomp=/home/operator/policies/inference.json

The path is local to the controller running sparkrun. ~ expands to that user’s home, and relative paths resolve against its working directory. sparkrun validates and snapshots the policy before replacing a workload, then carries its contents in each node’s launch script. The nodes do not need a copy at the same path. Docker validates target-specific policy support when creating the container. Custom profiles are used as supplied; include any io_uring permissions your workload needs. Preview generation reads and validates the local file without writing to remote hosts.

The shared Docker defaults do not add IPC_LOCK or SYS_NICE. A runtime or explicit executor_config.cap_add can add capabilities; Atlas adds both.

Normal non-privileged launches receive --ulimit memlock=-1:-1, removing the soft and hard locked-memory limits. That does not grant IPC_LOCK: a resource limit and a Linux capability control different things. Recipes and executor overrides can supply their own ulimit settings.

model: Qwen/Qwen3-1.7B
runtime: vllm
executor: local
executor_config:
working_dir: /opt/qwen3
log_dir: /var/log/sparkrun
env_file: /etc/sparkrun.env
command_prefix: nice -n 10 # prepend to the serve command
gpus: "device=0,2" # → CUDA_VISIBLE_DEVICES=0,2
defaults:
port: 8000
  • No images, no volumes, no Ray strategy.
  • GPU visibility honors gpus: all and gpus: device=0,1; anything else logs a warning and leaves visibility to the workload.
  • Process-group lifecycle is hand-coded (setsid + kill -- -<pgid>); there is no supervisor, no restart-on-crash.

Docker containers and native local workloads are invisible to each other’s introspection, so sparkrun queries every enabled executor on a cluster and merges the results. A native workload therefore shows up in sparkrun status and sparkrun cluster monitor alongside containers, and sparkrun stop --all tears down both.

For the contributor-facing executor reference (ExecutorConfig.from_chain field-by-field, EXT_EXECUTOR SAF discovery), see docs/EXECUTORS.md.

Per-host executor configuration and destination constraints are part of the plan, including filesystem/model access and ownership. Native workloads need verified process identity and recovery records; stale metadata is not authority to adopt, kill, or delete an unrelated process. See Execution Flow. Kubernetes requires the k8s plugin to be loaded plus the requested child capability and lives under sparkrun.plugins.k8s.