proxy
The sparkrun proxy commands manage a pluggable inference gateway. LiteLLM is the bundled default on all channels. To use SparkRoute, install its plugin and activate its gateway capability as shown below.
Overview
Section titled “Overview”The proxy sits in front of one or more inference workloads launched by sparkrun and provides:
- Live endpoint discovery using the same mechanism as
sparkrun cluster status - Auto-discovery background process or daemon service that periodically re-scans and syncs models
- Health checking via
GET /v1/modelson each discovered endpoint - Deduplication of endpoints reachable on multiple network interfaces (e.g. management IP vs ConnectX-7 IP)
- Model aliases so clients can address a model by a friendly name
- Load/unload models through
sparkrun proxy load/sparkrun proxy unload
LiteLLM implementation
Section titled “LiteLLM implementation”When LiteLLM is selected, the proxy runs it via uvx --from 'litellm[proxy]==1.82.6' litellm — no
permanent installation required.
Choose a gateway
Section titled “Choose a gateway”Install and activate SparkRoute, then start the selected gateway:
# Install the plugin and enable loadingsparkrun plugins add https://github.com/sparksq/sparkrun-sparkroute-plugin --enable
# Enable the gateway capability and select SparkRoute as the defaultsparkrun setup features enable gateway.sparkroute
# Start the selected gatewaysparkrun proxy start --host 127.0.0.1plugins add --enable enables plugin loading and preserves capability settings.
The feature-enable command activates gateway.sparkroute and saves SparkRoute
as proxy.gateway in proxy.yaml, even if the capability was already enabled.
Subsequent starts use that selection without a --gateway argument. If the
plugin is already installed but disabled, use sparkrun plugins enable sparkroute
before activating its gateway capability.
To return to the bundled LiteLLM gateway:
sparkrun setup features enable gateway.litellmsparkrun proxy start --restartSelection does not restart a running gateway. Use --restart when replacing an
existing instance, including when first switching to SparkRoute. Other gateway
capabilities remain enabled. proxy start --gateway NAME is also available to
select and persist a provider whose capability is already enabled.
Management binds to the recorded running implementation. An unavailable selected plugin does not silently switch to LiteLLM; process status/stop remain usable, while model/config operations need the plugin. See Managed Plugins.
proxy ui, proxy admin-token, aliases, and update mechanisms depend on gateway
capabilities. Unsupported management fails explicitly. A saved discovery cluster
from start --cluster persists; --clear-cluster returns to following the default.
sparkrun proxy ui --json returns the selected gateway’s console information as
JSON when that capability is supported.
Quick start
Section titled “Quick start”# Launch some inference workloads firstsparkrun run qwen3-1.7b-vllm --cluster mylab
# Start the proxy (discovers endpoints automatically)sparkrun proxy start --cluster mylab
# Query models through the unified APIcurl http://localhost:4000/v1/modelsOr use the proxy to manage the full lifecycle:
# Start the proxysparkrun proxy start
# Load a model through the proxysparkrun proxy load qwen3.5-0.8b-bf16-sglang
# Query modelscurl http://localhost:4000/v1/modelsCommands
Section titled “Commands”sparkrun proxy start
Section titled “sparkrun proxy start”Discovers endpoints, asks the selected gateway to prepare configuration, and launches it. The selected discovery backend periodically re-scans and syncs models with the proxy.
sparkrun proxy start # port 4000sparkrun proxy start --host 127.0.0.1 # recommended bind addresssparkrun proxy start --port 8080 # custom portsparkrun proxy start --cluster mylab # discover from cluster hostssparkrun proxy start --foreground # run in foreground (blocking)sparkrun proxy start --master-key sk-mykey # require a bearer tokensparkrun proxy start --no-auto-discover # disable periodic re-scanningsparkrun proxy start --discover-interval 60 # re-scan every 60s (default: 30)sparkrun proxy start --restart # replace an already-running proxysparkrun proxy start --dry-run # show what would be doneBy default the proxy daemonizes in the background. Logs are written to ~/.cache/sparkrun/proxy/litellm.log.
Starting when a proxy is already running is an error; pass --restart to
replace it with one carrying the new settings. Explicitly-supplied settings are
persisted to proxy.yaml either way.
Options
Section titled “Options”| Option | Description |
|---|---|
--port | Proxy listen port (default: 4000) |
--host | Bind address — persisted to proxy.yaml. Recommended: 127.0.0.1. |
--master-key | Bearer token for stateless LiteLLM auth (default: none) |
--hosts / -H | Comma-separated host list |
--hosts-file | File with hosts (one per line) |
--cluster | Save a cluster for subsequent discovery |
--clear-cluster | Forget saved discovery cluster and follow the default |
--gateway | Persist an explicit gateway selection |
--foreground | Run in foreground (blocking) |
--no-auto-discover | Disable periodic endpoint re-scanning |
--discovery-backend process|daemon | Select standalone or daemon-managed discovery |
--discover-interval | Seconds between discovery sweeps (default: 30) |
--restart | If a proxy is already running, stop it and start fresh with the new settings |
--dry-run / -n | Show what would be done without executing |
sparkrun proxy stop
Section titled “sparkrun proxy stop”Stops the running proxy and its discovery process. If a systemd unit owns the proxy, this stops that unit; it stays enabled for the next boot.
sparkrun proxy stopsparkrun proxy status
Section titled “sparkrun proxy status”Shows whether the proxy is running, its PID, bind address, auto-discover status, and queries models through the selected gateway. Query failures retain model_query_error; models exits nonzero in text and JSON modes rather than returning a successful empty list.
sparkrun proxy statusDiscovery failures appear as Last error, including safe gateway rejection
details. Successful imports can instead report Routing warning messages;
warnings do not mark the sweep as failed or delay the normal discovery interval.
Currently registered models can remain visible even if a later update fails.
Resolve the reported configuration issue and use proxy sync to retry.
sparkrun proxy sync
Section titled “sparkrun proxy sync”Re-discover endpoints and apply the desired model list, aliases, and gateway settings. For LiteLLM, changed routing settings trigger a restart even when no models were added or removed.
sparkrun proxy syncsparkrun proxy sync --jsonText output lists routing warnings separately. JSON includes warnings when
present, alongside the model change counts.
sparkrun proxy models
Section titled “sparkrun proxy models”Lists models currently registered with the running proxy. With --refresh, re-discovers endpoints and syncs the proxy — adding newly available models and removing stale entries whose backends are no longer healthy.
sparkrun proxy modelssparkrun proxy models --refreshsparkrun proxy load <recipe>
Section titled “sparkrun proxy load <recipe>”Launches an inference workload (detached) and registers it with the running proxy.
Unlike plain sparkrun run, proxy load automatically avoids port conflicts. When no --port is specified, it checks the head host over SSH to find the first available port. If the desired port is occupied, it increments until a free port is found:
$ sparkrun proxy load qwen3-1.7b-vllm# Uses port 8000
$ sparkrun proxy load qwen3.5-35b-a3b-fp8-sglang# Note: port 8000 in use on 10.24.11.13, using 8001 insteadThis is intentionally different from sparkrun run, which uses exactly the port specified (or the recipe default) and fails if it’s occupied. The proxy’s load command is designed for managing multiple concurrent models where automatic port assignment is expected.
Options
Section titled “Options”| Option | Description |
|---|---|
RECIPE_NAME | Recipe to load (required) |
--hosts / -H | Comma-separated host list |
--hosts-file | File with hosts (one per line) |
--cluster | Save a cluster for subsequent discovery |
--clear-cluster | Forget saved discovery cluster and follow the default |
--gateway | Persist an explicit gateway selection |
--tp / --tensor-parallel | Override tensor parallelism |
--pp / --pipeline-parallel | Override pipeline parallelism |
--gpu-mem | Override GPU memory utilization (0.0–1.0) |
--max-model-len | Override maximum model context length |
-o / --option | Override a default with -o key=value or one mapping field with -o key.field=value (repeatable) |
--image | Override container image |
--solo | Force single-node mode |
--offline / --online | Use cached assets or override the cluster’s offline launch policy |
--trust | Authorize the recipe’s hooks, mods, and trust-gated execution settings |
--port | Override serve port |
--dry-run / -n | Show what would be done without executing |
sparkrun proxy unload <recipe>
Section titled “sparkrun proxy unload <recipe>”Stops the inference workload containers directly (same logic as sparkrun stop) and syncs the proxy to remove the now-stale model entry.
sparkrun proxy unload qwen3-1.7b-vllm --cluster mylabsparkrun proxy alias
Section titled “sparkrun proxy alias”Manage model aliases so clients can reference models by friendly names.
sparkrun proxy alias add qwen3-small "Qwen/Qwen3-1.7B"sparkrun proxy alias remove qwen3-smallsparkrun proxy alias listAn alias is saved to proxy.yaml immediately. If a proxy is running, the
config is regenerated and the proxy restarted so the alias takes effect. An
alias whose target model has no healthy backend is saved but skipped in the
generated config — it starts working as soon as the target is loaded.
Run as a systemd service
Section titled “Run as a systemd service”On a Linux control machine, install a service to start the proxy at boot:
# Preview the default system unit and its scoped sudoers grant.sparkrun proxy systemd install --cluster mylab --dry-run
# Install, enable, and start the unit.sparkrun proxy systemd install --cluster mylab --nowsparkrun proxy systemd status --lines 50
# Change saved settings and restart the installed unit.sparkrun proxy start --port 4100 --restart
# Stop, disable, and remove the unit and its grant.sparkrun proxy systemd uninstallThe default system unit runs as your user and requires sudo for installation.
It installs a sudoers grant for start/stop/restart of that specific unit.
For a user unit, use sparkrun proxy systemd install --user --now. Boot startup
then requires logind lingering; the installer attempts to enable it and reports
the result.
The unit runs sparkrun proxy start --foreground, reading saved proxy.yaml
settings at each start. install --cluster NAME saves a discovery cluster;
otherwise discovery uses the default cluster. Foreground supervision follows
managed gateway restarts, allowing the service to keep supervising a replacement
process.
Once installed, proxy start, proxy start --restart, and proxy stop delegate
to the unit, and proxy status identifies its owner. Stopping leaves it enabled;
use systemd uninstall to remove boot startup. install --now replaces an
existing proxy running outside the unit.
| Subcommand | Options |
|---|---|
install | --user, --cluster NAME, --now, --dry-run |
status | --system or --user, --lines N (default 20), --json |
uninstall | --system or --user, --dry-run |
--system and --user are mutually exclusive. Status and uninstall detect the
installed scope when neither is specified.
How discovery works
Section titled “How discovery works”- Enumerates persisted job metadata (
~/.cache/sparkrun/jobs/*.yaml) for every workload sparkrun has launched - Takes a live cluster status snapshot to determine which of those are actually running — the same cross-executor source
sparkrun cluster statususes, so native (local) workloads are visible too, not only Docker containers - Normalises host IPs to management IPs (prefers management IPs over InfiniBand IPs)
- Performs parallel health checks via
GET /v1/models(3-second timeout) - Returns only healthy endpoints
When no host context is available, discovery falls back to metadata only and skips the liveness step.
Daemon discovery
Section titled “Daemon discovery”Select daemon-managed discovery after enabling the daemon feature:
sparkrun setup features enable daemonsparkrun proxy start --discovery-backend daemon --cluster mylabsparkrun daemon call proxy.discovery.statusThe gateway and discovery watcher run as separate durable service jobs, with their own capacity budget. They share scope, host coverage, and removal-grace behavior with the standalone process backend. See Daemon Mode for service slots, SparkRoute transport selection, credentials, and recovery.
Auto-discovery
Section titled “Auto-discovery”When the proxy starts with auto-discover enabled (the default), a background process runs alongside the proxy:
- Periodically re-discovers endpoints at the configured interval (default: 30 seconds)
- Reconciles models and aliases in one pass — they share a single config file, so applying them separately would restart the proxy twice per sweep
- For LiteLLM, rewrites the config and restarts when models, aliases, or routing settings change
- Re-reads the proxy PID each sweep, so it follows a restart instead of mistaking it for a shutdown
- Monitors that PID and exits automatically when the proxy dies
Disable with --no-auto-discover or set auto_discover: false in proxy.yaml.
Configuration
Section titled “Configuration”Persistent proxy settings are stored in ~/.config/sparkrun/proxy.yaml:
proxy: port: 4000 host: 127.0.0.1 # recommended; unset keeps the legacy 0.0.0.0 + warning master_key: null # set to require a bearer token (stateless; no DB) auto_discover: true discover_interval: 30 # seconds between re-scans gateway: litellm # optional; pins the gateway implementation
aliases: my-model: "Qwen/Qwen3-1.7B" gpt-4: "Qwen/Qwen3-30B-A3B"
gateways: litellm: balancing: least-in-flight router_settings: num_retries: 2CLI flags override config file values for a given invocation, and explicitly supplied values are persisted back.
Gateway-specific settings
Section titled “Gateway-specific settings”Put settings understood by one gateway under gateways.<name>. Each provider
reads only its own section; shared configuration saves preserve these sections.
Listener settings, gateway selection, discovery scheduling, and aliases retain
their shared locations. The provider interprets its own settings.
SparkRoute uses gateways.sparkroute for config, transport, admin_host,
admin_port, allow_insecure_admin_nonloopback, capability_policy,
auto_update_recipes, bindings, and discovered_models. Legacy locations are
not read or migrated. Recipe auto-updates default to true; set
gateways.sparkroute.auto_update_recipes: false and restart the proxy to require
manual updates. Updates wait until the previous workload is removed, preserving
running or sleeping instances. See
daemon transport configuration
for a namespaced example.
LiteLLM routing and balancing
Section titled “LiteLLM routing and balancing”gateways.litellm.router_settings passes LiteLLM router options through. Changes
are applied on the next sync or discovery sweep by regenerating configuration
and restarting the proxy.
Without a balancing override, LiteLLM uses simple-shuffle, choosing a random
backend per request. If several backends serve the same model, set
gateways.litellm.balancing: least-in-flight as in the example above to prefer
the backend with the fewest active requests. Ties are random.
- Counts are shared by a backend’s model names and aliases, but each request’s candidates remain limited to backends serving the requested model.
- Each request holds its own slot until success or failure, including retries. Concurrent requests sharing a session/trace ID are counted separately. A slot with no completion notification expires after 15 minutes.
- Retries prefer a backend the request has not tried when one is available. Failed backends are deprioritized for 10 seconds, except for request errors with status 400, 413, or 422.
- Counts are local to the proxy process; direct traffic to backends is not counted.
Avoid router_settings.routing_strategy: least-busy with Sparkrun’s pinned
LiteLLM release: its counter is not updated on the proxy’s async request path.
With balancing: least-in-flight, Sparkrun’s balancer controls backend selection
and overrides router_settings.routing_strategy, warning if both are supplied.
Removing balancing restores LiteLLM routing on the next sync/restart.
Gateway selection
Section titled “Gateway selection”LiteLLM is the bundled default on every channel. The litellm plugin, shared
gateway capability, and gateway.litellm capability all default on. Explicitly
running setup features enable gateway.<provider> selects that provider, even
if its flag was already enabled. Other provider flags stay unchanged.
plugins enable permits loading without selecting a gateway; features reset
restores a default without selecting one. Capability commands do not enable a
disabled parent plugin. Pinning a disabled or unavailable provider in
proxy.gateway raises an error. Restart consumers after changing plugin loading
or selection.
Disabling gateway rejects new gateway actions. Status, stop, service status, and
service removal remain available for recovery. No disable action automatically
stops a running service or selects a fallback.
State files
Section titled “State files”| Path | Purpose |
|---|---|
~/.config/sparkrun/proxy.yaml | Persistent proxy settings and aliases |
~/.cache/sparkrun/proxy/litellm_config.yaml | Generated LiteLLM config |
~/.cache/sparkrun/proxy/state.yaml | Running proxy state (PID, port, host, gateway, auto-discover PID, start time) |
~/.cache/sparkrun/proxy/litellm.log | Proxy process stdout/stderr |
~/.cache/sparkrun/proxy/autodiscover.yaml | Auto-discover process config |
~/.cache/sparkrun/proxy/autodiscover.log | Auto-discover process stdout/stderr |
~/.cache/sparkrun/jobs/*.yaml | Job metadata used for endpoint discovery |