Skip to content

proxy

The sparkrun proxy commands manage a pluggable inference gateway. LiteLLM is the bundled default on all channels. To use SparkRoute, install its plugin and activate its gateway capability as shown below.

The proxy sits in front of one or more inference workloads launched by sparkrun and provides:

  • Live endpoint discovery using the same mechanism as sparkrun cluster status
  • Auto-discovery background process or daemon service that periodically re-scans and syncs models
  • Health checking via GET /v1/models on each discovered endpoint
  • Deduplication of endpoints reachable on multiple network interfaces (e.g. management IP vs ConnectX-7 IP)
  • Model aliases so clients can address a model by a friendly name
  • Load/unload models through sparkrun proxy load / sparkrun proxy unload

When LiteLLM is selected, the proxy runs it via uvx --from 'litellm[proxy]==1.82.6' litellm — no permanent installation required.

Install and activate SparkRoute, then start the selected gateway:

Terminal window
# Install the plugin and enable loading
sparkrun plugins add https://github.com/sparksq/sparkrun-sparkroute-plugin --enable
# Enable the gateway capability and select SparkRoute as the default
sparkrun setup features enable gateway.sparkroute
# Start the selected gateway
sparkrun proxy start --host 127.0.0.1

plugins add --enable enables plugin loading and preserves capability settings. The feature-enable command activates gateway.sparkroute and saves SparkRoute as proxy.gateway in proxy.yaml, even if the capability was already enabled. Subsequent starts use that selection without a --gateway argument. If the plugin is already installed but disabled, use sparkrun plugins enable sparkroute before activating its gateway capability.

To return to the bundled LiteLLM gateway:

Terminal window
sparkrun setup features enable gateway.litellm
sparkrun proxy start --restart

Selection does not restart a running gateway. Use --restart when replacing an existing instance, including when first switching to SparkRoute. Other gateway capabilities remain enabled. proxy start --gateway NAME is also available to select and persist a provider whose capability is already enabled.

Management binds to the recorded running implementation. An unavailable selected plugin does not silently switch to LiteLLM; process status/stop remain usable, while model/config operations need the plugin. See Managed Plugins.

proxy ui, proxy admin-token, aliases, and update mechanisms depend on gateway capabilities. Unsupported management fails explicitly. A saved discovery cluster from start --cluster persists; --clear-cluster returns to following the default.

sparkrun proxy ui --json returns the selected gateway’s console information as JSON when that capability is supported.

Terminal window
# Launch some inference workloads first
sparkrun run qwen3-1.7b-vllm --cluster mylab
# Start the proxy (discovers endpoints automatically)
sparkrun proxy start --cluster mylab
# Query models through the unified API
curl http://localhost:4000/v1/models

Or use the proxy to manage the full lifecycle:

Terminal window
# Start the proxy
sparkrun proxy start
# Load a model through the proxy
sparkrun proxy load qwen3.5-0.8b-bf16-sglang
# Query models
curl http://localhost:4000/v1/models

Discovers endpoints, asks the selected gateway to prepare configuration, and launches it. The selected discovery backend periodically re-scans and syncs models with the proxy.

Terminal window
sparkrun proxy start # port 4000
sparkrun proxy start --host 127.0.0.1 # recommended bind address
sparkrun proxy start --port 8080 # custom port
sparkrun proxy start --cluster mylab # discover from cluster hosts
sparkrun proxy start --foreground # run in foreground (blocking)
sparkrun proxy start --master-key sk-mykey # require a bearer token
sparkrun proxy start --no-auto-discover # disable periodic re-scanning
sparkrun proxy start --discover-interval 60 # re-scan every 60s (default: 30)
sparkrun proxy start --restart # replace an already-running proxy
sparkrun proxy start --dry-run # show what would be done

By default the proxy daemonizes in the background. Logs are written to ~/.cache/sparkrun/proxy/litellm.log.

Starting when a proxy is already running is an error; pass --restart to replace it with one carrying the new settings. Explicitly-supplied settings are persisted to proxy.yaml either way.

OptionDescription
--portProxy listen port (default: 4000)
--hostBind address — persisted to proxy.yaml. Recommended: 127.0.0.1.
--master-keyBearer token for stateless LiteLLM auth (default: none)
--hosts / -HComma-separated host list
--hosts-fileFile with hosts (one per line)
--clusterSave a cluster for subsequent discovery
--clear-clusterForget saved discovery cluster and follow the default
--gatewayPersist an explicit gateway selection
--foregroundRun in foreground (blocking)
--no-auto-discoverDisable periodic endpoint re-scanning
--discovery-backend process|daemonSelect standalone or daemon-managed discovery
--discover-intervalSeconds between discovery sweeps (default: 30)
--restartIf a proxy is already running, stop it and start fresh with the new settings
--dry-run / -nShow what would be done without executing

Stops the running proxy and its discovery process. If a systemd unit owns the proxy, this stops that unit; it stays enabled for the next boot.

Terminal window
sparkrun proxy stop

Shows whether the proxy is running, its PID, bind address, auto-discover status, and queries models through the selected gateway. Query failures retain model_query_error; models exits nonzero in text and JSON modes rather than returning a successful empty list.

Terminal window
sparkrun proxy status

Discovery failures appear as Last error, including safe gateway rejection details. Successful imports can instead report Routing warning messages; warnings do not mark the sweep as failed or delay the normal discovery interval. Currently registered models can remain visible even if a later update fails. Resolve the reported configuration issue and use proxy sync to retry.

Re-discover endpoints and apply the desired model list, aliases, and gateway settings. For LiteLLM, changed routing settings trigger a restart even when no models were added or removed.

Terminal window
sparkrun proxy sync
sparkrun proxy sync --json

Text output lists routing warnings separately. JSON includes warnings when present, alongside the model change counts.

Lists models currently registered with the running proxy. With --refresh, re-discovers endpoints and syncs the proxy — adding newly available models and removing stale entries whose backends are no longer healthy.

Terminal window
sparkrun proxy models
sparkrun proxy models --refresh

Launches an inference workload (detached) and registers it with the running proxy.

Unlike plain sparkrun run, proxy load automatically avoids port conflicts. When no --port is specified, it checks the head host over SSH to find the first available port. If the desired port is occupied, it increments until a free port is found:

$ sparkrun proxy load qwen3-1.7b-vllm
# Uses port 8000
$ sparkrun proxy load qwen3.5-35b-a3b-fp8-sglang
# Note: port 8000 in use on 10.24.11.13, using 8001 instead

This is intentionally different from sparkrun run, which uses exactly the port specified (or the recipe default) and fails if it’s occupied. The proxy’s load command is designed for managing multiple concurrent models where automatic port assignment is expected.

OptionDescription
RECIPE_NAMERecipe to load (required)
--hosts / -HComma-separated host list
--hosts-fileFile with hosts (one per line)
--clusterSave a cluster for subsequent discovery
--clear-clusterForget saved discovery cluster and follow the default
--gatewayPersist an explicit gateway selection
--tp / --tensor-parallelOverride tensor parallelism
--pp / --pipeline-parallelOverride pipeline parallelism
--gpu-memOverride GPU memory utilization (0.0–1.0)
--max-model-lenOverride maximum model context length
-o / --optionOverride a default with -o key=value or one mapping field with -o key.field=value (repeatable)
--imageOverride container image
--soloForce single-node mode
--offline / --onlineUse cached assets or override the cluster’s offline launch policy
--trustAuthorize the recipe’s hooks, mods, and trust-gated execution settings
--portOverride serve port
--dry-run / -nShow what would be done without executing

Stops the inference workload containers directly (same logic as sparkrun stop) and syncs the proxy to remove the now-stale model entry.

Terminal window
sparkrun proxy unload qwen3-1.7b-vllm --cluster mylab

Manage model aliases so clients can reference models by friendly names.

Terminal window
sparkrun proxy alias add qwen3-small "Qwen/Qwen3-1.7B"
sparkrun proxy alias remove qwen3-small
sparkrun proxy alias list

An alias is saved to proxy.yaml immediately. If a proxy is running, the config is regenerated and the proxy restarted so the alias takes effect. An alias whose target model has no healthy backend is saved but skipped in the generated config — it starts working as soon as the target is loaded.

On a Linux control machine, install a service to start the proxy at boot:

Terminal window
# Preview the default system unit and its scoped sudoers grant.
sparkrun proxy systemd install --cluster mylab --dry-run
# Install, enable, and start the unit.
sparkrun proxy systemd install --cluster mylab --now
sparkrun proxy systemd status --lines 50
# Change saved settings and restart the installed unit.
sparkrun proxy start --port 4100 --restart
# Stop, disable, and remove the unit and its grant.
sparkrun proxy systemd uninstall

The default system unit runs as your user and requires sudo for installation. It installs a sudoers grant for start/stop/restart of that specific unit. For a user unit, use sparkrun proxy systemd install --user --now. Boot startup then requires logind lingering; the installer attempts to enable it and reports the result.

The unit runs sparkrun proxy start --foreground, reading saved proxy.yaml settings at each start. install --cluster NAME saves a discovery cluster; otherwise discovery uses the default cluster. Foreground supervision follows managed gateway restarts, allowing the service to keep supervising a replacement process.

Once installed, proxy start, proxy start --restart, and proxy stop delegate to the unit, and proxy status identifies its owner. Stopping leaves it enabled; use systemd uninstall to remove boot startup. install --now replaces an existing proxy running outside the unit.

SubcommandOptions
install--user, --cluster NAME, --now, --dry-run
status--system or --user, --lines N (default 20), --json
uninstall--system or --user, --dry-run

--system and --user are mutually exclusive. Status and uninstall detect the installed scope when neither is specified.

  1. Enumerates persisted job metadata (~/.cache/sparkrun/jobs/*.yaml) for every workload sparkrun has launched
  2. Takes a live cluster status snapshot to determine which of those are actually running — the same cross-executor source sparkrun cluster status uses, so native (local) workloads are visible too, not only Docker containers
  3. Normalises host IPs to management IPs (prefers management IPs over InfiniBand IPs)
  4. Performs parallel health checks via GET /v1/models (3-second timeout)
  5. Returns only healthy endpoints

When no host context is available, discovery falls back to metadata only and skips the liveness step.

Select daemon-managed discovery after enabling the daemon feature:

Terminal window
sparkrun setup features enable daemon
sparkrun proxy start --discovery-backend daemon --cluster mylab
sparkrun daemon call proxy.discovery.status

The gateway and discovery watcher run as separate durable service jobs, with their own capacity budget. They share scope, host coverage, and removal-grace behavior with the standalone process backend. See Daemon Mode for service slots, SparkRoute transport selection, credentials, and recovery.

When the proxy starts with auto-discover enabled (the default), a background process runs alongside the proxy:

  • Periodically re-discovers endpoints at the configured interval (default: 30 seconds)
  • Reconciles models and aliases in one pass — they share a single config file, so applying them separately would restart the proxy twice per sweep
  • For LiteLLM, rewrites the config and restarts when models, aliases, or routing settings change
  • Re-reads the proxy PID each sweep, so it follows a restart instead of mistaking it for a shutdown
  • Monitors that PID and exits automatically when the proxy dies

Disable with --no-auto-discover or set auto_discover: false in proxy.yaml.

Persistent proxy settings are stored in ~/.config/sparkrun/proxy.yaml:

proxy:
port: 4000
host: 127.0.0.1 # recommended; unset keeps the legacy 0.0.0.0 + warning
master_key: null # set to require a bearer token (stateless; no DB)
auto_discover: true
discover_interval: 30 # seconds between re-scans
gateway: litellm # optional; pins the gateway implementation
aliases:
my-model: "Qwen/Qwen3-1.7B"
gpt-4: "Qwen/Qwen3-30B-A3B"
gateways:
litellm:
balancing: least-in-flight
router_settings:
num_retries: 2

CLI flags override config file values for a given invocation, and explicitly supplied values are persisted back.

Put settings understood by one gateway under gateways.<name>. Each provider reads only its own section; shared configuration saves preserve these sections. Listener settings, gateway selection, discovery scheduling, and aliases retain their shared locations. The provider interprets its own settings.

SparkRoute uses gateways.sparkroute for config, transport, admin_host, admin_port, allow_insecure_admin_nonloopback, capability_policy, auto_update_recipes, bindings, and discovered_models. Legacy locations are not read or migrated. Recipe auto-updates default to true; set gateways.sparkroute.auto_update_recipes: false and restart the proxy to require manual updates. Updates wait until the previous workload is removed, preserving running or sleeping instances. See daemon transport configuration for a namespaced example.

gateways.litellm.router_settings passes LiteLLM router options through. Changes are applied on the next sync or discovery sweep by regenerating configuration and restarting the proxy.

Without a balancing override, LiteLLM uses simple-shuffle, choosing a random backend per request. If several backends serve the same model, set gateways.litellm.balancing: least-in-flight as in the example above to prefer the backend with the fewest active requests. Ties are random.

  • Counts are shared by a backend’s model names and aliases, but each request’s candidates remain limited to backends serving the requested model.
  • Each request holds its own slot until success or failure, including retries. Concurrent requests sharing a session/trace ID are counted separately. A slot with no completion notification expires after 15 minutes.
  • Retries prefer a backend the request has not tried when one is available. Failed backends are deprioritized for 10 seconds, except for request errors with status 400, 413, or 422.
  • Counts are local to the proxy process; direct traffic to backends is not counted.

Avoid router_settings.routing_strategy: least-busy with Sparkrun’s pinned LiteLLM release: its counter is not updated on the proxy’s async request path. With balancing: least-in-flight, Sparkrun’s balancer controls backend selection and overrides router_settings.routing_strategy, warning if both are supplied. Removing balancing restores LiteLLM routing on the next sync/restart.

LiteLLM is the bundled default on every channel. The litellm plugin, shared gateway capability, and gateway.litellm capability all default on. Explicitly running setup features enable gateway.<provider> selects that provider, even if its flag was already enabled. Other provider flags stay unchanged.

plugins enable permits loading without selecting a gateway; features reset restores a default without selecting one. Capability commands do not enable a disabled parent plugin. Pinning a disabled or unavailable provider in proxy.gateway raises an error. Restart consumers after changing plugin loading or selection.

Disabling gateway rejects new gateway actions. Status, stop, service status, and service removal remain available for recovery. No disable action automatically stops a running service or selects a fallback.

PathPurpose
~/.config/sparkrun/proxy.yamlPersistent proxy settings and aliases
~/.cache/sparkrun/proxy/litellm_config.yamlGenerated LiteLLM config
~/.cache/sparkrun/proxy/state.yamlRunning proxy state (PID, port, host, gateway, auto-discover PID, start time)
~/.cache/sparkrun/proxy/litellm.logProxy process stdout/stderr
~/.cache/sparkrun/proxy/autodiscover.yamlAuto-discover process config
~/.cache/sparkrun/proxy/autodiscover.logAuto-discover process stdout/stderr
~/.cache/sparkrun/jobs/*.yamlJob metadata used for endpoint discovery