Diagnostics
Diagnostics collect configuration, host information, operation feedback, and bounded workload log tails into a versioned NDJSON file. Collection is opt-in, does not change host configuration, and never uploads the file automatically.
- Use
sparkrun setup diagnoseto inspect hosts without launching a workload. - Use
sparkrun run --collect-diagnostics FILEto capture planning, launch, readiness, and failure evidence around a run. - Applications and daemon clients can collect a saved workload or capture a run or benchmark through the same Diagnostics API.
Host diagnostics
Section titled “Host diagnostics”# Diagnose all hosts in a clustersparkrun setup diagnose --cluster mylab
# Diagnose a saved subgroupsparkrun setup diagnose --cluster pair-a
# Custom output filesparkrun setup diagnose --cluster mylab -o my_diagnostics.ndjson
# Include sudo diagnostics (dmidecode: BIOS, system board, memory details)sparkrun setup diagnose --cluster mylab --sudo
# Print JSON summary to stdoutsparkrun setup diagnose --cluster mylab --jsonBy default, the output file is named spark_diag_<timestamp>.ndjson.
What it collects
Section titled “What it collects”The diagnostic script runs on each host over SSH (no elevated privileges required for the base collection) and gathers:
| Category | Details |
|---|---|
| Hardware | CPU model, cores, threads, RAM total/available, disk usage (root and home), GPU name, memory, driver, power state, temperature, serial, UUID |
| Firmware | Hostname, OS name/version, kernel, architecture, BIOS version, board name, product name, JetPack version, CUDA version |
| Network | All interfaces (name, state, MTU, MAC, speed, IP), default interface, management IP |
| Docker | Docker version, storage driver, root directory, NVIDIA runtime status, running container count, sparkrun container count |
| Firmware updates | Device inventory and update history from fwupdmgr (if available) |
With --sudo, additional DMI data is collected via dmidecode:
| Category | Details |
|---|---|
| DMI | BIOS vendor/version/date, system manufacturer/product/version/serial/UUID, board manufacturer/product/version/serial, memory slots/populated/max |
Run diagnostics
Section titled “Run diagnostics”The hidden --collect-diagnostics option captures the CLI lifecycle:
sparkrun run my-recipe --cluster mylab --collect-diagnostics run_diag.ndjsonCapture includes planned placement, configuration summaries, recipe and launch results, phase timings, operation logs, structured feedback, and exceptions. Failed operations attempt to retain workload log tails. Plugins can contribute bounded diagnostic sections; OCI Relay records source/route decisions, failure evidence before receiver cleanup, and transfer/cleanup outcomes.
Only logs attributed to the captured operation are included. Capture admits DEBUG
and higher logs independently of CLI verbosity; -vvv is not required and console
output retains its normal threshold. Explicit named-logger thresholds and
logging.disable() still apply. Host probes are deduplicated across nested planning
and launch calls. Workload log tails default to 200 lines per source.
For Python or daemon benchmarks, capture saves log tails before cleanup of an owned inference workload, including after successful measurements. That preserves logs if later export or publication fails. There is no separate benchmark CLI capture flag; use Python capture or daemon capture.
Read a bundle
Section titled “Read a bundle”Records include _type, _seq, and _ts. Consumers should ignore unfamiliar
record types and plugin sections. Useful records include:
| Record | Meaning |
|---|---|
diag_header | Bundle metadata and schema_version: 1 |
host_hardware, host_firmware, host_network, host_docker | Host observations |
host_firmware_updates, host_dmi, host_error | Optional firmware/DMI data or probe errors |
config_sparkrun, config_clusters, config_registries | Configuration summaries |
log | Python logs attributed to this operation |
plugin_diagnostic | Plugin-owned section and versioned payload under data |
plugin_diagnostics_metadata, plugin_diagnostics_summary | Plugin provenance and bounded omission/error counts |
bundle_section | Section status: ok, error, skipped, or cancelled |
bundle_complete | Final partial, truncated, and operation_success values |
A missing bundle_complete means capture did not finish. A partial bundle can
still contain useful evidence. Skipped probes, read failures, or truncation do
not by themselves mean the workload failed. Native Kubernetes uses resource-aware
workload log readers; SSH/Linux host probes are explicitly skipped. Kubernetes
events, manifests, and node-level collectors are outside this version.
# Pretty-print records from one bundlejq . run_diag.ndjson
# Inspect completion and section failuresjq 'select(._type == "bundle_complete" or ._type == "bundle_section")' run_diag.ndjson
# Inspect plugin evidencejq 'select(._type == "plugin_diagnostic")' run_diag.ndjsonFile limits and sharing
Section titled “File limits and sharing”The default bundle budget is 16 MiB; oversized records and excess output are
bounded, with space reserved for completion. Python and daemon callers can set
max_bytes and log_tail. Files are created privately and exclusively: an
existing output path is never overwritten or appended to. Cancellation stops
new probes and still attempts to finalize the bundle. setup diagnose --dry-run
previews collection without remote probes or log reads.
All records pass through the shared redactor, including plugin payloads, logs, and errors. It removes known credentials and secret patterns, but arbitrary text can still contain unrecognized secrets. Hostnames, hardware identity, paths, and configuration summaries remain useful troubleshooting data. Review the file before attaching it to a support request.
Interpreting preparation and startup failures
Section titled “Interpreting preparation and startup failures”Start with read-only setup check --cluster mylab --json for applicable host
prerequisites and duplicate-identity advisories. Then use
model-cache inspection to distinguish missing inventory,
damaged files, broken shared-blob links and access failures before targeted repair.
A full transfer counter does not attest to final target usability.
Default run readiness failures can leave detached workloads available for
inspection. Docker-start port-open/HTTP-ready/TTFT observations overlap; do not
sum them or confuse them with total CLI duration. See run and
Troubleshooting.
Diagnostic capture, live workload logs, operation feedback, monitoring, and anonymous usage telemetry serve different purposes. See Diagnostics API for library integration, limits, and daemon downloads, and Log Streaming API for reusable live log readers.