Skip to content

Diagnostics

Diagnostics collect configuration, host information, operation feedback, and bounded workload log tails into a versioned NDJSON file. Collection is opt-in, does not change host configuration, and never uploads the file automatically.

  • Use sparkrun setup diagnose to inspect hosts without launching a workload.
  • Use sparkrun run --collect-diagnostics FILE to capture planning, launch, readiness, and failure evidence around a run.
  • Applications and daemon clients can collect a saved workload or capture a run or benchmark through the same Diagnostics API.
Terminal window
# Diagnose all hosts in a cluster
sparkrun setup diagnose --cluster mylab
# Diagnose a saved subgroup
sparkrun setup diagnose --cluster pair-a
# Custom output file
sparkrun setup diagnose --cluster mylab -o my_diagnostics.ndjson
# Include sudo diagnostics (dmidecode: BIOS, system board, memory details)
sparkrun setup diagnose --cluster mylab --sudo
# Print JSON summary to stdout
sparkrun setup diagnose --cluster mylab --json

By default, the output file is named spark_diag_<timestamp>.ndjson.

The diagnostic script runs on each host over SSH (no elevated privileges required for the base collection) and gathers:

CategoryDetails
HardwareCPU model, cores, threads, RAM total/available, disk usage (root and home), GPU name, memory, driver, power state, temperature, serial, UUID
FirmwareHostname, OS name/version, kernel, architecture, BIOS version, board name, product name, JetPack version, CUDA version
NetworkAll interfaces (name, state, MTU, MAC, speed, IP), default interface, management IP
DockerDocker version, storage driver, root directory, NVIDIA runtime status, running container count, sparkrun container count
Firmware updatesDevice inventory and update history from fwupdmgr (if available)

With --sudo, additional DMI data is collected via dmidecode:

CategoryDetails
DMIBIOS vendor/version/date, system manufacturer/product/version/serial/UUID, board manufacturer/product/version/serial, memory slots/populated/max

The hidden --collect-diagnostics option captures the CLI lifecycle:

Terminal window
sparkrun run my-recipe --cluster mylab --collect-diagnostics run_diag.ndjson

Capture includes planned placement, configuration summaries, recipe and launch results, phase timings, operation logs, structured feedback, and exceptions. Failed operations attempt to retain workload log tails. Plugins can contribute bounded diagnostic sections; OCI Relay records source/route decisions, failure evidence before receiver cleanup, and transfer/cleanup outcomes.

Only logs attributed to the captured operation are included. Capture admits DEBUG and higher logs independently of CLI verbosity; -vvv is not required and console output retains its normal threshold. Explicit named-logger thresholds and logging.disable() still apply. Host probes are deduplicated across nested planning and launch calls. Workload log tails default to 200 lines per source.

For Python or daemon benchmarks, capture saves log tails before cleanup of an owned inference workload, including after successful measurements. That preserves logs if later export or publication fails. There is no separate benchmark CLI capture flag; use Python capture or daemon capture.

Records include _type, _seq, and _ts. Consumers should ignore unfamiliar record types and plugin sections. Useful records include:

RecordMeaning
diag_headerBundle metadata and schema_version: 1
host_hardware, host_firmware, host_network, host_dockerHost observations
host_firmware_updates, host_dmi, host_errorOptional firmware/DMI data or probe errors
config_sparkrun, config_clusters, config_registriesConfiguration summaries
logPython logs attributed to this operation
plugin_diagnosticPlugin-owned section and versioned payload under data
plugin_diagnostics_metadata, plugin_diagnostics_summaryPlugin provenance and bounded omission/error counts
bundle_sectionSection status: ok, error, skipped, or cancelled
bundle_completeFinal partial, truncated, and operation_success values

A missing bundle_complete means capture did not finish. A partial bundle can still contain useful evidence. Skipped probes, read failures, or truncation do not by themselves mean the workload failed. Native Kubernetes uses resource-aware workload log readers; SSH/Linux host probes are explicitly skipped. Kubernetes events, manifests, and node-level collectors are outside this version.

Terminal window
# Pretty-print records from one bundle
jq . run_diag.ndjson
# Inspect completion and section failures
jq 'select(._type == "bundle_complete" or ._type == "bundle_section")' run_diag.ndjson
# Inspect plugin evidence
jq 'select(._type == "plugin_diagnostic")' run_diag.ndjson

The default bundle budget is 16 MiB; oversized records and excess output are bounded, with space reserved for completion. Python and daemon callers can set max_bytes and log_tail. Files are created privately and exclusively: an existing output path is never overwritten or appended to. Cancellation stops new probes and still attempts to finalize the bundle. setup diagnose --dry-run previews collection without remote probes or log reads.

All records pass through the shared redactor, including plugin payloads, logs, and errors. It removes known credentials and secret patterns, but arbitrary text can still contain unrecognized secrets. Hostnames, hardware identity, paths, and configuration summaries remain useful troubleshooting data. Review the file before attaching it to a support request.

Interpreting preparation and startup failures

Section titled “Interpreting preparation and startup failures”

Start with read-only setup check --cluster mylab --json for applicable host prerequisites and duplicate-identity advisories. Then use model-cache inspection to distinguish missing inventory, damaged files, broken shared-blob links and access failures before targeted repair. A full transfer counter does not attest to final target usability.

Default run readiness failures can leave detached workloads available for inspection. Docker-start port-open/HTTP-ready/TTFT observations overlap; do not sum them or confuse them with total CLI duration. See run and Troubleshooting.

Diagnostic capture, live workload logs, operation feedback, monitoring, and anonymous usage telemetry serve different purposes. See Diagnostics API for library integration, limits, and daemon downloads, and Log Streaming API for reusable live log readers.