TensorRT-LLM
TensorRT-LLM is NVIDIA’s high-performance inference engine with TensorRT optimizations. sparkrun’s trtllm runtime uses MPI (OpenMPI) for multi-node tensor parallelism. Solo mode (tp=1) uses the standard sleep-infinity + exec pattern.
When to use
Section titled “When to use”- You want NVIDIA’s TensorRT-LLM inference engine with TensorRT optimizations
- You need high-throughput serving with features like FP8 KV cache, CUDA graphs, and MoE backends
- Your model is supported by TensorRT-LLM’s PyTorch or TensorRT backend
Container images
Section titled “Container images”Default image prefix: nvcr.io/nvidia/tensorrt-llm/release
sparkrun resolves container tags from the recipe. For example, container: nvcr.io/nvidia/tensorrt-llm/release:0.18.0 pulls the 0.18.0 release image from NVIDIA’s NGC registry.
Example recipe
Section titled “Example recipe”model: nvidia/Llama-3.1-8B-Instruct-FP8runtime: trtllmcontainer: nvcr.io/nvidia/tensorrt-llm/release:0.18.0defaults: port: 8000 host: 0.0.0.0 tensor_parallel: 1 backend: pytorchConfiguration options
Section titled “Configuration options”Use runtime and defaults and omit command for the supported settings
below. Add a command only when you need an unusual engine flag that sparkrun
does not map. These are the flags generated by sparkrun; the selected engine
image must also support them.
Set values under defaults, for example port: 8000, or override them with
-o port=9000. Unset values are omitted unless a runtime/platform default
supplies them. Boolean toggles emit a bare flag when true and omit it when false,
except where the table lists an explicit negative or valued form. Mapping/list
values are rendered as one JSON argument for flags that accept JSON. Unknown
keys do not automatically become engine flags.
The generated command is trtllm-serve MODEL; model is a top-level recipe
field. The backend defaults to pytorch when unset. Sparkrun wraps multi-node
launches in MPI and provides coordination settings.
Mappings: sparkrun 0.4.0 source.
| Engine flag | Key under defaults | Value form |
|---|---|---|
--port | port | Value |
--host | host | Value |
--tp_size | tensor_parallel | Value |
--pp_size | pipeline_parallel | Value |
--ep_size | expert_parallel | Value |
--max_num_tokens | max_num_tokens | Value |
--max_batch_size | max_batch_size | Value |
--max_seq_len | max_model_len | Value |
--backend | backend | Value |
--tokenizer | tokenizer | Value |
--kv_cache_free_gpu_memory_fraction | kv_cache_free_gpu_memory_fraction | Value |
--trust_remote_code | trust_remote_code | Boolean toggle; true emits the flag |
Extra LLM API config
Section titled “Extra LLM API config”Certain advanced options are passed via a generated extra-llm-api-config.yml file rather than CLI flags. When any of these keys appear in effective recipe defaults or -o overrides, sparkrun generates the YAML file, writes it into the container, and passes it via --extra_llm_api_options. This also applies to single-node launches when the recipe itself declares none of these keys:
| Recipe key | Config section | YAML field |
|---|---|---|
free_gpu_memory_fraction | kv_cache_config | free_gpu_memory_fraction |
kv_cache_dtype | kv_cache_config | dtype |
kv_cache_enable_block_reuse | kv_cache_config | enable_block_reuse |
cuda_graph_padding | cuda_graph_config | enable_padding |
cuda_graph_max_batch_size | cuda_graph_config | max_batch_size |
moe_backend | moe_config | backend |
print_iter_log | (top-level) | print_iter_log |
Example recipe using extra config keys:
model: some/moe-modelruntime: trtllmcontainer: nvcr.io/nvidia/tensorrt-llm/release:0.18.0defaults: port: 8000 host: 0.0.0.0 tensor_parallel: 2 backend: pytorch free_gpu_memory_fraction: 0.85 kv_cache_dtype: fp8_e5m2 cuda_graph_padding: true moe_backend: cutlassThis generates an extra-llm-api-config.yml like:
kv_cache_config: free_gpu_memory_fraction: 0.85 dtype: fp8_e5m2cuda_graph_config: enable_padding: truemoe_config: backend: cutlassMulti-node usage
Section titled “Multi-node usage”TRT-LLM uses MPI for multi-node inference. sparkrun handles the full orchestration automatically — container lifecycle, InfiniBand detection, MPI configuration, and environment setup:
sparkrun run my-trtllm-recipe --tp 2Requirements
Section titled “Requirements”- SSH keys at
~/.sshon the control machine (mounted into containers) - All hosts must be SSH-accessible from each other (use
sparkrun setup sshto configure)
Docker options
Section titled “Docker options”TRT-LLM containers automatically get these extra Docker options for large memory allocations:
--ulimit memlock=-1--ulimit stack=67108864
See Execution Flow for the full MPI orchestration details, including the rsh wrapper mechanism and environment variables.
Auto-detection
Section titled “Auto-detection”sparkrun auto-detects TRT-LLM recipes by checking the command template. If the command starts with trtllm-serve or mpirun...trtllm, the trtllm runtime is selected automatically.
Startup and cache policy
Section titled “Startup and cache policy”TensorRT-LLM does not declare streaming inference readiness observers. Default
run uses supported endpoint checks and waits for required hooks, without
reporting unavailable inference timings. See run.
Memory values in these recipes are explicit engine settings, separate from scheduling reservations. Runtime compilation/autotuning cache reuse is also separate from model-file validation; see Model Cache Management before pruning either cache.