Skip to content

TensorRT-LLM

TensorRT-LLM is NVIDIA’s high-performance inference engine with TensorRT optimizations. sparkrun’s trtllm runtime uses MPI (OpenMPI) for multi-node tensor parallelism. Solo mode (tp=1) uses the standard sleep-infinity + exec pattern.

  • You want NVIDIA’s TensorRT-LLM inference engine with TensorRT optimizations
  • You need high-throughput serving with features like FP8 KV cache, CUDA graphs, and MoE backends
  • Your model is supported by TensorRT-LLM’s PyTorch or TensorRT backend

Default image prefix: nvcr.io/nvidia/tensorrt-llm/release

sparkrun resolves container tags from the recipe. For example, container: nvcr.io/nvidia/tensorrt-llm/release:0.18.0 pulls the 0.18.0 release image from NVIDIA’s NGC registry.

model: nvidia/Llama-3.1-8B-Instruct-FP8
runtime: trtllm
container: nvcr.io/nvidia/tensorrt-llm/release:0.18.0
defaults:
port: 8000
host: 0.0.0.0
tensor_parallel: 1
backend: pytorch

Use runtime and defaults and omit command for the supported settings below. Add a command only when you need an unusual engine flag that sparkrun does not map. These are the flags generated by sparkrun; the selected engine image must also support them.

Set values under defaults, for example port: 8000, or override them with -o port=9000. Unset values are omitted unless a runtime/platform default supplies them. Boolean toggles emit a bare flag when true and omit it when false, except where the table lists an explicit negative or valued form. Mapping/list values are rendered as one JSON argument for flags that accept JSON. Unknown keys do not automatically become engine flags.

The generated command is trtllm-serve MODEL; model is a top-level recipe field. The backend defaults to pytorch when unset. Sparkrun wraps multi-node launches in MPI and provides coordination settings.

Mappings: sparkrun 0.4.0 source.

Engine flagKey under defaultsValue form
--portportValue
--hosthostValue
--tp_sizetensor_parallelValue
--pp_sizepipeline_parallelValue
--ep_sizeexpert_parallelValue
--max_num_tokensmax_num_tokensValue
--max_batch_sizemax_batch_sizeValue
--max_seq_lenmax_model_lenValue
--backendbackendValue
--tokenizertokenizerValue
--kv_cache_free_gpu_memory_fractionkv_cache_free_gpu_memory_fractionValue
--trust_remote_codetrust_remote_codeBoolean toggle; true emits the flag

Certain advanced options are passed via a generated extra-llm-api-config.yml file rather than CLI flags. When any of these keys appear in effective recipe defaults or -o overrides, sparkrun generates the YAML file, writes it into the container, and passes it via --extra_llm_api_options. This also applies to single-node launches when the recipe itself declares none of these keys:

Recipe keyConfig sectionYAML field
free_gpu_memory_fractionkv_cache_configfree_gpu_memory_fraction
kv_cache_dtypekv_cache_configdtype
kv_cache_enable_block_reusekv_cache_configenable_block_reuse
cuda_graph_paddingcuda_graph_configenable_padding
cuda_graph_max_batch_sizecuda_graph_configmax_batch_size
moe_backendmoe_configbackend
print_iter_log(top-level)print_iter_log

Example recipe using extra config keys:

model: some/moe-model
runtime: trtllm
container: nvcr.io/nvidia/tensorrt-llm/release:0.18.0
defaults:
port: 8000
host: 0.0.0.0
tensor_parallel: 2
backend: pytorch
free_gpu_memory_fraction: 0.85
kv_cache_dtype: fp8_e5m2
cuda_graph_padding: true
moe_backend: cutlass

This generates an extra-llm-api-config.yml like:

kv_cache_config:
free_gpu_memory_fraction: 0.85
dtype: fp8_e5m2
cuda_graph_config:
enable_padding: true
moe_config:
backend: cutlass

TRT-LLM uses MPI for multi-node inference. sparkrun handles the full orchestration automatically — container lifecycle, InfiniBand detection, MPI configuration, and environment setup:

Terminal window
sparkrun run my-trtllm-recipe --tp 2
  • SSH keys at ~/.ssh on the control machine (mounted into containers)
  • All hosts must be SSH-accessible from each other (use sparkrun setup ssh to configure)

TRT-LLM containers automatically get these extra Docker options for large memory allocations:

  • --ulimit memlock=-1
  • --ulimit stack=67108864

See Execution Flow for the full MPI orchestration details, including the rsh wrapper mechanism and environment variables.

sparkrun auto-detects TRT-LLM recipes by checking the command template. If the command starts with trtllm-serve or mpirun...trtllm, the trtllm runtime is selected automatically.

TensorRT-LLM does not declare streaming inference readiness observers. Default run uses supported endpoint checks and waits for required hooks, without reporting unavailable inference timings. See run.

Memory values in these recipes are explicit engine settings, separate from scheduling reservations. Runtime compilation/autotuning cache reuse is also separate from model-file validation; see Model Cache Management before pruning either cache.