Skip to content

Ship a prebuilt ExecuTorch TensorRT delegate - #4458

Draft
shoumikhin wants to merge 3 commits into
pytorch:mainfrom
shoumikhin:executorch-wheel-consumable
Draft

Ship a prebuilt ExecuTorch TensorRT delegate#4458
shoumikhin wants to merge 3 commits into
pytorch:mainfrom
shoumikhin:executorch-wheel-consumable

Conversation

@shoumikhin

@shoumikhin shoumikhin commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

The problem

Torch-TensorRT can already export a model that runs through TensorRT on ExecuTorch. What it
cannot do is give you the runtime piece: the delegate that executes the TensorRT parts has to be
compiled from source, which means a full C++ build with the TensorRT SDK installed.

So a user who exports a model here cannot run it from a C++ application without building this
project first.

The change

Builds the delegate as a shared library, ships it in the wheel, and adds a CMake package so a C++
application can link it from an install.

torch_tensorrt/lib/libexecutorch_trt_backend.so    the delegate
torch_tensorrt/share/cmake/                        find_package support
find_package(executorch REQUIRED)
find_package(torch_tensorrt_executorch REQUIRED)
target_link_libraries(my_app PRIVATE
  executorch::runtime
  executorch::backend_cuda
  torchtrt::executorch_backend
)

In Python, importing the subpackage loads the delegate so it registers itself:

import torch_tensorrt.executorch   # loads and registers the delegate

The delegate requires the exact ExecuTorch version it was built against, in CMake and in its
Python metadata. That is stricter than a lower bound on purpose: the two share C++ types that
are free to change between releases, so a mismatched pair fails at find_package time with a
clear message rather than at run time with something confusing.

It is off by default. Turning it on needs an ExecuTorch release that publishes the GPU wheels it
pins, so a release build that enabled it today would produce a wheel whose dependency no package
index can satisfy. Everything else is ready: the delegate builds, the CMake package exports, the
version pairing is enforced, and the Python import path loads it.

Before and after

BEFORE                                  AFTER

export a model through TensorRT         export a model through TensorRT
  works                                   works

run it from a C++ application          find_package(torch_tensorrt_executorch)
  build the delegate from source        target_link_libraries(app PRIVATE
  first, with the TensorRT SDK            torchtrt::executorch_backend)

Test plan

Built the delegate on Linux aarch64 with a Jetson device and on x86_64, then tested from a clean
environment with no source checkout reachable:

  • find_package(torch_tensorrt_executorch) resolves and a C++ program links the delegate beside
    the ExecuTorch runtime and CUDA delegate
  • the delegate survives the link and registers itself, even though nothing in the application
    references a symbol from it, which is what the retention options exist for
  • a model whose graph splits across both delegates runs and matches the eager model, with both
    backend identities present in the program, so neither delegate is silently claiming everything
  • the exact-version dependency resolves against an installed ExecuTorch package: an exact request
    for the installed version is accepted and a different version rejected
  • importing the Python subpackage registers the delegate in the same process the C++ side uses,
    so there is one registry rather than one per consumer

@meta-cla meta-cla Bot added the cla signed label Aug 1, 2026
@github-actions github-actions Bot added component: tests Issues re: Tests component: api [C++] Issues re: C++ API labels Aug 1, 2026
@shoumikhin
shoumikhin force-pushed the executorch-wheel-consumable branch 10 times, most recently from 8b572f6 to c2e8f0e Compare August 2, 2026 22:47
@github-actions github-actions Bot added the component: build system Issues re: Build system label Aug 2, 2026
@shoumikhin
shoumikhin force-pushed the executorch-wheel-consumable branch 16 times, most recently from 0ac7b67 to 0f8c1ad Compare August 3, 2026 16:37
@shoumikhin
shoumikhin force-pushed the executorch-wheel-consumable branch 5 times, most recently from 542bb47 to c30c1cd Compare August 4, 2026 16:59
@github-actions github-actions Bot added the component: api [Python] Issues re: Python API label Aug 4, 2026
@shoumikhin
shoumikhin force-pushed the executorch-wheel-consumable branch from c30c1cd to f23ba48 Compare August 5, 2026 04:13
Two problems in the new verification path.

The install check looked for the delegate under a hardcoded lib directory, but the
install honors the platform library directory, which is lib64 on several
distributions. A completely successful build would have been reported as a failed
install. The configure now pins the directory and the check searches for the file
rather than assuming where it landed.

The device-input path handed a device pointer to the runtime without checking how
that input is supplied. A memory-planned input is copied into the plan with a host
memcpy, so a device pointer there would be read from the host. Only a non-planned
input has its pointer aliased, which is what makes device memory safe. The runner
now reports that clearly instead of corrupting memory.

Test plan: confirmed the install check finds the library when it lands in either
lib or lib64, and that a host-side copy of a device pointer is what the runtime
would do for a memory-planned input.
@shoumikhin
shoumikhin force-pushed the executorch-wheel-consumable branch from f23ba48 to 4965bf5 Compare August 5, 2026 16:30
The CMake targets here were named against an earlier draft of the ExecuTorch package. The
version being released names them differently, so a consumer following this example fails
while configuring with a missing-target error:

    executorch::kernels        ->  executorch::kernels_optimized
    executorch::cuda_backend   ->  executorch::backend_cuda

The released names group by kind, so the kernel sets and the delegates each read consistently.

Test plan:

Confirmed against an installed ExecuTorch package that the exact-version dependency this
delegate relies on resolves: an exact request for the installed version is accepted and a
request for a different version is rejected, which is what stops a mismatched pair linking.
@shoumikhin shoumikhin changed the title Let a C++ application use the ExecuTorch TensorRT delegate from an install Ship a prebuilt ExecuTorch TensorRT delegate Aug 7, 2026
The two example runners consume ExecuTorch differently, and each had the other's target names.

The reference runner builds ExecuTorch from source with add_subdirectory. The source tree defines
executorch::kernels, so asking for the installed package's spelling failed at configure time with
an unknown target.

The wheel runner links the installed package, which exports executorch::kernels_optimized and
executorch::backend_xnnpack. It asked for the source spellings, and because each component is
wrapped in an existence check, the misses produced no diagnostic. That runner linked no CPU kernels
and no XNNPACK delegate, configured and built cleanly, and would fail at run time on the first
operator outside the TensorRT partition.

Test plan:

Checked each runner against the target list of the tree it actually consumes: the source aliases for
the one that builds from source, and the installed package's components for the one that links a
wheel.
@lanluo-nvidia

Copy link
Copy Markdown
Collaborator

this duplicates the delegate-runtime responsibility introduced by #4398,
but with a different distribution layout,
import/registration mechanism, CI path
Please rebase onto #4398 after it merges,
then either close this PR in favor of the runtime-wheel implementation or extract only the pieces that are still needed
and move them into the dedicated runtime package.
Also one of the major reason we ship a seperate delegate-runtime is that for the users which does not use executorch load feature, they do not need to install the delegate-runtime wheel, and it is a requirement from DLFW, they don't want introduce executorch dependency in the DLFW container image and they already complain that bundle the executorch related .so into our torch_tensorrt wheel which make it too big.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla signed component: api [C++] Issues re: C++ API component: api [Python] Issues re: Python API component: build system Issues re: Build system component: tests Issues re: Tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants