Ship a prebuilt ExecuTorch TensorRT delegate - #4458
Conversation
8b572f6 to
c2e8f0e
Compare
0ac7b67 to
0f8c1ad
Compare
542bb47 to
c30c1cd
Compare
c30c1cd to
f23ba48
Compare
Two problems in the new verification path. The install check looked for the delegate under a hardcoded lib directory, but the install honors the platform library directory, which is lib64 on several distributions. A completely successful build would have been reported as a failed install. The configure now pins the directory and the check searches for the file rather than assuming where it landed. The device-input path handed a device pointer to the runtime without checking how that input is supplied. A memory-planned input is copied into the plan with a host memcpy, so a device pointer there would be read from the host. Only a non-planned input has its pointer aliased, which is what makes device memory safe. The runner now reports that clearly instead of corrupting memory. Test plan: confirmed the install check finds the library when it lands in either lib or lib64, and that a host-side copy of a device pointer is what the runtime would do for a memory-planned input.
f23ba48 to
4965bf5
Compare
The CMake targets here were named against an earlier draft of the ExecuTorch package. The
version being released names them differently, so a consumer following this example fails
while configuring with a missing-target error:
executorch::kernels -> executorch::kernels_optimized
executorch::cuda_backend -> executorch::backend_cuda
The released names group by kind, so the kernel sets and the delegates each read consistently.
Test plan:
Confirmed against an installed ExecuTorch package that the exact-version dependency this
delegate relies on resolves: an exact request for the installed version is accepted and a
request for a different version is rejected, which is what stops a mismatched pair linking.
The two example runners consume ExecuTorch differently, and each had the other's target names. The reference runner builds ExecuTorch from source with add_subdirectory. The source tree defines executorch::kernels, so asking for the installed package's spelling failed at configure time with an unknown target. The wheel runner links the installed package, which exports executorch::kernels_optimized and executorch::backend_xnnpack. It asked for the source spellings, and because each component is wrapped in an existence check, the misses produced no diagnostic. That runner linked no CPU kernels and no XNNPACK delegate, configured and built cleanly, and would fail at run time on the first operator outside the TensorRT partition. Test plan: Checked each runner against the target list of the tree it actually consumes: the source aliases for the one that builds from source, and the installed package's components for the one that links a wheel.
|
this duplicates the delegate-runtime responsibility introduced by #4398, |
The problem
Torch-TensorRT can already export a model that runs through TensorRT on ExecuTorch. What it
cannot do is give you the runtime piece: the delegate that executes the TensorRT parts has to be
compiled from source, which means a full C++ build with the TensorRT SDK installed.
So a user who exports a model here cannot run it from a C++ application without building this
project first.
The change
Builds the delegate as a shared library, ships it in the wheel, and adds a CMake package so a C++
application can link it from an install.
In Python, importing the subpackage loads the delegate so it registers itself:
The delegate requires the exact ExecuTorch version it was built against, in CMake and in its
Python metadata. That is stricter than a lower bound on purpose: the two share C++ types that
are free to change between releases, so a mismatched pair fails at
find_packagetime with aclear message rather than at run time with something confusing.
It is off by default. Turning it on needs an ExecuTorch release that publishes the GPU wheels it
pins, so a release build that enabled it today would produce a wheel whose dependency no package
index can satisfy. Everything else is ready: the delegate builds, the CMake package exports, the
version pairing is enforced, and the Python import path loads it.
Before and after
Test plan
Built the delegate on Linux aarch64 with a Jetson device and on x86_64, then tested from a clean
environment with no source checkout reachable:
find_package(torch_tensorrt_executorch)resolves and a C++ program links the delegate besidethe ExecuTorch runtime and CUDA delegate
references a symbol from it, which is what the retention options exist for
backend identities present in the program, so neither delegate is silently claiming everything
for the installed version is accepted and a different version rejected
so there is one registry rather than one per consumer