TRI Stereo Depth

TRI's learned stereo depth model is an optional depth backend for ZED cameras in rd convert. It produces high-quality depth maps tailored for robot manipulation scenes and supports two model variants:

Variant	Flag	Description
c64	--tri-stereo-variant c64	Higher quality (default)
c32	--tri-stereo-variant c32	Faster, lighter

For each variant, Raiden automatically selects the fastest available inference backend:

1. TensorRT — fastest; requires a pre-compiled .engine file
2. ONNX Runtime — no compilation needed; requires a .onnx file

Prerequisites

1. Pull models via Git LFS

The ONNX models are tracked in this repository via Git LFS under weights/tri_stereo/. If you haven't already, install Git LFS and pull the files:

git lfs install   # one-time setup per machine
git lfs pull

2. Install the tri-stereo extra

uv sync --extra tri-stereo

This installs onnxruntime-gpu for the ONNX backend.

Usage

rd convert --stereo-method tri_stereo
rd convert --stereo-method tri_stereo --tri-stereo-variant c32

Raiden uses the ONNX backend by default. If a TensorRT engine is present it is used automatically instead.

TensorRT compilation

Compile ONNX models to TensorRT FP16 engines for fastest inference. This step runs once per machine and takes a few minutes.

1. Install TensorRT

Add the TensorRT apt repository following the local repo installation guide, then install:

sudo apt-get install --no-install-recommends libnvinfer-bin libnvinfer-dev libnvinfer-headers-dev
uv sync --extra tri-stereo --extra tri-stereo-trt-cu12   # CUDA 12
uv sync --extra tri-stereo --extra tri-stereo-trt-cu13   # CUDA 13

# with ZED cameras
uv sync --extra zed --extra tri-stereo --extra tri-stereo-trt-cu12

Note

sudo apt-get install tensorrt fails on Python 3.11+ systems due to a dependency conflict. See FAQ for details.

Verify both are available:

trtexec --help
uv run python -c "import tensorrt; print(tensorrt.__version__)"

2. Compile engines:

uv run python scripts/tri_stereo_tensorrt_build.py                    # c64 (default)
uv run python scripts/tri_stereo_tensorrt_build.py --variant c32

Once the engine files are present, rd convert --stereo-method tri_stereo automatically uses TensorRT. You will see [TRIStereo-C64] Loaded TRT engine in the output when TensorRT is active.

Engines are machine-specific

TensorRT engines are compiled for the specific GPU and driver version on the build machine. Recompile after upgrading drivers, CUDA, or TensorRT, or when moving to a different GPU.

Weights directory layout

weights/tri_stereo/       # Git LFS — all model files live here
    stereo_c64.onnx       # ONNX model (c64)       — distributed via Git LFS
    stereo_c64.onnx.data  # ONNX external data (c64)
    stereo_c32.onnx       # ONNX model (c32)
    stereo_c32.onnx.data  # ONNX external data (c32)
    stereo_c64.engine     # TensorRT engine (c64)  — generated by tri_stereo_tensorrt_build.py
    stereo_c32.engine     # TensorRT engine (c32)

Acknowledgments

TRI Stereo Depth is developed at Toyota Research Institute. For more details, see the project page.

Citation

If you use TRI Stereo Depth, please cite:

@article{shankar2022learned,
  title={A learned stereo depth system for robotic manipulation in homes},
  author={Shankar, Krishna and Tjersland, Mark and Ma, Jeremy and Stone, Kevin and Bajracharya, Max},
  journal={IEEE Robotics and Automation Letters},
  volume={7},
  number={2},
  pages={2305--2312},
  year={2022},
  publisher={IEEE}
}