NVlabs/Fast-FoundationStereo

What it solves

Fast-FoundationStereo addresses the trade-off between accuracy and speed in stereo matching. While existing foundation models for stereo vision provide strong zero-shot generalization (the ability to work on new data without retraining), they are typically too slow for real-time use. Conversely, efficient architectures often lack robustness and require expensive per-domain fine-tuning. This project provides a family of architectures that maintain high zero-shot accuracy while running at real-time frame rates.

How it works

The project uses a "divide-and-conquer" acceleration strategy consisting of three main technical components:

1. Knowledge Distillation: Compresses a complex hybrid backbone into a single, efficient student model.
2. Blockwise Neural Architecture Search (NAS): Automatically finds the best cost filtering designs based on specific latency budgets to reduce search complexity.
3. Structured Pruning: Removes redundancy from the iterative refinement module.

To support training, the authors developed an automatic pseudo-labeling pipeline that curated 1.4 million in-the-wild stereo pairs from the Stereo4D dataset to supplement synthetic training data.

Who it’s for

This tool is designed for developers and researchers working on real-time depth estimation, robotics, and computer vision applications that require high-quality stereo matching without the need for domain-specific fine-tuning.

Highlights

• Real-Time Performance: Capable of running over 10— faster than the original FoundationStereo.
• Zero-Shot Generalization: Maintains strong accuracy on unseen data without requiring additional training.
• Flexible Trade-offs: Offers multiple checkpoints and configuration flags (like valid_iters and scale) to balance speed versus precision.
• Deployment Ready: Supports export to ONNX and TensorRT (TRT) for optimized GPU inference.