Pollen-Vision: Unified Interface for Zero-Shot Vision Models in Robotics

Pollen-vision is an open-source library designed to provide robots with the autonomy to grasp unknown objects by offering a curated collection of zero-shot vision models. It enables the creation of 3D object detection pipelines that can determine the spatial coordinates (x, y, z) of objects without requiring prior training.

Core Vision Models in Pollen-Vision

To ensure versatility and real-time capability on consumer GPUs, the library integrates three primary zero-shot models:

  • OWL-ViT (Open World Localization - Vision Transformer): Developed by Google Research, this model performs text-conditioned 2D object localization in RGB images, producing bounding boxes.
  • Mobile SAM (Segment Anything Model): A lightweight version of Meta AI's SAM, this model provides zero-shot image segmentation when prompted with bounding boxes or points.
  • RAM (Recognize Anything Model): Created by the OPPO Research Institute, RAM performs zero-shot image tagging to identify the presence of objects based on textual descriptions, which helps optimize the performance of other models.

3D Object Detection Pipeline and Implementation

pollen-vision allows developers to combine independent modules into a pipeline that converts 2D image data into 3D spatial coordinates.

From 2D Detection to Segmentation

Using the library, a pipeline can be established where OWL-ViT detects an object (e.g., "paper cups") and generates bounding boxes, which are then passed to Mobile SAM to create a precise segmentation mask.

Calculating 3D Position

To translate a 2D image into a 3D coordinate, the library utilizes the following process:

  1. Centroid Estimation: The (u, v) position in pixel space is calculated by computing the centroid of the binary segmentation mask. Using a mask instead of a bounding box prevents background pixels from skewing depth averages.
  2. Depth Integration: By combining the pixel coordinates with depth information (in meters) and the camera's intrinsic matrix (K), the system calculates the (x, y, z) position of the object's centroid in the camera's reference frame.
  3. Robot Frame Transformation: If the camera's position relative to the robot's origin is known, the coordinates are transformed into the robot's frame, allowing the end effector to move to the object's location for grasping.

Performance and Optimization

Inference time for OWL-ViT varies based on the number of prompts provided. Testing on a laptop with an RTX 3070 GPU showed the following latency:

  • 1 prompt: ~75ms per frame
  • 2 prompts: ~130ms per frame
  • 5 prompts: ~330ms per frame
  • 10 prompts: ~650ms per frame

Because of this linear increase in latency, the RAM model is used to identify only the objects present in an image before prompting OWL-ViT, thereby optimizing processing speed.

Current Limitations and Future Roadmap

While pollen-vision establishes a foundation for basic robotic manipulation, the Pollen Robotics team has identified several areas for improvement:

  • Detection Consistency: OWL-ViT can be inconsistent and may not detect all objects; the team is seeking superior alternatives.
  • Temporal and Spatial Consistency: Currently, all data is recomputed every frame. The team is working on integrating point tracking to improve consistency.
  • Grasping Sophistication: Current capabilities are limited to front grasping. Future work will focus on 6D detection and improved grasping pose generation.
  • System Speed: There are opportunities to improve the overall execution speed of the pipeline.

Sources