inclusionAI/UI-Venus

What it solves

UI-Venus-2 is a general-purpose foundation GUI agent designed to automate interactions across mobile apps, web platforms, and desktop operating systems. It addresses the challenge of creating a unified agent capable of handling diverse environments and complex, long-horizon tasks while maintaining high accuracy and robustness against "reward hacking" during verification.

How it works

The system operates as a closed-loop perception-reasoning-action agent. Key technical improvements in version 2 include:

• Unified Interaction: Integrates mobile, web, and desktop OS capabilities into a single end-to-end model.
• Keypoint-Grounded Verification: Instead of looking at the whole screen, it judges task completion based on specific visual keypoints and uses multi-model voting to reduce bias.
• Verification-Augmented Reflection: Distills verified feedback back into training as supervision, allowing the agent to recover from errors and distinguish partial progress from full completion.
• Scaled Environments: Uses a deep-research-driven query generation strategy to ground tasks in real application functionality across English and Chinese ecosystems.

Who it’s for

• Developers building autonomous GUI agents for cross-platform automation.
• Researchers focusing on multimodal foundation models for operating system and application interaction.
• Users needing a single agent capable of navigating mobile, web, and desktop interfaces.

Highlights

• Cross-Platform: Supports Android (via ADB), Chrome (via a dedicated plugin), and desktop OS.
• Multilingual: Specifically scaled for both Chinese and English app ecosystems.
• Robust Verification: Employs keypoint-grounded verification and multi-model voting to ensure task success.
• Specialized Capabilities: Includes a dedicated benchmark and capability for solving CAPTCHA tasks.