TurixAI/TuriX-CUA

What it solves

TuriX provides a way for AI models to interact directly with a computer's desktop interface. It eliminates the need for app-specific APIs by allowing an AI agent to perform actions (like clicking and typing) on any software a human can use, such as web browsers, office suites, and internal company tools.

How it works

The system uses a Vision Language Model (VLM) as a "brain" to perceive the screen and decide on actions. It can be configured with different models (including local ones via Ollama) and supports a multi-agent architecture where different models can handle planning, acting, and memory. It integrates with the Model Context Protocol (MCP) to allow other agents, like Claude for Desktop, to trigger desktop actions. Additionally, it uses "Skills"—markdown-based playbooks—that provide the agent with specific instructions for complex tasks.

Who it’s for

• Developers and researchers looking for an open-source computer-use agent.
• Power users who want to automate repetitive desktop workflows across multiple applications.
• AI engineers wanting to test different VLM policies for GUI automation.

Highlights

• High Success Rate: Achieves 64.2% on the OSWorld benchmark and over 80% on a macOS-specific benchmark.
• API-Independent: Operates on the GUI, meaning it works with any application that doesn't have a public API.
• Hot-Swappable Models: Users can change the underlying AI model via a simple configuration file without changing code.
• Cross-Platform Support: Available for macOS, Windows, and Linux.
• MCP Integration: Compatible with the Model Context Protocol for seamless connection to other AI agents.