moritzkremb/jev-voice-browser

What it solves

This project provides a way to control a web browser using voice commands in real-time. It allows users to navigate websites, search for information, click elements, and manage tabs hands-free, with the ability to act on commands even before the user has finished speaking the entire sentence.

How it works

The system uses a Node.js server that controls a headed Chromium window via Playwright. It streams speech from the browser's Web Speech API to the server, which then sends the transcript to the Jev model (by TypeSafe).

Instead of generating text, the model answers a series of typed questions (e.g., "Is this a command?", "What is the intent?") to determine the action. The server maintains a state of the current page, a snapshot of available elements, and a history of recent actions to resolve context-dependent commands like "no, the other one." A policy engine then decides whether to act, wait for more speech, or ask for confirmation for destructive actions.

Who it’s for

Users looking for a voice-controlled web browsing experience, developers interested in LLM-driven browser automation, and those exploring low-latency intent recognition for real-time interfaces.

Highlights

• Real-time execution: Acts on partial transcripts to reduce perceived latency.
• Context-aware: Remembers previous pages and actions to handle corrections and relative references.
• Zero-text generation: Uses typed probabilities rather than generative text to ensure reliability and verbatim extraction of search queries and URLs.
• Safety gates: Requires spoken confirmation for destructive actions (like placing an order).
• Disambiguation: Provides numbered overlays on the webpage when the model is unsure which element to click.