amywork777/lipflow

What it solves

Lipflow allows users to dictate text into any application without speaking aloud or using a microphone. By using a webcam to read lip movements, it enables silent communication and text entry in environments where sound is not an option or desired.

How it works

The system captures live video of the user's face via webcam. It uses MediaPipe FaceLandmarker to isolate and crop the mouth area in real-time. These crops are processed by a Visual Speech Recognition (VSR) encoder (based on Auto-AVSR) and a Transformer decoder with beam search to translate lip shapes into text.

To improve accuracy, the project employs several layers of refinement:

• LLM Cleanup: Because many sounds look identical on the lips, an LLM (such as Claude, a local Qwen model via MLX/Ollama, or basic rules) analyzes the top guesses to determine the most likely intended sentence.
• Personalization: The app can import dictation history (e.g., from Wispr Flow) and allow users to record practice sentences to fine-tune the lip reader and language model to their specific face and phrasing.
• Whisper Mode: An optional mode that combines lip reading with soft audio input for significantly higher accuracy.

Who it’s for

People who want to dictate text silently on Mac, Windows, or Linux, and those interested in visual speech recognition technology.

Highlights

• Cross-platform: Supports macOS, Windows, and Linux.
• Local Execution: Runs primarily on-device, with optional local LLM support for cleanup.
• Customizable: Users can define custom words and train the model on their own facial movements.
• Integrated Workflow: Pastes text directly at the cursor position in any active application.