awwaiid/ghostwriter

What it solves

Ghostwriter enables a reMarkable tablet to act as an interactive AI canvas. It allows users to write or draw prompts by hand on the screen and receive responses from a Vision-LLM that are written back to the device as either typewritten text or SVG drawings.

How it works

The software runs as a background process on the reMarkable device. When a user triggers the assistant (by default, by tapping the upper-right corner of the screen), the system takes a screenshot of the current page and sends it to a Vision-LLM (such as GPT-4o, Claude 3.5 Sonnet, or Gemini 2.0 Flash).

The model then decides how to respond using specific tools:

• Text Output: Uses a virtual keyboard to type responses into the reMarkable's text layer.
• SVG Drawing: Generates SVG code that the system rasterizes and draws as dots on the screen.
• Spatial Awareness: An optional image segmentation step can be used to help the model understand the coordinates of elements on the screen for more accurate placement of answers.

Who it’s for

Enthusiasts and developers using reMarkable 2 or reMarkable Paper Pro tablets who want to integrate generative AI directly into their handwriting workflow.

Highlights

• Multi-Model Support: Compatible with OpenAI, Anthropic, and Google Gemini engines, as well as local models via Groq.
• Hybrid Output: Can switch between drawing line-art and typing text based on the prompt.
• Customizable Triggers: Supports configurable touch-trigger corners for activation.
• Spatial Intelligence: Includes an image segmentation feature to improve the model's ability to place responses in specific locations (e.g., filling in a box).
• Advanced LLM Features: Supports Anthropic's "thinking" mode and web search capabilities.