CarWatch: Local AI Agent for Vehicles via Raspberry Pi 5

CarWatch is an open-source project that integrates a local Large Language Model (LLM) into a vehicle using a Raspberry Pi 5, turning the car into a functional chat-room agent. By running a quantized Qwen model locally, the system provides hands-free voice interaction, owner's manual retrieval via RAG, and real-time vehicle telemetry without relying on cloud connectivity for core operations.

Local AI Architecture and Performance

CarWatch leverages a Raspberry Pi 5 (16 GB RAM) to run Qwen3.6-35B-A3B using an Unsloth UD-Q3_K_S dynamic quantization. This setup allows the system to operate entirely offline, ensuring functionality in garages or dead zones where internet access is unavailable.

Key performance metrics on real hardware include:

  • Generation Speed: 3.5 tokens per second.
  • Prompt Processing: 25+ tokens per second.
  • Thermal Performance: Sustained temperature of 65°C (active cooling required).
  • Memory Footprint: The model occupies approximately 14.3 GB of RAM.

Core Functional Capabilities

Grounded Knowledge and RAG

To prevent hallucinations and ensure technical accuracy, CarWatch utilizes a lexical Retrieval-Augmented Generation (RAG) system. The system ships with the car's 745-page owner's manual on the SD card. The LLM is instructed to provide answers based strictly on the manual, including page citations, and is designed to refuse answers that are not contained within the source text.

Real-time Vehicle State Monitoring

The system integrates with the vehicle's hardware to provide grounded self-knowledge. It can read live data regarding:

  • Hardware State: CPU temperature, throttling status, fan speed, memory usage, and disk/network status.
  • Vehicle Telemetry: Through an ethernet-to-OBD (DoIP/ENET) cable, the system is built to monitor RPM, coolant temperature, speed, and voltage (currently verified against a simulated gateway).

Hands-Free Voice Interface

CarWatch implements a continuous listener pipeline on the Pi: Energy VAD → whisper.cpp → Grounded LLM Pipeline → Room Output. This removes the need for wake-word ceremonies and cloud-based speech-to-text (STT), keeping all audio processing local.

Connectivity and Deployment Strategy

CarWatch employs a three-tier connectivity model to balance local utility with remote enrichment:

  1. Always Local: Voice input, LLM answers, RAG retrieval, and the phone dashboard (served by the car) work with zero signal.
  2. Queued Connectivity: Room posts, dashcam clip uploads, and mention replies are stored in a persistent on-disk outbox and delivered once a connection is established.
  3. Online-Only: Remote reachability via a dial-out tunnel (cloudflared), self-updates from GitHub, and optional escalation to cloud models.

Hardware Reference Build

For those replicating the build, the reference hardware includes:

  • SBC: Raspberry Pi 5, 16 GB RAM with active cooling.
  • Audio: Class-compliant USB microphone.
  • Dashcam: WOLFBOX G900 3-channel camera (connected via WiFi AP for event clip retrieval).
  • OBD Access: Ethernet-to-OBD (DoIP/ENET) cable or ELM327-class adapter.
  • Power: 5V/5A USB-C feed via a 12V PD adapter or 230V socket.

Community Insights and Critiques

While the project demonstrates the feasibility of local LLMs in automotive environments, community discussion on Hacker News highlighted several technical and practical concerns:

Accuracy and Safety: Some users warned that LLMs often struggle with highly specific automotive details (e.g., oil weights for specific model years), suggesting that reliance on LLMs for mechanical maintenance could be risky.

Utility and Use Cases: Critics questioned the practical advantage of a voice agent over physical buttons for tasks like adjusting AC or locking doors, suggesting that the project's value lies more in the "agentic" nature of the car (e.g., reporting arrivals/departures to a chat room) than in replacing existing car controls.

Integration Complexity: Questions were raised regarding the difficulty of interfacing with manufacturer APIs and the security implications of creating a "second key" to the vehicle via a Raspberry Pi.

"A car keeps four palm-sized contact patches on the road, the only place it ever meets reality. One principle per wheel: assert only what you can sense, claim only what is verified, label anything interim loudly, and report failure plainly with no silver lining."

— CarWatch development log

Sources

Related