AtomicBot-ai/Atomic-Chat

What it solves

Atomic Chat provides a private, local environment to run open-weight Large Language Models (LLMs) on a user's own machine, eliminating the need to rely on cloud providers for every interaction. It also serves as a local inference engine that other AI agents and tools can use via an API.

How it works

The application integrates three different inference engines—a custom TurboQuant fork of llama.cpp, the official llama.cpp, and MLX-VLM for Apple Silicon—to execute models from HuggingFace. It exposes an OpenAI-compatible server at http://localhost:1337/v1, allowing any tool that supports the OpenAI SDK to use the local models as a backend. It also supports cloud model providers for users who wish to mix local and cloud AI.

Who it’s for

It is designed for users who want total privacy for their AI conversations, developers building AI agents who need a local inference backend, and enthusiasts who want to optimize LLM performance on their specific hardware (macOS, Windows, Linux, iOS, and Android).

Highlights

• High-Performance Decoding: Supports Multi-Token Prediction (MTP), DFlash block-diffusion, and Flash Attention to significantly boost throughput.
• Memory Optimization: Features TurboQuant KV cache to reduce memory footprint on CPU and GPU.
• Agent Integration: One-click launch for various agents (e.g., Claude Code, Cline, OpenHands) and support for the Model Context Protocol (MCP).
• Versatile Deployment: Available as a desktop app for macOS, Windows, and Linux, as well as mobile apps for iOS and Android.
• Developer Friendly: Provides a drop-in OpenAI-compatible API for easy integration with existing AI tools.