AtomicBot-ai/Atomic-Chat

Atomic Chat – Local AI chat & inference engine

What it is – A desktop (macOS, Windows, Linux) and mobile (iOS, Android) application that lets you run open‑weight LLMs on your own machine. It bundles a small “core” service that starts an OpenAI‑compatible HTTP server at http://localhost:1337/v1. Any tool that talks to the OpenAI API (coding agents, IDE plugins, CLI helpers, etc.) can point at that address and use the locally‑loaded model as if it were a cloud service.

---

Key capabilities (as described in the README)

• Local model support – Hugging Face GGUF models such as Llama, Gemma, Qwen, Mistral, Phi, etc.
• Speed‑up tricks –
  • Multi‑Token Prediction (MTP) speculative decoding (30‑70 % faster, up to 3× on Gemma 4).
  • DFlash block‑diffusion decoding (up to 6× faster on certain models, Apple‑Silicon only).
  • TurboQuant KV‑cache for llama.cpp (≈4.3× smaller memory, works on Windows, Linux, macOS, CPU & GPU).
  • EAGLE‑3 speculative decoding for Gemma 4 on Apple‑Silicon (MLX).
• Engine options – three back‑ends are selectable behind the same API:
  1. atomic-llama-cpp-turboquant (fork of llama.cpp with TurboQuant).
  2. Upstream llama.cpp (default on Windows/Linux, provides MTP).
  3. MLX‑VLM (Apple‑Silicon native, also runs vision‑language models).
• Cloud model fallback – built‑in providers for OpenAI, Anthropic, Mistral, Groq, MiniMax, Qwen, Moonshot; you can mix local and cloud models in the same chat.
• One‑click agent integrations – buttons to launch popular open‑source agents (Atomic Agent, Claude Code, OpenCode, Goose, Kilo Code, etc.) that automatically use the local server.
• Privacy‑first – the server binds to 127.0.0.1 by default; no API keys are needed for local models, and all conversation data stays on your device.
• API compatibility – identical request shape to the OpenAI chat/completions endpoint; works with curl, the official openai Python client, or any IDE plugin that expects an OpenAI endpoint.

---

Getting started (quick‑start)

1. Download the appropriate binary from the Releases page (macOS DMG, Windows EXE, Linux AppImage) or install the mobile app from the App Store / Play Store.
2. Launch the app, add a model from Hugging Face (the UI will download the GGUF file for you).
3. Once the model is loaded, the app automatically starts the local OpenAI‑compatible server on port 1337.
4. Point any client at http://localhost:1337/v1 – e.g.:

   curl http://localhost:1337/v1/chat/completions \
        -H "Content-Type: application/json" \
        -d '{"model":"gemma-2b-it","messages":[{"role":"user","content":"Say hello"}]}'
   

   or in Python:

   from openai import OpenAI
   client = OpenAI(base_url="http://localhost:1337/v1", api_key="not-needed")
   resp = client.chat.completions.create(model="gemma-2b-it", messages=[{"role":"user","content":"Say hello"}])
   print(resp.choices[0].message.content)
   

---

Installation from source (for developers)

• Prerequisites: Node ≥ 20, Yarn ≥ 4.5, Make, Rust (for Tauri), and on Apple Silicon the Metal toolchain.
• Build:

  git clone https://github.com/AtomicBot-ai/Atomic-Chat
  cd Atomic-Chat
  make dev   # installs deps, builds core, launches the app
  

  Available make targets: dev, build, test, clean.
• The heavy lifting (model download, engine launch, API routing) lives in the separate repo atomic‑chat‑core, which can also be run as a standalone CLI (atomic-chat-core serve …).

---

System requirements

OS	Minimum hardware
macOS 13.6+	8 GB RAM for 3 B models, 16 GB for 7 B, 32 GB for 13 B
Windows 10/11 x64	Same RAM guidelines
Linux x86_64 (glibc ≥ 2.35)	Same RAM; optional Vulkan driver for GPU acceleration
iOS / Android	App store versions

---

Who might use this?

• Privacy‑concerned developers who want to experiment with LLMs without sending data to the cloud.
• Tool builders (coding assistants, IDE plugins, autonomous agents) that already speak the OpenAI API – they can instantly switch to a local model by changing the base URL.
• Researchers / hobbyists looking for a ready‑made UI and API layer on top of llama.cpp, MLX‑VLM, or other open‑weight models.

---

Community & support

• Discord: https://discord.com/invite/8wGSsvmg4V
• GitHub issues for bug reports.
• Contributing: PRs accepted in both the UI repo (this one) and the core repo.

---

TL;DR

Atomic Chat is a cross‑platform GUI that turns open‑weight LLMs into a local OpenAI‑compatible server, offering several speed‑optimising inference back‑ends and out‑of‑the‑box integrations with popular open‑source AI agents. It’s aimed at anyone who wants private, on‑device LLM inference with a familiar API.