oobabooga/textgen

What it solves

TextGen is a desktop application that allows users to run Large Language Models (LLMs) locally on their own hardware, ensuring 100% privacy and zero telemetry. It provides a comprehensive interface for interacting with models, removing the the complexity of setting up local inference engines and managing model files.

How it works

The app integrates multiple backends to support a wide variety of model formats and hardware acceleration (CUDA, Vulkan, ROCm, and CPU). It supports GGUF (via llama.cpp), Transformers, ExLlamaV3, and TensorRT-LLM. Users can simply download a model file and place it in a designated folder for the app to detect it automatically.

Who it’s for

Privacy-conscious users, AI enthusiasts, and developers who want to run LLMs locally without relying on on-cloud APIs, as well as those who want to perform local fine-tuning (LoRA) or image generation.

Highlights

• Multimodal capabilities: Supports vision models for visual understanding and file attachments (PDF, .docx, text) for document-based chat.
• Flexible interface: Includes an instruct mode for ChatGPT-like interactions, a chat mode for custom characters, and a notebook for free-form generation.
• Local API: Provides an OpenAI/Anthropic-compatible API for use as a local drop-in replacement.
• Tool-calling: Models can execute custom Python functions, including web search and math, and supports MCP servers.
• LORA Training: Built-in support for fine-tuning LoRAs on chat or raw text datasets.
• Image Generation: Dedicated tab for diffusers models with quantization and a persistent gallery.
• Privacy: Completely offline operation with no external resource requests.