foldl/chatllm.cpp

What it solves

ChatLLM.cpp provides a high-performance, memory-efficient way to run large language models (LLMs) locally on a computer's CPU or GPU. It enables real-time multimodal chat and Retrieval Augmented Generation (RAG) without relying on cloud services, supporting models ranging from under 1 billion to over 300 billion parameters.

How it works

Built as a pure C++ implementation based on the ggml library, the project uses int4/int8 quantization to reduce memory usage and accelerate inference. It employs an object-oriented approach to handle various Transformer-based models and optimizes performance through a parallel computing and an optimized KV cache. It also includes a conversion tool (convert.py) to transform models from Hugging Face format into a custom quantized GGML format.

Who it’s for

It is designed for users who want to run a wide variety of LLMs locally with low hardware requirements, as well as developers looking for a C++ based inference engine with bindings for Python, JavaScript, C, and Nim.

Highlights

• Multimodal Support: Capable of handling text, images (vision), audio (speech), and TTS (text-to-speech).
• RAG Integration: Built-in support for Retrieval Augmented Generation.
• Distributed Inference: Supports RPC for distributed computing.
• Hardware Acceleration: Includes GPU acceleration and Vulkan support.
• Flexible Quantization: Allows for customized quantization per tensor using regex patterns.
• Broad Model Support: Compatible with a vast array of models including Llama, Gemma, Qwen, and InternVL.