volotat/mini-AGI

What it solves

mini-AGI is designed to overcome the limitations of traditional large language models (LLMs) that are frozen after training. It solves the problem of "catastrophic forgetting," where a model loses previous knowledge when trained on new data. It also addresses the hardware barrier, allowing a user to train a language model from scratch on a single consumer-grade GPU (8 GB VRAM) by moving weights from disk to VRAM as needed.

How it works

The project implements a byte-level language model with a unique architecture that allows for continual learning:

• Paging System: To fit on 8 GB VRAM, the model stores experts (weights and Adam moments) as files on disk and pages them into VRAM (the working set) based on the model's prediction of what the upcoming text requires.

• Recurrent Block with Experts: Instead of a fixed stack of layers, it uses two dense prelude blocks and a recurrent block applied up to 24 times. Each single application of the block picks top-8 experts from a shared pool.

• Adaptive Depth: A halting head determines when a character stops processing; simple characters take fewer rows of computation, while complex ones take more.

• Continual Learning Mechanism: To prevent forgetting, the model uses a significantly lower learning rate for the "trunk" (embeddings, attention, routers, and halting head) compared to the the experts. This confines updates to the specific experts selected by the routing, preserving the rest of the model.

• Byte-Level Processing: It operates on 256 byte values, eliminating the need for a tokenizer and allowing it to read any file type.

Who it’s for

Developers and AI researchers interested in in-situ training and continual learning who have access to a modest hardware setup (PC or laptop with at least 8 GB VRAM GPU).

Highlights

• Continual Learning: Capable of learning from a single stream of data without catastrophic forgetting.
• Disk-Based Weight Storage: Parameter count is limited by disk space rather than VRAM.
• Self-Assembling Architecture: The model grows new experts via recombination of existing ones and prunes unused experts.
• End-to-End Training: Training and reading are the same process, with no separate fine-tuning regime.
• Direct File Reading: Users can point the model at their own files or directories to teach it specific knowledge.