volotat/mini-AGI
Continual learning model trained from scratch on 8GB VRAM laptop with batch-1 stream of data.
What it solves
This project addresses the limitation that most language models are "frozen" after training, preventing them from learning continuously from new data without forgetting what they already know. It also solves the hardware barrier by allowing a model to grow its knowledge capacity based on available disk space rather than being strictly limited by the size of your GPU's video memory (VRAM).
How it works
Instead of loading an entire model into memory, this system stores its "experts" (small neural network blocks) as individual files on your disk. When the model processes text, it dynamically loads only the specific experts needed for that task into your GPU's VRAM. As the model learns, it can create new experts or prune unused ones. It uses a "continual learning" approach where reading and training happen simultaneously, ensuring the model evolves as it consumes new data without needing a separate, static training phase.
Who it’s for
This project is for experimenters and hobbyists with modest hardware (at least 6 GB of VRAM) who want to explore building a model that learns from a continuous stream of data. It is a research-oriented, "toy-level" experiment rather than a production-ready tool for high-end tasks.
Highlights
- Disk-backed capacity: The model's parameter count is limited by your hard drive space, not your GPU memory.
- Continual learning: It learns from every character it reads, avoiding the "catastrophic forgetting" common in standard models.
- Adaptive architecture: The model automatically decides how much computing power to spend on each character, using more "rows" of processing for complex tasks and fewer for simple ones.
- Hardware-efficient: Designed to run and train on a single consumer-grade GPU with 8 GB of VRAM.
- No tokenizer: It processes raw byte values, allowing it to handle any input without needing a pre-defined vocabulary.
Related
- Dispatch
- Project
- Dispatch
- Project
- Dispatch