AutoArk/TinyEngram

What it solves

TinyEngram addresses the challenge of injecting new knowledge or concepts into large models without causing "catastrophic forgetting"—where the model loses its general capabilities while learning new, specialized information. It also provides a more parameter-efficient alternative to traditional fine-tuning methods like LoRA.

How it works

The project implements the Engram architecture, which adds a compact N-gram memory module and a gated retrieval mechanism to transformer layers. Instead of modifying the base model's weights, it uses a trigger-indexed system to retrieve and integrate specific memories.

This approach is extended to both text and vision:

• For LLMs (Qwen): It injects specialized knowledge (e.g., biomedical data) into the decoder layers.
• For Vision (Stable Diffusion): It injects visual concepts into the Text Encoder by recognizing specific N-grams in the prompt, allowing the model to generate new subjects without fine-tuning the massive U-Net or DiT backbone.

Who it’s for

AI researchers and developers who want to integrate specialized domain knowledge into LLMs or new visual concepts into image generation models while maintaining the original model's general performance and parameter efficiency.

Highlights

• Resistance to Catastrophic Forgetting: Outperforms LoRA in retaining general capabilities while adapting to new domains.
• Multimodal Capability: Successfully extends memory injection to Stable Diffusion (SD1.5 and SD3.5) for lightweight concept injection.
• Parameter Efficiency: Provides a surgical, lightweight way to "teach" models new information without full fine-tuning.
• Infinite Composability: In vision tasks, memories do not interfere with each other, allowing thousands of different concepts to be stacked without degrading the base model.