sapientinc/HRM-Text

What it solves

HRM-Text addresses the high computational and data costs associated with pretraining foundation models from scratch. It enables the creation of a 1B parameter text generation model using significantly less compute (130-600x less) and data (150-900x less) than traditional scaling laws suggest, making foundation model pretraining accessible at a much lower cost (approximately $1,000).

How it works

The project implements a hierarchical recurrent architecture (HRM) strengthened by task completion and latent space reasoning. The framework utilizes several high-performance components to optimize training:

• Architecture: A hierarchical recurrent model that supports various configurations, including a standard Transformer wrapper and several recursive baselines.
• Training Pipeline: Uses PyTorch FSDP2 for distributed training, FlashAttention 3 kernels for the attention path, and PrefixLM sequence packing to improve efficiency.
• Data Handling: Integrates with a companion data_io pipeline for cleaning, tokenization, and stratified sampling of the pretraining corpus.
• Tooling: Includes a compiled generation engine for inference, evaluation scripts for standard benchmarks (like MMLU and GSM8k), and a conversion tool to export checkpoints to Hugging Face Transformers format.

Who it’s for

This is designed for AI researchers and developers who want to pretrain their own foundation models from scratch without the massive infrastructure and compute budgets typically required for large-scale LLMs.

Highlights

• Extreme Efficiency: Pretrain a 1B model in roughly 46 hours on 16 H100 GPUs for under $1,500.
• High Performance: Competitive benchmark results across reasoning and knowledge tasks (e.g., 84.7% on GSM8k for the XL model).
• Full Lifecycle Support: Provides the complete pipeline from data preparation and pretraining to evaluation and SFT (Supervised Fine-Tuning).
• Flexible Architecture: Supports multiple model sizes (B to XXL) and various recurrent and transformer-based architectures.