alexiglad/EBT

What it solves

Energy-Based Transformers (EBTs) address the limitations of standard feed-forward Transformers by enabling "System 2 Thinking" (generalizable reasoning) across various modalities. The project aims to create models that can "think" over every single prediction—such as every token in a language model—to improve generalization and scalability across data, depth, parameters, and FLOPs.

How it works

EBTs replace or augment the standard autoregressive process with an energy-based approach. The codebase provides implementations for both "System 1" (fast, intuitive) and "System 2" (slow, reasoning-based) EBTs. It supports multiple modalities including NLP, images, and video, utilizing PyTorch Lightning for distributed training and inference. The architecture can bet adapted for different tasks, with specific implementations based on Llama 2 and DiT (Diffusion Transformer).

Who it’s for

AI researchers and developers focusing on scalable reasoning, energy-based models, and multimodal learning who want to move beyond standard feed-forward architectures.

Highlights

• Multimodal Scalability: Supports NLP, image, and video data.
• System 2 Thinking: Enables models to perform self-verification and reasoning during inference.
• Outscales Transformers: Claims to outperform feed-forward Transformers across multiple axes of scaling.
• Flexible Training: Includes pre-configured job scripts for bash and Slurm HPC environments.