facebookresearch/PhysicsLM4

Physics of Language Models: Part 4.2, Canon Layers at Scale where Synthetic Pretraining Resonates in Reality

What it solves

This project provides the research code and data for the Physics of Language Models series, specifically focusing on the design of "Canon layers" to improve the architecture of Large Language Models (LLMs). It aims to bridge the gap between synthetic pretraining experiments and real-world performance, demonstrating how specific architectural changes can enhance model capacity and knowledge storage.

How it works

The repository implements several key architectural modifications and provides the tools to reproduce their results:

  • Canon Layers: Implements "Canon-ABCD" and "Canon-AbCD" layers for both Transformer-based models (like Llama) and linear models (like GLA, GDN, and Mamba2).
  • Model Implementations: Provides Hugging Face-compatible models such as LlamaCanon, which includes QK-norm and partial RoPE support.
  • Optimized Training: Uses a modified version of Meta's Lingua codebase for efficient pretraining, incorporating z-loss and other stability improvements.
  • Synthetic and Real-life Data: Includes data generators for synthetic datasets (Depo, Brevo, Capo, Mano, Lano) and real-life evaluation benchmarks (multi-hop and Babilong) to test the models' ability to retrieve and store knowledge.

Who it’s for

This is primarily for AI researchers and practitioners interested in the fundamental architecture design of LLMs, scaling laws, and the physics of how models learn hierarchical structures and store knowledge.

Highlights

  • Architectural Innovations: Introduces Canon layers to both Transformer and linear models.
  • Comprehensive Benchmarks: Provides training recipes and weights for 16 Llama-based models and 48 linear models.
  • Crossover Validation: Validates synthetic pretraining findings in real-world pretraining settings.
  • Integration: Seamlessly integrates with Hugging Face and Meta's Lingua framework.

Related

  • Project
    ml-explore/mlx-lm

    A Python package for generating text and fine-tuning large language models on Apple silicon using the MLX framework.

  • Project
    scaleapi/llm-engine

    An open-source engine for fine-tuning and serving large language models, providing tools to deploy models on hosted or self-managed Kubernetes infrastructure.

  • Project
    Farzad-R/LLM-Zero-to-Hundred

    A collection of LLM projects and tutorials covering RAG, agentic workflows, multimodal chatbots, and fine-tuning pipelines to move from basic to advanced AI application development.

  • Project
    microsoft/unilm

    A comprehensive collection of foundation models and architectures from Microsoft focusing on large-scale self-supervised pre-training across text, vision, speech, and multimodal tasks.

  • Project
    ZHZisZZ/dllm

    A library that unifies the training, inference, and evaluation of diffusion language models, providing scalable pipelines and recipes for converting AR models into diffusion models.