microsoft/unilm

Large-scale Self-supervised Pre-training Across Tasks, Languages, and Modalities

What it solves

This repository serves as a comprehensive collection of foundation models and architectures designed to achieve "The Big Convergence"—large-scale self-supervised pre-training across diverse tasks, languages (100+), and modalities including text, vision, speech, and document layout.

How it works

The project implements a wide array of specialized architectures and models:

  • Foundation Architectures: Includes TorchScale for general foundation model development, DeepNet for scaling Transformers to 1,000+ layers, and X-MoE for scalable sparse Mixture-of-Experts.
  • Model Innovations: Features BitNet (1-bit Transformers), RetNet (a Transformer successor), and LongNet for handling up to 1 billion tokens.
  • Modality-Specific Models:
    • Language: UniLM for unified understanding and generation, and MiniLM for fast, small-scale models.
    • Vision: BEiT and DiT for self-supervised image and document image pre-training.
    • Speech: WavLM for full-stack speech tasks and VALL-E for neural codec-based TTS.
    • Multimodal: LayoutLM series for Document AI (text + layout + image) and Kosmos series for multimodal LLMs.

Who it’s for

AI researchers and developers looking for state-of-the-art pre-trained foundation models, scalable transformer architectures, or specialized tools for multimodal and multilingual processing.

Highlights

  • Cross-Modal Versatility: Supports text, image, audio, and layout/format in single or combined models.
  • Architectural Scaling: Research into extreme depth (DeepNet) and extreme sequence length (LongNet).
  • Efficiency Focus: Includes 1-bit quantization (BitNet) and knowledge distillation (MiniLLM).
  • Document AI Leadership: Extensive suite of models (LayoutLM, MarkupLM, XDoc) for understanding visually-rich documents.

Related

  • Project
    axolotl-ai-cloud/axolotl

    A free and open-source LLM fine-tuning framework that streamlines post-training for text, vision, and audio models using a single YAML configuration.

  • Project
    tracel-ai/models

    A collection of deep learning model implementations built with the Burn framework, covering text, image, and audio modalities.

  • Project
    pengzhangzhi/Open-dLLM

    A complete open-source framework for diffusion-based large language models, providing the entire pipeline from pretraining and datasets to evaluation and inference.

  • Project
    modelscope/ms-swift

    A large model and multimodal fine-tuning and deployment framework that supports over 1,000 models and provides a full pipeline from training to deployment.

  • Project
    Osilly/Vision-DeepResearch

    A framework and benchmark for developing Multimodal Large Language Models capable of deep research through iterative visual and textual search across images and videos.