kyegomez/zeta

Build high-performance AI models with modular building blocks

What it solves

Zeta is a modular PyTorch framework that simplifies AI model development by providing a library of reusable, high-performance building blocks. It eliminates the need for developers to rebuild common architectural components from scratch, allowing for the rapid assembly of state-of-the-art models.

How it works

Zeta provides a comprehensive collection of optimized PyTorch modules that can be used as drop-in replacements in AI architectures. These include:

  • Attention Mechanisms: Implementations of Multi-query, Sigmoid, and Flash attention.
  • Mixture of Experts (MoE): Tools for efficient expert routing and gating.
  • Quantization: Techniques like BitLinear (1-bit Transformers) and dynamic quantization via Niva.
  • Neural Network Modules: Specialized activation functions (e.g., SwiGLU), normalization layers, and feedforward networks.
  • Architectures: Pre-built implementations of Transformers, Vision Transformers (ViT), U-Net, and the PalmE multi-modal architecture.
  • Performance Tools: Fused operations (e.g., FusedDenseGELUDense) to reduce kernel overhead and a hyper_optimize decorator for mixed precision and compilation.

Who it’s for

AI researchers and developers who want to build, optimize, and experiment with modern neural network architectures using PyTorch without implementing every low-level component manually.

Highlights

  • Modular Design: Components are designed as seamless PyTorch drop-in replacements.
  • High Performance: Includes fused kernels and optimized implementations for speedups.
  • Broad Scope: Covers everything from text and vision to multi-modal models and RLHF (via Direct Policy Optimization).
  • Production-Ready: Optimized for reliability with comprehensive test coverage.

Related

  • Project
    Lightning-AI/pytorch-lightning

    A deep learning framework that abstracts away the engineering boilerplate of PyTorch, enabling researchers to pretrain and finetune AI models at scale across various hardware.

  • Project
    sktime/pytorch-forecasting

    A PyTorch-based library for time series forecasting that provides a high-level API and state-of-the-art deep learning architectures like Temporal Fusion Transformers and N-BEATS.

  • Project
    pytorch/pytorch

    A Python-based machine learning framework that provides GPU-accelerated tensor computation and a dynamic autograd system for building deep neural networks.

  • Project
    xlite-dev/LeetCUDA

    A comprehensive learning resource and kernel library for modern CUDA programming with PyTorch, featuring over 200 kernels ranging from basic operations to high-performance FlashAttention and HGEMM implementations.

  • Project
    catalyst-team/catalyst

    A PyTorch framework for deep learning R&D that removes training loop boilerplate to accelerate experimentation and reproducibility.