magenta/magenta-realtime

Magenta RealTime 2: An Open-Weights Live Music Model

What it solves

It enables the real-time generation of music, allowing users to create audio streams faster than they are played back. This allows for the creation of live performance tools, DAW plugins, and interactive music applications.

How it works

The project provides an open-weights model available in two sizes (230M and 2.4B parameters) and a multi-layered infrastructure for inference. It uses a Python library (magenta-rt) with JAX and MLX backends for general inference, and a specialized C++ inference engine (magentart::core) optimized for efficient streaming audio generation on Apple Silicon MacBooks.

Who it’s for

Developers and musicians who want to embed real-time music generation into applications, build AUv3 plugins for Digital Audio Workstations (DAWs), or experiment with music AI models via Python.

Highlights

  • Multi-Backend Support: Supports JAX and MLX for Python inference.
  • Apple Silicon Optimization: Specialized C++ engine for low-latency streaming on M-series chips.
  • Flexible Model Sizes: Offers a small model for broader compatibility and a base model for higher quality.
  • DAW Integration: Includes an all-in-one AUv3 plugin example for seamless integration into music production software.

Related

  • Project
    antirez/h3.c

    A native Metal inference engine for MiniMax-H3 on Apple Silicon, enabling local text-to-video and text-to-audio generation with advanced memory and performance optimizations.

  • Project
    Blaizzy/mlx-audio

    MLX‑Audio is a Python (and optional Swift) library that runs a large collection of modern speech‑to‑text, text‑to‑speech, speech‑to‑speech, voice‑cloning, VAD/diarization, and music‑generation models on Apple’s MLX framework. It provides CLI tools, a Python API, a web UI, and an OpenAI‑compatible REST endpoint, with support for 3‑ to 8‑bit quantisation to fit big models on M‑series chips.

  • Project
    TheStageAI/TheWhisper

    A high-performance speech-to-text solution based on fine-tuned Whisper models, optimized for low-latency streaming and on-device inference on NVIDIA GPUs and Apple Silicon.

  • Project
    jjang-ai/vmlx

    A self-hosted inference server for Apple Silicon that provides OpenAI and Anthropic compatible APIs for running LLMs, VLMs, and image generation models locally.

  • Project
    raullenchai/Rapid-MLX

    Rapid‑MLX is a native Apple‑silicon inference engine that runs LLMs, diffusion, audio and video models locally and exposes a drop‑in OpenAI/Anthropic HTTP API, letting any client (LangChain, Claude Code, Aider, etc.) run fully offline on an M‑series Mac.