multimodal-art-projection/YuE

YuE2: frontier music generation with symbolic planning, zero-shot covers, and agentic music editing.

What it solves

YuE2 is a music generation system that combines symbolic composition (scores) with high-quality audio rendering. It solves the "black box" problem of AI music generation by making the underlying melody and chords explicit and editable, allowing users to refine a song's structure before it is rendered into audio.

How it works

YuE2 uses an AR–NAR Mixture-of-Transformers backbone. It first predicts a symbolic score and semantic tokens autoregressively, then generates acoustic latents using flow matching, which are finally decoded into stereo audio via a VAE. The system supports three primary workflows:

  • Creation: Generates a score and audio from lyrics and style prompts.
  • Covering: Uses a transcribed recording (via SheetSage2) to create a new interpretation in a different style.
  • Editing: Allows users or agents to modify the ABC score to change harmony, melody, or tempo, then re-render the song.

Who it’s for

It is designed for musicians, producers, and AI researchers who need precise control over music generation, as well as developers building agentic music tools.

Highlights

  • White-box Generation: Provides editable ABC scores for melody and chords.
  • Frontier Quality: Competitive with Suno v5/v6 on the WildSongBench benchmark.
  • Agentic Editing: Includes a dedicated yue2-music skill for agents to transcribe, edit scores, and organize comparisons.
  • Zero-shot Covers: Can reimagine existing songs in new styles without specific fine-tuning.
  • Integrated Ecosystem: Includes SheetSage2 for audio-to-score transcription and MERT2 for music understanding.

Related

  • Project
    GetBindu/Bindu

    Bindu is an open‑source framework that turns any LLM‑based agent into a secure, identity‑verified, payment‑enabled internet service. It provides built‑in mTLS, DID signatures, OAuth2 via Hydra, USDC (x402) payment enforcement, skill advertising, push notifications, and polyglot SDKs (Python, TypeScript, Kotlin). With a single `bindufy()` call you get a JSON‑RPC endpoint, optional public tunneling, and deployment helpers, making it easy to build swarms, paid APIs, or regulated B2B agent integrations.

  • Project
    awslabs/mcp

    Open‑source MCP (Model Context Protocol) servers that let LLM‑powered tools fetch live AWS documentation, IaC guidance, and safely invoke AWS APIs. Includes managed remote servers and local binaries for docs, IaC, EKS, etc., all accessed via a stdio‑based protocol.

  • Project
    espnet/espnet

    An end-to-end speech processing toolkit built on PyTorch that provides unified, reproducible recipes for ASR, TTS, speech translation, and enhancement.

  • Project
    drivendataorg/cookiecutter-data-science

    Cookiecutter Data Science (CCDS) is a CLI tool that creates a standardized, best‑practice project skeleton for data‑science work, including folders for raw/processed data, notebooks, a Python package with training/prediction scripts, docs, and a Makefile. Install via pipx, run `ccds`, answer prompts, and you get a ready‑to‑use, reproducible project layout.