tong-io/tongflow

What it solves

TongFlow is a modality-first generative AI platform that allows users to create complex workflows combining text, images, audio, video, and 3D models. It solves the problem of fragmented AI tools by providing a single canvas where outputs from one model can be used as inputs for another, enabling the creation of sophisticated content like talking-head avatars or full music videos.

How it works

The platform treats different forms of information (modalities) as building blocks. Users define a workflow by selecting a material (e.g., a photo), a capability (e.g., transform to video), and an implementation (a specific model/plugin). The system uses an ABI (Application Binary Interface) to ensure that capabilities are independent of the specific models running them, allowing users to swap models without breaking the workflow structure.

Who it’s for

It is designed for creators and developers who want to visually orchestrate multi-modal AI generation pipelines. It can be used via a cloud studio, a desktop app, and is self-hostable. Additionally, it provides an npm package and MCP client support for agents to build and run workflows.

Highlights

• Modality-First Approach: Supports text, image, audio, video, 3D, documents, and URLs as primary materials.
• Extensible Plugin System: Includes a wide array of official plugins for API providers (OpenAI, Gemini, DeepSeek, etc.) and GPU/CPU runtimes (Modal).
• Visual Canvas: A node-based interface for adding, transforming, combining, and splitting AI-generated content.
• Agent-Compatible: Workflows can be created and managed by AI agents through a shared node registry.
• Broad Capability Set: Covers everything from lip-syncing and pose detection to music repaint and open-vocabulary sound separation.