Shumai: An Open-Source Frame.io Alternative for Creative Work

Shumai is an open-source platform designed as an alternative to Frame.io, enabling creative teams to manage assets and collaborate through precise, frame-specific feedback. It combines traditional media review tools with modern AI capabilities and flexible storage options to streamline production pipelines.

Core Collaboration and Asset Management

Shumai provides tools for high-precision review and secure distribution of creative assets.

Frame-by-Frame Annotations

Reviewers can provide precise feedback using timestamped comments and frame-specific drawing tools. This allows teams to pinpoint exact moments in a video or specific areas of an image that require adjustment, reducing the ambiguity often found in creative feedback.

Storage and Access Control

Shumai supports flexible storage backends and granular security:

• Storage Options: Assets can be stored on a local filesystem or any S3-compatible cloud storage, including AWS S3, Cloudflare R2, and MinIO.
• Sharing: Users can create curated media collections and secure public share links for external stakeholders and clients.
• Permissions: The platform implements role-based access controls (RBAC) at both the team and project levels to manage workspace permissions.

Production Pipeline Integration

To handle the resource-intensive nature of video work, Shumai utilizes Temporal to orchestrate a background worker pool for distributed transcoding. Additionally, users can define custom dynamic metadata fields to tailor the platform to their specific production pipeline requirements.

The Shumai AI Agent

Shumai integrates a context-aware AI agent directly into the project workspace to automate asset management and extend platform functionality.

AI-Powered Automation

• Metadata Generation: Using Google Gemini, the agent can automatically generate tags, descriptions, and custom metadata for new assets.
• Semantic Search: The platform uses vector embeddings to enable semantic search, allowing users to locate assets based on conceptual or visual queries rather than just filenames or tags.

Extensibility and Security

The AI agent is designed to be extensible and secure:

• Custom Skills: Developers can register custom scripts and automation tools to expand the agent's capabilities.
• Sandboxed Execution: To ensure system security, all agent-submitted scripts are executed within an isolated sandbox environment.

Technical Architecture and Installation

Shumai is built to be deployable via multiple methods, requiring a PostgreSQL database with the pgvector extension for its AI and search capabilities.

Deployment Options

• Docker Compose: The fastest deployment method, which allows users to launch the platform without manual package installation.
• NPM/Package Manager: Available as @shumai-one/shumai, allowing for global or local installation via NPM, PNPM, or Bun.
• Source Build: For developers, the project can be cloned and run using Bun.

System Dependencies

Depending on the installation method, the following system-level dependencies are required:

• ffmpeg: Used for media transcoding and metadata extraction.
• bubblewrap: Provides the sandboxing environment for the AI agent (not required on macOS).
• socat: Used for bidirectional socket relay for sandbox network bridging.
• ripgrep: Used for fast search of workspace security policies.

Command Line Interface

Shumai includes a CLI tool that enables users to manage projects, folders, and assets, upload files, and create new versions directly from the terminal.