Eventual-Inc/Daft

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

What it solves

Daft addresses the challenge of processing massive amounts of multimodal data—such as images, audio, video, and embeddings—alongside traditional structured data. It eliminates the need for complex JVM-based setups and provides a high-performance way to run AI operations at scale.

How it works

It is a Python-native data engine powered by Rust for high performance. Daft uses a vectorized execution engine and an Arrow-backed architecture to handle data efficiently. It can run locally or scale to distributed clusters using Ray or Kubernetes, and it connects to various data sources including S3, GCS, Iceberg, Delta Lake, and Hugging Face.

Who it’s for

Data engineers and AI practitioners who need to process large-scale multimodal datasets and integrate AI inference (like LLM prompts or embedding generation) directly into their data pipelines.

Highlights

  • Native Multimodal Support: Handles images, audio, and video in a single framework.
  • Built-in AI Ops: Direct integration for running LLM prompts, generating embeddings, and classifying data via OpenAI, Transformers, or custom models.
  • Rust-Powered: Combines Python's ease of use with Rust's speed.
  • Flexible Scaling: Seamlessly moves from local execution to distributed clusters.
  • Universal Connectivity: Native access to cloud storage and modern table formats like Iceberg and Delta Lake.

Related

  • Project
    databendlabs/databend

    An open-source enterprise data warehouse built in Rust that integrates analytics, vector search, and secure Python sandboxes for AI agent orchestration.

  • Project
    NVIDIA/raft

    A library of CUDA-accelerated fundamental algorithms and primitives for machine learning and data mining, providing high-performance building blocks for AI application developers.

  • Project
    WeaveMindAI/weft

    Weft is a compiled, graph‑based language and runtime for orchestrating AI workflows. It lets you connect LLM calls, agents, APIs, databases, and human‑in‑the‑loop steps as nodes in a typed graph, validates the design before execution, persists long‑running jobs, and records a journal of all data flows. Integrated with the AI assistant “Tangle”, it supports automatic code generation, testing, and versioned runs. Early‑stage but usable, with Docker‑based installer, VS Code extension, and examples like a WhatsApp support bot.

  • Project
    rustfs/rustfs

    RustFS is a high‑performance, distributed object storage system written in Rust. It provides full S3‑compatible APIs (versioning, encryption, lifecycle, replication, etc.), native OpenStack Swift support, a web console, and optional protocols like FTPS/WebDAV/SFTP. Distributed deployments can run on Docker, Kubernetes (Helm chart), or as a single‑node binary, and it integrates with Vault/AWS KMS, OIDC, and webhook notifications. Licensed under Apache 2.0, RustFS is positioned for AI/ML data‑lake workloads, edge devices, and enterprise‑grade storage.

  • Project
    livekit/rust-sdks

    LiveKit’s Rust SDK is a cross‑platform client library for real‑time video, audio and data streams. It lets Rust apps connect to LiveKit servers (cloud or self‑hosted) using WebRTC, supporting modern codecs, hardware acceleration, simulcast, and data channels. The repo ships multiple crates (`livekit`, `livekit-api`, etc.) and includes examples, build instructions, and integration notes for AI voice agents.