arkflow-rs/arkflow

High performance Rust stream processing engine seamlessly integrates AI capabilities, providing powerful real-time data processing and intelligent analysis.

ArkFlow

Summary High‑performance Rust stream processing engine that integrates AI capabilities for real‑time data processing, inference, anomaly detection, and complex event processing.

Purpose Provides a durable, extensible platform to ingest data from many sources, process it with SQL, Python UDFs, JSON/Protobuf transforms, etc., and write results to various sinks while enabling the loading and execution of machine learning models for intelligent streaming analytics.

How it works

  • Built on Rust and the Tokio async runtime for low‑latency, high‑throughput processing.
  • Users define streams in a YAML config: an input (Kafka, MQTT, HTTP, file, generate, SQL, etc.), a pipeline with processors (json_to_arrow, SQL, Python UDFs, VRL, batch, Protobuf), an output, optional error_output, and optional buffer (memory, session/sliding/tumbling windows).
  • Events flow asynchronously through the pipeline; durability is achieved via per‑stream write‑ahead log (WAL) giving at‑least‑once delivery, with exactly‑once available for transactional sinks.
  • An optional Hub and web console form a control plane to observe, configure, and operate multiple ArkFlow nodes as a fleet.

Key features

  • High performance: Rust + Tokio.
  • Durable delivery: WAL‑based at‑least‑once, optional exactly‑once for transactional sinks.
  • Multiple data sources: Kafka, MQTT, HTTP, files (CSV/JSON/Parquet/Avro/Arrow), SQL, NATS, Pulsar, Redis, WebSocket, Modbus, memory, generate, and combinable multi‑input.
  • Powerful processing: SQL queries, Python UDFs, JSON/Protobuf transforms, VRL, batch processing.
  • Streaming codecs: JSON, Protobuf, Debezium CDC envelopes, Confluent Schema Registry wire‑format.
  • Control plane: Hub + web console for fleet management.
  • Extensible: modular design to add new input, buffer, output, processor components.
  • Buffers: memory, session window, sliding window, tumbling window for backpressure and windowing.

Limitations

  • The README does not specify any particular limitations; details about AI/ML model loading or exact‑once guarantees beyond transactional sinks are not elaborated.

Best for Developers and operators who need a cloud‑native, high‑throughput stream processing engine that can run machine learning models on streaming data for real‑time inference, anomaly detection, and complex event processing across heterogeneous data sources.

Related

  • Project
    lakesoul-io/LakeSoul

    LakeSoul is an open‑source lakehouse framework with a Rust‑native metadata and I/O core, offering ACID transactions, incremental upserts, automatic compaction, fine‑grained RBAC, and support for Spark, Flink, Presto, Ray, Daft, and Python‑based tools. It lets you store regular rows and multimodal/vector data (via the Vortex format) and provides real‑time CDC ingestion, snapshot queries, and seamless integration with AI/ML pipelines.

  • Project
    infiniflow/ragflow

    An open-source RAG engine that combines deep document understanding and agentic capabilities to create a high-fidelity context layer for LLMs using complex unstructured data.

  • Project
    Eventual-Inc/Daft

    A high-performance, Rust-powered data engine designed to process multimodal AI workloads, allowing users to handle images, audio, and video alongside structured data at scale.

  • Project
    OpenDCAI/DataFlow

    DataFlow is an open‑source Python framework that lets you build, run, and share low‑code pipelines for generating, cleaning, and evaluating LLM training data. It provides a library of reusable operators, a visual WebUI, an AI‑assistant for auto‑creating pipelines, and a Ray‑based distributed execution layer.

  • Project
    timeplus-io/proton

    A high-performance C++ SQL pipeline engine for real-time stream processing and analytics, serving as a lightweight alternative to Apache Flink and ksqlDB.