ArkFlow – Rust‑based Stream Processing Engine with AI Extensions

What it is

• ArkFlow is an open‑source, high‑performance stream‑processing platform written in Rust and built on the Tokio async runtime.
• It lets you ingest data from many sources, transform it with a rich set of processors (SQL, Python UDFs, Protobuf, VRL, etc.), and write the results to a variety of sinks.
• Unique to ArkFlow are first‑class AI processors: you can call OpenAI‑compatible LLM APIs, generate embeddings, and perform vector similarity searches (Qdrant, Milvus, pgvector) as part of a streaming pipeline.

Key capabilities

Category	Highlights
Performance	Rust + Tokio → low‑latency, high‑throughput; optional exactly‑once delivery via per‑stream WAL.
Inputs	Kafka, MQTT, HTTP, files (CSV/JSON/Parquet/Avro/Arrow), SQL databases, NATS, Pulsar, Redis, WebSocket, Modbus, in‑memory, synthetic generator, and composite multi‑input streams.
Processing	Built‑in JSON↔Arrow conversion, SQL queries, Protobuf codecs, batch windows, VRL, Python UDFs, Embedding (OpenAI‑compatible), LLM chat completions, vector search (Qdrant, Milvus, pgvector), lookup enrichment with caching.
Outputs	Kafka, MQTT, HTTP, InfluxDB, MongoDB, NATS, Pulsar, Redis, SQL databases, stdout, drop, Qdrant, pgvector, Milvus.
Buffers & Windows	Memory buffer, session window, tumbling window (event‑time), plus deprecated sliding‑window placeholder.
Codecs	JSON, Protobuf, Debezium CDC JSON, Confluent Schema Registry (Avro/Protobuf).
Control plane	Optional Hub and web console for managing fleets of ArkFlow nodes.
Extensibility	Modular architecture; new inputs, outputs, processors, or buffers can be added as plugins.

Typical use cases

• Real‑time IoT telemetry ingestion → enrichment with Python UDFs → anomaly detection via LLM prompts.
• Streaming log data → SQL filtering → embedding generation → nearest‑neighbor search in Qdrant for similarity alerts.
• CDC (Change Data Capture) from databases → Debezium JSON decoding → transformation → write to a downstream Kafka topic.
• High‑frequency synthetic data generation for load testing pipelines.

Getting started

# Clone and build (requires Rust toolchain)
git clone https://github.com/arkflow-rs/arkflow.git
cd arkflow
cargo build --release

# Run a quick test configuration
cat > config.yaml <<'EOF'
logging:
  level: info
streams:
  - input:
      type: "generate"
      context: '{ "timestamp": 1625000000000, "value": 10, "sensor": "temp_1" }'
      interval: 1s
      batch_size: 10
    pipeline:
      thread_num: 4
      processors:
        - type: "json_to_arrow"
        - type: "sql"
          query: "SELECT * FROM flow WHERE value >= 10"
    output:
      type: "stdout"
    error_output:
      type: "stdout"
EOF

./target/release/arkflow --config config.yaml

The example generates synthetic sensor records, converts JSON to Arrow, filters with SQL, and prints matching rows.

Documentation & community

• Official docs: https://arkflow-rs.com/docs/ (including 0.5.x intro).
• Community Discord: https://discord.gg/CwKhzb8pux.
• Plugin examples repo: https://github.com/arkflow-rs/arkflow-plugin-examples.

License

• Apache License 2.0.

Bottom line ArkFlow combines the speed and safety of Rust with a flexible, declarative streaming model and built‑in AI/ML operators, making it a solid choice for developers who need real‑time data pipelines that can run inference, generate embeddings, or perform vector search on the fly.