shaiwz/data-platform-open

What it solves

This project provides a zero-code, visual drag-and-drop platform for big data integration. It eliminates the need for manual coding when synchronizing, cleaning, and filtering data across various sources, significantly reducing the time required to set up data pipelines from days to minutes.

How it works

The platform uses a visual canvas where users can drag and drop components to build data flows. It integrates AI capabilities to assist in constructing these flows and creating query templates via natural language prompts. The system supports a wide range of data sources (e.g., MySQL, StarRocks, Doris, Oracle, Kafka, ElasticSearch) via a plugin architecture and includes a data alignment feature to verify consistency between different data sources and automatically repair discrepancies.

Who it’s for

It is designed for junior developers or operations personnel who need to manage complex big data pipelines without writing extensive code, as well as organizations looking to optimize their data synchronization and cleaning workflows.

Highlights

• AI-Driven Construction: Use natural language to build data flows and query templates, with the ability to to modify or delete specific parts of a flow incrementally to save tokens.
• Zero-Code Visual Canvas: Drag-and-drop interface for data synchronization, cleaning, and filtering.
• Multi-Source Support: Extensive support for various databases and message queues with dedicated consoles for each.
• Data Consistency Tools: Built-in tools for consistency checking, alerting, and automated repair between arbitrary data sources.
• Scalable Architecture: Supports dynamic scaling and automatic fault tolerance for server resources.