shaiwz/data-platform-open

What it solves

It addresses the complexity and time-consuming nature of big data integration. Instead of requiring high-level developers to write hard-coded synchronization, filtering, and transformation logic, it allows users to build data flows using a zero-code, visual drag-and-drop interface.

How it works

The platform provides a visual canvas where users can drag and drop components to define data synchronization, cleaning, and filtering operations. It integrates AI capabilities to assist in building these data flows and creating query templates via natural language. The backend supports various data sources (MySQL, StarRocks, Doris, Oracle, Kafka, ElasticSearch) and can ingest messages from FlinkCDC and Canal. It also includes a data alignment system to verify consistency between different data sources and automate repairs.

Who it’s for

It is designed for junior developers or operations personnel who need to manage big data pipelines without writing extensive code, as well as organizations looking to reduce the time spent on data integration tasks.

Highlights

• Zero-Code Visual Interface: Build complex data flows and report generation via a drag-and-drop canvas.
• AI-Powered Assistance: Use natural language to construct data flows and query templates, with detailed AI reasoning logs and token tracking.
• Intelligent Data Querying: An "AI Smart Query" feature that provides answers based on data.
• Data Alignment: Automated consistency checking and repair between different data sources and table types.
• High Performance: Supports high throughput for both stream processing and batch processing.
• Dynamic Scaling: Service instances support dynamic expansion and contraction to handle load.