spitfireuptown/datalinkx

🔥🔥DatalinkX异构数据源之间的数据同步系统，支持海量数据的增量或全量同步，同时支持HTTP、Oracle、MySQL、ES等数据源之间的数据流转，支持中间transform算子如SQL算子、大模型算子，底层依赖Flink、Seatunnel引擎，提供流转任务管理、任务级联配置、任务日志采集等功能🔥🔥

What it solves

DatalinkX is a data synchronization service designed to move data between different heterogeneous data sources. It centralizes the management of synchronization tasks and logs, reducing the manual effort required for cross-departmental data collaboration (e.g., moving crawler data into a data warehouse).

How it works

The system provides a web-based interface for configuring data sources and synchronization tasks. It leverages high-performance computing engines like Flink and SeaTunnel to execute the data movement. It supports batch tasks (moving data between databases), real-time tasks (specifically for Kafka), and computation tasks that allow for data transformation using transform operators. Scheduling is handled via xxl-job using cron expressions.

Who it’s for

It is intended for organizations with large-scale data needs where different teams need to share data across various database types and want a centralized way to manage and maintain these pipelines.

Highlights

  • High-Performance Engines: Uses Flink and SeaTunnel for efficient data processing.
  • Plugin-based Architecture: Supports custom data sources by developing drivers according to fixed rules.
  • Visual Configuration: Allows users to create data sources and and synchronization tasks through a GUI rather than manual coding.
  • Task Lineage and Cascading: Includes features for tracking task relationships and configuring task sequences.
  • AI Integration: Integrates with Ollama for LLM execution and Solon for MCP (Model Context Protocol) capabilities.