qiantongtech/qData

qData is an open-source data governance and data development platform that integrates ETL, data development, metadata management, data quality, data assets, API services, and AI-powered data Q&A.

What it solves

qData is an open-source data middle platform designed to simplify enterprise data governance and development. It addresses the challenge of managing fragmented data across multiple databases (such as MySQL, Oracle, and DM8) by providing a centralized system for ETL, data modeling, and asset management.

How it works

The platform uses a frontend-backend separated architecture (Vue 3 and Spring Boot) and integrates several execution engines and schedulers to handle different workloads:

  • Data Integration: Uses DataX for lightweight synchronization and Spark for large-scale offline processing.
  • Scheduling: Employs Quartz for simple periodic tasks and DolphinScheduler for complex visual task orchestration.
  • AI Integration: Incorporates Text2SQL and natural-language processing to allow business users to query data using plain language.
  • Governance: Provides modules for metadata collection, data quality auditing, and asset cataloging.

Who it’s for

  • Data Development Teams: For building ETL workflows and SQL-based data processing tasks.
  • Data Governance Teams: For managing data standards, metadata, and quality inspection.
  • Business Analysis Teams: For querying and analyzing data via AI-powered natural language interfaces.
  • Developers/ISVs: As a foundation for building custom data service platforms.

Highlights

  • Visual ETL: Drag-and-drop configuration for data access, cleansing, and transformation.
  • AI-Powered Data Q&A: Natural-language data questions converted to SQL (Text2SQL) with intelligent chart analysis.
  • Multi-Engine Support: Flexible choice between DataX/Spark for execution and Quartz/DolphinScheduler for scheduling.
  • Comprehensive Governance: Integrated tools for data modeling, metadata management, and data quality auditing.
  • API Data Services: Ability to encapsulate SQL query results as API services for system integration.

Related

  • Project
    alldatacenter/alldata

    AllData is a comprehensive data middle-platform that provides an end-to-end suite for data integration, storage, governance, and AI application development.

  • Project
    clidey/whodb

    A lightweight, self-hosted database workspace for exploring, editing, and querying data with optional AI-powered natural language interfaces.

  • Project
    steedos/steedos-platform

    An AI-native infrastructure that uses a universal metadata standard to enable the instant generation of enterprise software from text descriptions.

  • Project
    datahub-project/datahub

    DataHub is an open‑source, AI‑ready metadata catalog (originating from LinkedIn) that unifies data‑asset information across an organization. It offers real‑time streaming ingestion from 80+ connectors, a searchable UI with lineage and governance views, extensive APIs/SDKs, and native support for LLM agents (Analytics Agent). Deployable via Docker, Helm, or SaaS, it enables discovery, impact analysis, compliance, and natural‑language data queries.

  • Project
    open-metadata/OpenMetadata

    An open-source context layer and metadata management platform that provides AI agents and users with the governed business semantics, lineage, and organizational memory needed to trust and use enterprise data.