4paradigm/OpenMLDB

What it solves

OpenMLDB addresses the gap between offline feature engineering and online inference in machine learning. It eliminates the need for engineering teams to refactor Python scripts into C++ or database code for production, solving common issues like data leakage, feature backfilling, and the inconsistency between training and serving (training-serving skew).

How it works

OpenMLDB acts as a feature platform that implements a "Development as Deployment" philosophy. It uses SQL as a unified programming language to define features, which are then handled by two specialized engines:

• Batch SQL Engine: Based on a tailored Spark distribution for offline training data generation.
• Real-time SQL Engine: A custom-built engine optimized for time-series data to provide ultra-low latency features for online inference.

A unified execution plan generator bridges these two engines to ensure that the same SQL logic produces consistent features regardless of whether they are used for training or serving.

Who it’s for

Data scientists and ML engineers who need to serve real-time features for production ML applications, such as personalized recommendations, risk analytics, and IoT monitoring.

Highlights

• Consistent Features: Guarantees consistency between offline training and online inference to prevent data leakage.
• Ultra-Low Latency: Custom real-time engine capable of producing features in a few milliseconds.
• SQL-Based Definition: Uses extended SQL syntax (e.g., LAST JOIN, WINDOW UNION) for easier feature management.
• Production-Ready: Supports distributed storage, fault recovery, high availability, and seamless scale-out.