mindsdb/lightwood

What it solves

Lightwood is an AutoML framework designed to reduce the repetitive boilerplate code associated with the machine learning (ML) life cycle. It allows users to focus on the unique aspects of their data and models rather than the manual setup of data preparation and pipeline construction.

How it works

Lightwood uses a declarative syntax called "JSON-AI" to define ML pipelines. It abstracts the process into three core stages:

1. Pre-processing and cleaning: The framework analyzes the dataset to identify data types (numeric, categorical, etc.) and automatically generates a cleaning and splitting strategy.
2. Feature Engineering: It uses "encoders" to transform pre-processed data into numerical representations. These encoders can be rule-based (following specific instructions) or learned (trained representations).
3. Model Building and Training: A "mixer" model takes the encoded features and produces a prediction. While it primarily uses PyTorch, it supports other model architectures.

Users can either accept the same default behaviors or override specific steps in the JSON-AI syntax to customize their pipeline, which Lightwood then converts into executable Python code.

Who it’s for

It is built for developers and users who want to democratize machine learning, enabling them to build predictive models without needing to be expert data scientists.

Highlights

• Declarative Pipeline: Uses JSON-AI to define and customize ML pipelines without writing boilerplate code.
• Multimodal Data Support: Works with numbers, dates, categories, tags, text, arrays, and various multimedia formats.
• Time-Series Support: Includes a dedicated mode for problems with between-row dependencies.
• Extensible Architecture: Supports "Bring Your Own Model" (BYOM) allowing users to introduce custom cleaners, splitters, and mixers.