m3dev/gokart

What it solves

Gokart simplifies the creation of reproducible machine learning pipelines. It addresses common challenges in ML development such as managing task dependencies, ensuring consistent results through seed fixing, and maintaining a clean codebase by encouraging SOLID principles.

How it works

Built as a wrapper for Luigi, Gokart organizes ML workflows into a series of tasks. It automatically tracks metadata for each task—including output data, module versions, processing time, and random seeds—and stores them in separate pickle files with hash values. If task parameters change, the pipeline automatically reruns the affected parts. To keep memory usage low, intermediate results are exchanged as files, and the system supports cloud storage (GCS and S3) for these results. It also includes built-in support for pandas DataFrame column checking during I/O and uses Redis to lock tasks when running in parallel.

Who it’s for

Machine learning engineers and data scientists who need a robust, reproducible environment for batch processing and team development.

Highlights

• Automatic Reproducibility: Automatically fixes seeds for numpy and random and tracks all module versions and parameters.
• Smart Rerunning: Triggers pipeline reruns automatically when task parameters are changed.
• Cloud Integration: Native support for S3 and GCS as data stores for intermediate results.
• Type Safety: Supports type-annotations and a mypy plugin to ensure pipeline robustness and prevent type errors between tasks.
• Memory Efficient: Exchanges data between tasks via intermediate files rather than keeping everything in memory.