insitro/redun

Yet another redundant workflow engine

redun is a _redundant workflow engine_ for Python. It lets you define data-processing pipelines as ordinary Python code, then it schedules those pipelines for you — automatically running steps in parallel, skipping steps that haven't changed, and recording a full history of how every result was produced.

Related

  • Project
    magda-io/magda

    Magda is a data catalog system that gives organizations a single place to discover, enrich, search, track, and prioritize all their data—whether it's stored internally, externally, or as files, databases, or APIs. It uses federation to combine metadata from many sources into one searchable view.

  • Project
    lightonai/next-plaid

    NextPlaid is a local-first multi-vector search database with a REST API; ColGREP is a semantic code-search tool built on it — a single Rust binary that indexes your codebase locally and answers natural-language queries (optionally via a small embedded agent model), combining regex filtering with semantic ranking. NextPlaid solves the storage cost of multi-vector embeddings (hundreds per document) with quantization, memory-mapped indices, incremental updates, and SQL metadata pre-filtering, all CPU-friendly. Both tools are Apache-2.0 licensed, use ONNX-exported ColBERT models, and keep all data on your machine.

  • Project
    OvidijusParsiunas/myvision

    A free online image annotation tool for generating computer vision ML training data with support for automated bounding box generation and dataset format conversion.

  • Project
    apache/datafusion-python

    DataFusion Python is a bridge that exposes Apache DataFusion's high-performance Rust-based SQL and DataFrame query engine to Python, letting developers run optimized SQL queries on CSV/Parquet/JSON data, plug in custom Python functions, and exchange results with Pandas. It is widely used as the foundation for other data systems, is installable via common package managers, and includes tooling for contributors and AI-assisted development.