Meesho/BharatMLStack

BharatMLStack is an open-source, end-to-end machine learning infrastructure stack built at Meesho to support real-time and batch ML workloads at Bharat scale

What it solves

BharatMLStack is a production-ready, cloud-agnostic ML infrastructure platform designed to help organizations deploy machine learning models to production faster and more reliably. It addresses the challenges of scaling ML systems to handle millions of queries per second (QPS) while reducing infrastructure costs and avoiding vendor lock-in with hyperscaler managed services.

How it works

The platform consists of several integrated components that handle different parts of the ML lifecycle:

  • TruffleBox UI: A web console for managing the feature registry and approval workflows.
  • Online Feature Store: Provides sub-10ms feature retrieval at massive scale with streaming ingestion.
  • Inferflow: A DAG-based orchestration engine for real-time inference pipelines.
  • Numerix: A high-performance math compute engine written in Rust for matrix operations.
  • Skye: A vector similarity search engine with pluggable backends.
  • Horizon: A control plane that orchestrates all services.
  • Interaction Store: A ScyllaDB-backed store for user interaction signals.

Who it’s for

It is built for organizations needing to serve ML models at internet scale (millions of QPS) across public cloud, on-prem, or edge environments, specifically for use cases like personalized ranking, fraud detection, and LLM-powered recommender systems.

Highlights

  • High Performance: Supports up to 2.4M QPS for feature stores and 1M+ QPS for model inference.
  • Cloud-Agnostic: Kubernetes-native and designed to run anywhere without vendor lock-in.
  • Economic Efficiency: Claims a 60-70% reduction in infrastructure costs compared to managed services.
  • Rapid Deployment: Reduces model onboarding time by 95% and speeds up experiment-to-deployment cycles by 3x.

Related

  • Project
    gpustack/gpustack

    An open-source GPU cluster manager for AI model serving and instance provisioning that orchestrates inference engines across hybrid environments.

  • Project
    deepset-ai/haystack

    An open-source AI orchestration framework for building production-ready RAG systems, autonomous agents, and LLM applications in Python.

  • Project
    feast-dev/feast

    An open-source feature store for machine learning that manages the flow of data from offline training sets to low-latency online inference.

  • Project
    mlrun/mlrun

    An open-source AI orchestration platform that automates the delivery of ML and generative AI pipelines, from data management and training to serverless deployment and monitoring.

  • Project
    mosecorg/mosec

    A high-performance model serving framework that uses a Rust core and Python interface to turn ML models into efficient, cloud-ready microservices.