mosecorg/mosec
A high-performance ML model serving framework, offers dynamic batching and CPU/GPU pipelines to fully exploit your compute machine
What it solves
Mosec is a high-performance model serving framework designed to bridge the gap between trained machine learning models and efficient online service APIs. It simplifies the process of turning a model into a production-ready microservice while optimizing for cloud deployment and high throughput.
How it works
It uses a hybrid architecture where the web layer and task coordination are built with Rust for speed and efficient CPU utilization via async I/O, while the user interface remains in Python for ease of use. The framework allows users to to define their services as Python classes inheriting from mosec.Worker and implement a forward method for the model logic.
Key technical mechanisms include:
- Dynamic Batching: Aggregates individual requests from multiple users into a single batch for inference to increase throughput.
- Pipelined Stages: Allows the creation of multi-stage workflows (e.g., preprocessing on CPU and inference on GPU) where each stage runs in separate processes to maximize hardware utilization.
- Flexible Serialization: Supports JSON by default but provides mixins for binary formats like msgpack for handling images or embeddings more efficiently.
Who it’s for
ML engineers and backend developers who need to deploy ML models as scalable, high-performance APIs in the cloud, particularly those using Kubernetes or other container orchestration systems.
Highlights
- Rust-powered core: Blazing speed and efficient resource management.
- Framework-agnostic: Works with any ML framework (PyTorch, JAX, etc.).
- Cloud-native: Includes built-in Prometheus monitoring metrics, model warmup, and graceful shutdown.
- High throughput: Optimized via dynamic batching and multi-process pipelining.