mosecorg/mosec

Mosec – High‑Performance Model‑Serving Framework

What it is – Mosec (Model Serving made Efficient in the Cloud) is a Python‑first library that lets you turn any trained ML model into a fast, production‑ready HTTP service. The heavy‑lifting (network I/O, request batching, process orchestration) is written in Rust, giving you low‑latency, high‑throughput inference while you stay in familiar Python code.

Key capabilities

• Dynamic batching – automatically groups incoming requests into batches (configurable size & wait‑time) to keep GPUs busy.
• Pipeline‑style workers – chain multiple processes (CPU‑only, GPU‑only, I/O) so you can split preprocessing, inference, and post‑processing across separate stages.
• Rust‑backed server – async I/O and efficient CPU usage, plus built‑in Prometheus metrics and graceful shutdown.
• Flexible serialization – JSON by default, or Msgpack/other mixins for binary data such as images or embeddings.
• Cloud‑ready – works out‑of‑the‑box with containers, Kubernetes, and the official Docker image mosecorg/mosec.
• OpenAPI generation – automatic Swagger docs for any Worker implementation.

Typical use‑cases

• Deploying large generative models (e.g., Stable Diffusion) as a REST endpoint.
• Serving embeddings, reranking, or other lightweight inference services.
• Building multi‑stage pipelines where preprocessing runs on CPU and the model runs on GPU.
• Providing an OpenAI‑compatible API for custom models.

Installation

# From PyPI (Linux/macOS)
pip install -U mosec
# Or via Conda
conda install -c conda-forge mosec
# Or Pixi
pixi add mosec

If you want to compile from source you need Rust installed and then run make package to produce a wheel.

Quick example – serving Stable Diffusion

from mosec import Server, Worker, get_logger
from mosec.mixin import MsgpackMixin
from diffusers import StableDiffusionPipeline
import torch, io

class StableDiffusion(MsgpackMixin, Worker):
    def __init__(self):
        self.pipe = StableDiffusionPipeline.from_pretrained(
            "sd-legacy/stable-diffusion-v1-5", torch_dtype=torch.float16
        )
        self.pipe.enable_model_cpu_offload()
        self.example = ["warmup prompt"] * 4   # warm‑up batch

    def forward(self, prompts: list[str]) -> list[memoryview]:
        results = self.pipe(prompts)
        images = []
        for img in results[0]:
            buf = io.BytesIO()
            img.save(buf, format="JPEG")
            images.append(buf.getbuffer())
        return images

if __name__ == "__main__":
    srv = Server()
    srv.append_worker(StableDiffusion, num=1, max_batch_size=4, max_wait_time=10)
    srv.run()

Run the script, then POST a JSON payload with a prompt field to http://127.0.0.1:8000. The service returns the generated JPEG image via Msgpack (binary‑friendly). Swagger UI is available at /openapi/swagger/.

Ecosystem & integrations

• Works with HuggingFace diffusers, transformers, PyTorch, JAX, etc.
• Provides mixins for custom serialization, shared‑memory IPC, request validation, and metric extensions.
• Official Docker image and examples for Kubernetes deployments.
• Used by projects such as Modelz (serverless ML), Tencent Cloud ML Platform, and the MOSS conversational model.

Where to get help

• Discord: https://discord.gg/Jq5vxuH69W
• GitHub Issues: open a ticket for bugs or feature requests.
• Documentation: https://mosecorg.github.io/mosec/

Citation If you use Mosec in research, cite the provided software reference.