bentoml/BentoML

BentoML – Unified Model Serving Framework

What it is

• A Python library that lets you turn any AI/ML model (PyTorch, TensorFlow, Transformers, etc.) into a production‑ready inference API with just a few lines of code.
• Handles packaging, Docker image creation, and deployment to Kubernetes or other environments.

Key capabilities

• Simple API definition – Decorate a class with @bentoml.service and @bentoml.api to expose functions as HTTP endpoints.
• Automatic containerisation – bentoml build creates a Bento (a versioned artifact) and bentoml containerize produces a Docker image that bundles code, models, and dependencies.
• Performance optimisations – built‑in dynamic batching, model parallelism, multi‑stage pipelines, and multi‑model orchestration to maximise CPU/GPU utilisation.
• Framework‑agnostic – works with any Python‑based model library; you specify required packages in the service definition.
• Observability & scaling – supports distributed serving, adaptive batching, and integrates with monitoring tools.

Typical workflow

1. Install pip install -U bentoml (Python ≥ 3.9).
2. Write a service (e.g., a summarisation API using Hugging Face Transformers).
3. Run locally with bentoml serve to test at http://localhost:3000.
4. Package the service: bentoml build → creates a versioned Bento.
5. Deploy: bentoml containerize <bento_name> → Docker image → run on any host or Kubernetes.

Example snippet

import bentoml

@bentoml.service(
    image=bentoml.images.Image(python_version="3.11").python_packages("torch", "transformers"),
)
class Summarization:
    def __init__(self):
        import torch
        from transformers import pipeline
        device = "cuda" if torch.cuda.is_available() else "cpu"
        self.pipeline = pipeline('summarization', device=device)

    @bentoml.api(batchable=True)
    def summarize(self, texts: list[str]) -> list[str]:
        return [r['summary_text'] for r in self.pipeline(texts)]

Running bentoml serve launches the API instantly.

Ecosystem & examples

• Ready‑made examples for LLMs (Llama 3.2, Mistral, DeepSeek), diffusion models, embeddings, audio models, computer‑vision detectors, and advanced patterns like function calling, LangGraph, and CrewAI.
• Full docs at https://docs.bentoml.com and a community forum for troubleshooting and sharing patterns.

Community & contribution

• Open‑source under the Apache 2.0 license.
• Active GitHub repo with CI, issue tracker, and contribution guides.
• Users can opt‑out of anonymous usage telemetry via --do-not-track or an environment variable.

Who should use it

• ML engineers who need to expose models as scalable APIs without writing boilerplate Docker/Kubernetes code.
• Teams building multi‑model pipelines or services that must run efficiently on CPUs/GPUs.
• Anyone looking for a reproducible, version‑controlled way to ship AI models to production.