SeldonIO/MLServer

What it solves

MLServer provides a standardized way to deploy and serve machine learning models as APIs. It removes the need for developers to manually build custom API wrappers around every model, ensuring that models can be integrated into production environments using a consistent interface regardless of the underlying framework.

How it works

MLServer acts as an inference server that exposes REST and gRPC interfaces compliant with the V2 Inference Protocol. It uses "inference runtimes" as backend glue to connect the server to specific machine learning frameworks (such as Scikit-Learn, XGBoost, or HuggingFace). This architecture allows it to support multiple models within a single process and handle requests through a pool of inference workers.

Who it’s for

It is designed for machine learning engineers and DevOps professionals who need to scale model serving in production, particularly those using Kubernetes-native frameworks like Seldon Core or KServe.

Highlights

• Multi-model serving: Run multiple different models within the same process.
• Adaptive batching: Groups inference requests on the fly to improve throughput.
• Parallel inference: Scales vertically by running inference across multiple workers.
• Broad framework support: Pre-packaged runtimes for popular tools like Scikit-Learn, XGBoost, LightGBM, and HuggingFace.
• Standardized protocol: Fully compliant with the V2 Inference Protocol for both REST and gRPC.