xorbitsai/inference

Swap GPT for any LLM by changing a single line of code. Xinference lets you run open-source, speech, and multimodal models on cloud, on-prem, or your laptop — all through one unified, production-ready inference API.

What it solves

Xinference simplifies the deployment and serving of large-scale AI models. It removes the complexity of setting up infrastructure for different model types, allowing users to deploy state-of-the-art open-source models with a single command.

How it works

Xinference acts as a model serving layer that supports a wide variety of modalities including text (LLMs), speech recognition, and multimodal models. It leverages engines like vLLM and ggml to optimize performance across heterogeneous hardware (GPUs and CPUs). It provides multiple interaction interfaces, including an OpenAI-compatible RESTful API, a WebUI, and a Python client, and supports distributed deployment across multiple machines.

Who it’s for

It is designed for researchers, developers, and data scientists who need to an easy way to host and serve AI models for experimentation or production environments.

Highlights

  • Multi-modal support: Serves language, audio, and image models (text-to-image) and embedding models.
  • Heterogeneous hardware: Intelligently utilizes both GPUs and CPUs (via ggml) and supports platforms like Metal.
  • Distributed serving: Enables model inference to be distributed across multiple devices or machines.
  • OpenAI compatibility: Provides a RESTful API that is compatible with OpenAI, including support for function calling.
  • Broad integration: Works seamlessly with frameworks like LangChain, LlamaIndex, Dify, and RAGFlow.

Related

  • Project
    triton-inference-server/server

    An open-source inference serving software that streamlines the deployment of AI models from multiple frameworks across GPUs, CPUs, and AWS Inferentia.

  • Project
    SeldonIO/MLServer

    An open-source inference server that provides a standardized REST and gRPC interface for serving machine learning models across various frameworks.

  • Project
    CerebriumAI/examples

    A collection of ready-to-use deployment examples for the Cerebrium platform, covering LLMs, voice agents, image generation, and inference optimization.

  • Project
    triton-inference-server/client

    Client libraries and examples for C++, Python, and Java that simplify communication with Triton Inference Server for model inferencing and server management.

  • Project
    jolibrain/deepdetect

    A deep learning runtime and REST server that provides a unified API for training and inference across images, text, and tabular data.