xorbitsai/inference

Swap GPT for any LLM by changing a single line of code. Xinference lets you run open-source, speech, and multimodal models on cloud, on-prem, or your laptop — all through one unified, production-ready inference API.

What it solves

Xinference simplifies the complex process of deploying and serving large-scale AI models. It removes the friction of setting up infrastructure for different model types, allowing users to move from experimentation to production with a single command.

How it works

Xinference acts as a model serving layer that supports a wide variety of modalities including text, speech recognition, and multimodal models. It leverages various inference engines (like vLLM, GGML, and TensorRT) and intelligently utilizes heterogeneous hardware (CPUs and GPUs) to optimize performance. It provides a unified, OpenAI-compatible RESTful API, a WebUI, and a CLI for model management.

Who it’s for

Researchers, developers, and data scientists who need to deploy open-source AI models quickly across different hardware configurations, including multi-node clusters.

Highlights

  • Multi-modal support: Serves LLMs, image (text-to-image), audio, and embedding models.
  • Hardware flexibility: Supports CPU, Metal, and GPU acceleration via GGML and other engines.
  • Distributed deployment: Enables running models across multiple devices or machines.
  • OpenAI compatibility: Provides a RESTful API that supports function calling.
  • Broad integration: Works seamlessly with LangChain, LlamaIndex, Dify, and RAGFlow.