xorbitsai/inference

Swap GPT for any LLM by changing a single line of code. Xinference lets you run open-source, speech, and multimodal models on cloud, on-prem, or your laptop — all through one unified, production-ready inference API.

What it solves

Xinference simplifies the deployment and serving of large-scale AI models. It removes the complexity of managing infrastructure for language, speech recognition, and multimodal models, allowing users to deploy state-of-the-art open-source models with a single command.

How it works

Xinference acts as a model serving library that provides a unified interface for various inference engines (such as vLLM, GGML, and TensorRT). It supports heterogeneous hardware utilization, enabling models to run across GPUs and CPUs. It also offers distributed deployment to spread inference tasks across multiple machines or devices.

Who it’s for

Researchers, developers, and data scientists who need to serve AI models for experimentation or production without managing complex backend infrastructure.

Highlights

  • Multi-modal support: Serves LLMs, image generation, text embedding, and audio models.
  • OpenAI-compatible: Provides a RESTful API that is compatible with OpenAI, including support for Function Calling.
  • Hardware flexibility: Intelligently utilizes GPUs, CPUs, and Metal (Apple Silicon) via GGML.
  • Enterprise-ready: Supports multi-node cluster deployment and integrates with tools like LangChain, LlamaIndex, and Dify.
  • Performance optimizations: Includes auto-batching for higher throughput and continuous batching via Xllamacpp.

Related

  • Project
    sgl-project/sglang

    An open-source inference framework optimized for fast serving of LLMs, vision-language, and diffusion models, particularly for agentic workloads and RL rollouts.

  • Project
    SeldonIO/MLServer

    An open-source inference server that provides a standardized REST and gRPC interface for serving machine learning models across various frameworks.

  • Project
    vllm-project/vllm-omni

    vLLM‑Omni is an open‑source extension of the vLLM inference engine that adds fast, scalable serving for multimodal models (text, image, audio, video, and action). It supports both autoregressive and diffusion‑style architectures, provides a full‑duplex realtime API, works across GPUs, NPUs, and other accelerators, and ships ready‑to‑run recipes for many popular Hugging Face models. The project is production‑grade, Apache‑2.0 licensed, and backed by documentation, a Slack community, and recent technical reports.

  • Project
    ai-dynamo/nixl

    NVIDIA Inference Xfer Library (NIXL) is a high-performance communication library that accelerates point-to-point data transfers across CPU, GPU, and various storage backends for AI inference frameworks.

  • Project
    transformerlab/transformerlab-app

    An open-source machine learning platform that unifies training, fine-tuning, inference, and evaluation into a single interface for individuals and research labs.