michaelfeil/infinity

Infinity is a high-throughput, low-latency serving engine for text-embeddings, reranking models, clip, clap and colpali

What it solves

Infinity provides a high-throughput, low-latency REST API for serving various AI models used in retrieval-augmented generation (RAG) and search. It simplifies the deployment of embedding, reranking, and multi-modal models from HuggingFace, removing the need to build custom inference servers for these specific tasks.

How it works

Built on FastAPI, Infinity orchestrates multiple models using fast inference backends like PyTorch, Optimum (ONNX/TensorRT), and CTranslate2. It utilizes dynamic batching, dedicated worker threads for tokenization, and FlashAttention to optimize performance across various hardware accelerators including NVIDIA CUDA, AMD ROCm, CPU, AWS INF2, and Apple MPS.

Who it’s for

Developers and ML engineers who need to deploy scalable, OpenAI-compatible embedding and reranking services for search, clustering, or multi-modal retrieval systems.

Highlights

  • Broad Model Support: Serves text embeddings, reranking models, CLIP, CLAP, ColBERT, and ColPali models.
  • Hardware Flexibility: Supports a wide range of accelerators including NVIDIA, AMD, Apple Silicon, and CPUs.
  • ** uma OpenAI-Compatible API**: Aligns with OpenAI's API specifications for ease of integration.
  • Multi-Model Orchestration: Ability to launch and mix-and-match multiple models simultaneously via a single CLI or environment variables.
  • Efficient Inference: Implements dynamic batching and supports experimental int8 and fp8 precision.

Related

  • Project
    Luce-Org/lucebox

    Lucebox is an open‑source C++ inference engine for large language models that uses speculative pre‑fill and custom GPU kernels (CUDA/HIP) to dramatically speed up token generation on consumer and mixed‑GPU setups. It supports many models (Qwen, DeepSeek, Gemma, etc.), offers vision input, and provides ready‑made hardware‑specific profiles for AMD and NVIDIA cards, delivering hundreds of tokens per second on a single GPU and even higher throughput in heterogeneous configurations.

  • Project
    NVIDIA/gpu-operator

    An automation tool for Kubernetes that manages the installation and lifecycle of NVIDIA GPU drivers, runtimes, and monitoring tools as containers.

  • Project
    ModelTC/LightX2V

    LightX2V is an open‑source inference framework for fast image and video generation. It supports many models (MiniMax‑H3, LTX‑2, HunyuanVideo‑1.5, Wan‑2.x, Qwen‑Image, etc.) and offers speed‑up tricks like 4‑step distilled LoRAs, FP8/NVFP4 quantisation, tensor‑parallelism, and disaggregated deployment on a wide range of accelerators. Users interact via a single `LightX2VPipeline` class, choosing a model, variant and a JSON config, then call `generate` with a task (e.g., t2av, t2v, i2i). Reported benchmarks show up to ~4× speedup over Diffusers on H100 GPUs. The project provides Docker images, a macOS app, online demo, and extensive docs and examples.

  • Project
    thu-pacman/chitu

    A production-grade high-performance inference engine that supports diverse hardware backends and scalable deployment for large language models.