OpenNMT/CTranslate2

Fast inference engine for Transformer models

What it solves

CTranslate2 is designed to make the inference of Transformer models faster and more memory-efficient. It addresses the high resource consumption and slow execution speeds typical of general-purpose deep learning frameworks when deploying Transformer-based models in production environments.

How it works

The library implements a custom runtime that applies several performance optimization techniques, including:

  • Weight Quantization: Reducing the precision of weights to 16-bit (FP16, BF16, INT16) or 8-bit (INT8) and 4-bit (AWQ INT4), which reduces memory usage and speeds up computation.
  • Layer Fusion: Combining multiple operations into a single kernel to reduce overhead.
  • Batch Reordering and Padding Removal: Optimizing how data is processed to avoid unnecessary computations on padding tokens.
  • Hardware Optimization: Integrating optimized backends for various CPU architectures (x86-64, ARM64) using libraries like Intel MKL, oneDNN, and Apple Accelerate.
  • Tensor Parallelism: Allowing very large models to be split across multiple GPUs for distributed inference.

Models must first be converted from frameworks like PyTorch (Transformers), Fairseq, or OpenNMT into an optimized format specific to CTranslate2.

Who it’s for

It is intended for developers and engineers who need to deploy Transformer models (including LLMs like Llama, Mistral, and Gemma) for high-performance, production-oriented inference on both CPU and GPU.

Highlights

  • Broad Model Support: Supports encoder-decoder, decoder-only (LLMs), and encoder-only models.
  • C++ and Python APIs: Provides simple integration via Python and C++ interfaces.
  • Low Resource Footprint: Quantization can reduce model size on disk by up to 4 times with minimal accuracy loss.
  • Llama/Mistral/Gemma Support: Optimized for modern decoder-only architectures.
  • OpenAI-Compatible API: An optional web server wrapper allows integration with applications that already support the OpenAI API.

Related

  • Project
    NVIDIA/TensorRT-LLM

    TensorRT‑LLM is NVIDIA’s open‑source library that compiles large language (and visual‑generation) models into ultra‑fast TensorRT inference engines. It provides custom CUDA kernels, quantization, sparse attention, speculative decoding, and multi‑GPU parallelism, delivering 2‑10× higher token throughput while keeping accuracy. The Python‑friendly runtime integrates with Triton, Kubernetes, and Jetson, and includes docs and examples for models like GPT‑OSS‑120B, Llama‑4, DeepSeek‑R1, and diffusion models.

  • Project
    alibaba/rtp-llm

    A high-performance LLM inference acceleration engine developed by Alibaba that optimizes throughput and latency for production-scale deployments.

  • Project
    amd/RyzenAI-SW

    A suite of tools and runtime libraries for optimizing and deploying AI inference on AMD Ryzen AI-based PCs to maximize hardware acceleration.

  • Project
    InternLM/lmdeploy

    LMDeploy is a comprehensive toolkit for compressing, deploying, and serving large language and vision models, designed to maximize inference speed and hardware efficiency.

  • Project
    facebookincubator/AITemplate

    A Python framework that transforms deep neural networks into optimized CUDA/HIP C++ code for high-performance inference serving on NVIDIA and AMD GPUs.