InternLM/lmdeploy

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

What it solves

LMDeploy addresses the challenges of running large language models (LLMs) and vision-language models (VLMs) efficiently. It provides a toolkit to compress models, speed up their inference (the process of generating text), and serve them as scalable services across single or multiple machines.

How it works

LMDeploy offers two distinct inference engines: TurboMind, which is highly optimized for maximum performance using custom CUDA kernels, and a PyTorch-based engine designed for ease of use and rapid experimentation. It achieves high throughput through techniques like continuous batching, blocked KV cache, and tensor parallelism. Additionally, it supports advanced model quantization (such as 4-bit weight-only and KV cache quantization) to reduce memory usage and increase speed without significant quality loss.

Who it’s for

It is for developers and engineers who need to deploy LLMs or VLMs into production environments. It is suitable for those who want to maximize hardware utilization—whether on NVIDIA GPUs or Huawei Ascend platforms—or those who need a flexible, Python-native engine to quickly test and integrate new models.

Highlights

  • High-performance inference engines (TurboMind and PyTorch) optimized for throughput.
  • Supports a wide range of LLMs and VLMs, including Llama, Qwen, DeepSeek, and InternVL series.
  • Advanced quantization support (AWQ, 4-bit, FP8) to significantly boost performance over standard FP16.
  • Flexible deployment options, including multi-model, multi-machine, and multi-card serving.
  • Easy integration with popular model hubs like HuggingFace, ModelScope, and openMind.

Related

  • Project
    PaddlePaddle/FastDeploy

    A production-grade deployment toolkit for LLMs and VLMs based on PaddlePaddle, offering high-performance inference and broad hardware compatibility.

  • Project
    UbiquitousLearning/mllm

    A fast, lightweight multimodal LLM inference engine for mobile and edge devices that enables local execution of VLMs and LLMs across various NPUs and CPUs.

  • Project
    EvolvingLMMs-Lab/lmms-engine

    A unified, high-performance training engine for multimodal models that integrates distributed training, kernel fusion, and advanced optimizers to enable scalable pretraining and fine-tuning.

  • Project
    ztxz16/fastllm

    A high-performance LLM inference engine designed for local and server deployment, supporting mixed-device execution (GPU/CPU/Disk) and advanced optimization techniques like speculative decoding.

  • Project
    llm-d/llm-d

    A high-performance distributed inference serving stack for Kubernetes that optimizes LLM production deployments across various hardware accelerators.