InternLM/lmdeploy

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

What it solves

LMDeploy is a toolkit designed to simplify and accelerate the process of compressing, deploying, and serving Large Language Models (LLMs) and Vision Language Models (VLMs). It addresses the challenge of high computational costs and latency in LLM inference by providing high-performance engines and quantization techniques.

How it works

The project provides two distinct inference engines:

  • TurboMind: Focused on ultimate performance optimization, utilizing features like persistent batching (continuous batching), blocked KV cache, tensor parallelism, and high-performance CUDA kernels.
  • PyTorch Engine: A Python-based engine designed to lower the barrier for developers and allow for rapid experimentation with new features.

It also incorporates quantization methods (such as AWQ and KV cache quantization) to reduce model size and the memory footprint, and a request distribution service to manage multi-model deployments across multiple machines and GPUs.

Who it’s for

  • ML Engineers and DevOps: Those looking to deploy LLMs/VLMs in production environments with high throughput and low latency.
  • AI Researchers: Developers who need a flexible engine for rapid experimentation with new model architectures.
  • Enterprise Users: Organizations needing to scale LLM services across multiple GPUs and machines.

Highlights

  • High Throughput: Delivers up to 1.8x higher request throughput than vLLM in certain scenarios.
  • Broad Model Support: Compatible with a vast array of models including Llama, Qwen, InternLM, DeepSeek, Mistral, and Phi.
  • Multimodal Capabilities: Full support for Vision Language Models (VLMs) including InternVL and LLaVA.
  • Effective Quantization: Supports 4-bit weight-only and KV quantization, significantly increasing inference speed compared to FP16.
  • Flexible Deployment: Supports multi-model, multi-machine, and multi-card inference services.

Related

  • Project
    vllm-project/llm-compressor

    LLM Compressor is a Python library that quantizes and prunes large language models into the `compressed‑tensors` format, enabling memory‑efficient deployment with vLLM. It supports many low‑precision formats (NVFP4, FP8, INT4, etc.), several PTQ/GPTQ algorithms, DDP and disk‑offloading for huge models, and ships pre‑quantized checkpoints for popular LLMs.

  • Project
    PaddlePaddle/FastDeploy

    A production-grade deployment toolkit for LLMs and VLMs based on PaddlePaddle, offering high-performance inference and broad hardware compatibility.

  • Project
    ModelCloud/GPTQModel

    A comprehensive LLM quantization toolkit that compresses models using methods like GPTQ and AWQ with hardware acceleration for NVIDIA, AMD, Intel, and Huawei NPUs.

  • Project
    EricLBuehler/candle-vllm

    A high-performance inference and serving platform for local LLMs featuring an OpenAI-compatible API, multi-GPU/multi-node support, and aggressive KV cache compression.

  • Project
    EvolvingLMMs-Lab/lmms-engine

    A unified, high-performance training engine for multimodal models that integrates distributed training, kernel fusion, and advanced optimizers to enable scalable pretraining and fine-tuning.