InternLM/lmdeploy

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

What it solves

LMDeploy is a toolkit designed to simplify and accelerate the process of compressing, deploying, and serving Large Language Models (LLMs) and Vision Language Models (VLMs). It addresses the challenge of high computational costs and latency in LLM inference by providing high-performance engines and quantization techniques to increase throughput and reduce memory usage.

How it works

The project provides two distinct inference engines:

  • TurboMind: Focused on ultimate performance optimization, utilizing features like persistent batching (continuous batching), blocked KV cache, tensor parallelism, and high-performance CUDA kernels.
  • PyTorch Engine: A Python-based engine designed to lower the barrier for developers and allow for rapid experimentation with new features.

It also implements effective quantization (weight-only and KV cache quantization) to reduce the model size and speed up inference, and includes a request distribution service for deploying multi-model services across multiple machines and GPUs.

Who it’s for

This toolkit is for developers and AI engineers who need to deploy LLMs or VLMs in production environments where high request throughput and low latency are critical, as well as researchers who want to quickly experiment with new models.

Highlights

  • High Throughput: Delivers up to 1.8x higher request throughput than vLLM in certain scenarios.
  • Broad Model Support: Supports a vast array of LLMs (e.g., Llama 3.1, Qwen, DeepSeek, Mistral) and VLMs (e.g., InternVL, Qwen-VL, LLaVA).
  • Advanced Quantization: Supports 4-bit inference and AWQ, with performance up to 2.4x faster than FP16.
  • Flexible Deployment: Supports multi-model, multi-machine, and multi-card inference services via a proxy server.
  • Hardware Compatibility: Compatible with NVIDIA GPUs and Huawei Ascend platforms.