runpod-workers/worker-vllm

What it solves

It simplifies the deployment of high-performance Large Language Model (LLM) endpoints on Runpod Serverless. It removes the need to manually configure the vLLM inference engine for serverless environments, providing a ready-to-use worker that is fully compatible with the OpenAI API.

How it works

The project provides a Docker-based worker that wraps the vLLM inference engine. Users can deploy it in two ways: using a pre-built image and configuring the model via environment variables (like MODEL_NAME and QUANTIZATION), or by building a custom Docker image with the model weights baked in for faster startup. It supports various vLLM engine arguments through environment variables and can be configured via a config.yaml file. Once deployed, it exposes an OpenAI-compatible API, allowing users to swap their existing OpenAI base URL and API key for the Runpod endpoint.

Who it’s for

Developers and AI engineers who want to host their own open-source LLMs on serverless infrastructure with minimal setup and full compatibility with existing OpenAI-based codebases.

Highlights

• OpenAI Compatibility: Supports Chat Completions, Models, Responses, and Messages routes with both streaming and non-streaming options.
• Flexible Deployment: Offers both pre-built images for quick setup and "baked-in" model images for optimized loading.
• Deep vLLM Integration: Automatically discovers and applies AsyncEngineArgs from environment variables, allowing full access to vLLM's configuration options.
• Broad Model Support: Compatible with any model architecture supported by the vLLM engine on Hugging Face.