runpod-workers/worker-vllm

What it solves

This project provides a specialized worker for RunPod Serverless that allows users to deploy high-performance LLM endpoints. It simplifies the process of hosting large language models using the vLLM inference engine while ensuring the API is fully compatible with OpenAI's standards, making it easy to integrate into existing AI applications.

How it works

The worker wraps the vLLM inference engine and exposes it as a serverless endpoint. It can be deployed in two primary ways: using a pre-built Docker image where the model is specified via environment variables (like MODEL_NAME), or by building a custom Docker image with the model weights baked in for faster startup. The worker translates environment variables into vllm serve CLI flags and handles the routing of OpenAI-compatible requests (such as /v1/chat/completions) to the underlying engine.

Who it’s for

Developers and AI engineers who want to deploy LLMs on RunPod Serverless with minimal configuration, specifically those who need OpenAI-compatible APIs for their applications and want to leverage vLLM's speed.

Highlights

• OpenAI Compatibility: Fully supports OpenAI's API routes, including Chat Completions, Models, Responses, and Messages.
• Flexible Configuration: Supports extensive tuning via environment variables or a config.yaml file, including quantization methods (AWQ, GPTQ, etc.) and tensor parallelism.
• Smart Error Handling: Detects common startup failures (like GPU memory exhaustion) and returns descriptive error messages instead of entering a silent crash loop.
• Cuda Compatibility: Inherits the CUDA runtime from the official vLLM OpenAI image.