bentoml/OpenLLM

OpenLLM – Self‑hosted LLMs behind an OpenAI‑compatible API

What it is

• A Python package that lets you run any open‑source large language model (LLM) – from Llama 3.x to Qwen 2.5, Mistral, Phi‑4, etc. – as a local server exposing the same REST endpoints that OpenAI’s API provides.
• It bundles a lightweight chat UI, CLI tools, and Docker/Kubernetes deployment helpers, so you can spin up a production‑grade inference service with a single command.

Key capabilities

• One‑line model launch – openllm serve llama3.2:1b downloads the model (via Hugging Face), starts a vLLM‑backed inference server, and makes it reachable at http://localhost:3000.
• OpenAI‑compatible endpoints – /v1/chat/completions, /v1/completions, etc., work with any client library that talks to OpenAI (e.g., openai Python SDK, LlamaIndex).
• Built‑in chat UI – Visit http://localhost:3000/chat for an interactive web chat without writing any front‑end code.
• Extensible model catalog – A default GitHub‑hosted model repository lists supported models; you can add your own repository of BentoML‑packaged models.
• Enterprise‑ready deployment – Generates Docker images and Helm charts for Kubernetes, leveraging BentoML’s production serving stack.
• Backend flexibility – Uses the high‑performance vllm engine under the hood for fast inference on GPUs.

Getting started

pip install openllm          # install the package
openllm hello               # sanity‑check demo
openllm serve llama3.2:1b   # start a LLM server (requires HF token for gated models)

• The server listens on port 3000 and provides both API and /chat UI.
• Example Python client:

from openai import OpenAI
client = OpenAI(base_url='http://localhost:3000/v1', api_key='na')
resp = client.chat.completions.create(
    model='meta-llama/Llama-3.2-1B-Instruct',
    messages=[{'role':'user','content':'Explain superconductors like I’m five'}],
    stream=True,
)
for chunk in resp:
    print(chunk.choices[0].delta.content or '', end='')

Typical use cases

• Rapid prototyping of LLM‑powered features without paying for external API usage.
• Deploying private or regulated models on‑premises or in a cloud VPC.
• Building internal AI services that need to interoperate with existing OpenAI‑based tooling.
• Experimenting with new open‑source models by simply adding them to the model repository.

Community & contribution

• Active Slack community, forum, and GitHub issue tracker.
• Contributions welcome: add new models, improve the CLI, extend deployment scripts, or submit bug fixes.
• The project is Apache‑2.0 licensed and builds on BentoML, vLLM, and other open‑source components.

Bottom line OpenLLM abstracts away the heavy lifting of model download, GPU setup, and API design, giving developers a plug‑and‑play way to serve any open‑source LLM as if it were an OpenAI service, with optional enterprise‑grade Docker/Kubernetes deployment.