Hugging Face Jobs: Deploying vLLM Servers with a Single Command

Hugging Face Jobs: Deploying vLLM Servers with a Single Command

Hugging Face has introduced a method to spin up private, OpenAI-compatible LLM endpoints using vLLM on Hugging Face infrastructure with a single command. This approach eliminates the need for manual server provisioning or Kubernetes management, offering a pay-per-second billing model ideal for rapid testing, evaluations, and batch generation.

Deploying a vLLM Server via HF Jobs

Users can launch a vLLM server using the hf jobs run command, which functions similarly to docker run for Hugging Face infrastructure. By utilizing the official vllm/vllm-openai image and specifying hardware via the --flavor flag, users can expose the server to the public internet through Hugging Face's jobs proxy.

Example launch command:

hf jobs run --flavor a10g-large --expose 8000 --timeout 2h \
  vllm/vllm-openai:latest \
  vllm serve Qwen/Qwen3-4B --host 0.0.0.0 --port 8000

Key parameters include:

  • --expose 8000: Routes the container's port through the public jobs proxy.
  • --flavor: Specifies the GPU hardware (e.g., a10g-large).
  • --timeout: Sets a safety net to automatically stop the job after a specified duration.

Once started, the system provides a unique job ID and a reachable URL (e.g., https://<job_id>--8000.hf.jobs).

Querying and Authentication

Because vLLM implements the OpenAI API, the deployed endpoints can be queried using standard OpenAI clients or curl. All requests must include a Hugging Face token as a bearer token for authentication.

Authentication Requirements:

  • The endpoint is gated and not public.
  • Requests require an HF token with read access to the job's namespace.
  • Access is scoped to the user or their organization.

Scaling to Large Models

HF Jobs supports scaling to larger models by combining beefier hardware flavors and vLLM's tensor parallelism. For example, deploying a 122B parameter model like Qwen3.5-122B-A10B on two H200 GPUs requires the --tensor-parallel-size 2 flag to shard the model across GPUs.

For very large models, memory management is critical. The source notes that capping the context length (--max-model-len) and concurrent sequence count (--max-num-seqs) is the primary solution if the model fails to start due to out-of-memory (OOM) or cache-block errors.

Advanced Operational Capabilities

Interactive Debugging via SSH

Users can open a shell directly into a running job by adding the --ssh flag during launch. This allows for real-time monitoring using tools like nvidia-smi and interactive log inspection.

hf jobs ssh <job_id>

Integration with Coding Agents

Deployed vLLM servers can serve as backends for terminal coding agents like Pi. To enable this, the server must be launched with tool calling enabled via --enable-auto-tool-choice and a specific --tool-call-parser (e.g., hermes for Qwen3 models).

HF Jobs vs. Inference Endpoints

Hugging Face provides two primary ways to serve models, depending on the required level of control and stability:

Feature HF Jobs Inference Endpoints
Primary Use Case Experiments, one-off evals, batch generation Production-ready, long-lived services
Control Maximum flexibility (pick image, flags, hardware) Managed operational experience
Billing Pay-per-second for job duration Scale-to-zero (no billing during inactivity)
Access Control Gated via HF token/namespace Finer-grained (public, protected, or private)

Prerequisites for Use

To use HF Jobs for vLLM deployment, users need:

  • A payment method or positive prepaid credit balance.
  • huggingface_hub >= 1.20.0 installed via pip.
  • Local authentication via hf auth login.

Sources