Hetzner Inference Experimental API

Hetzner is Testing LLM Inference

Hetzner has launched an experimental LLM inference service to evaluate user demand, system scalability, and the types of loads the infrastructure can handle. This is not a production-ready product; it currently operates without billing, SLAs, or production guarantees, serving as a trial to determine if the company should formally enter the open-weight inference market.

API Specifications and Model Availability

The Hetzner Inference API is designed for ease of integration by maintaining compatibility with the OpenAI API. Users can generate an API token via the Hetzner Experiments dashboard and point their OpenAI clients to Hetzner's base URL.

Current Model: Qwen3.6-35B-A3B-FP8

Currently, the service offers a single model: Qwen/Qwen3.6-35B-A3B-FP8. This is a 35-billion-parameter Mixture-of-Experts (MoE) model with 3 billion active parameters. Key technical specifications include:

• Capabilities: Supports both text and image inputs.
• Context Window: 262K tokens.
• Quantization: Uses FP8-quantized weights.

Performance Benchmarks

Preliminary tests conducted on July 23, 2026, indicate high performance for small-scale requests:

• Time to First Token (TTFT): 153 ms median across seven short requests on an open connection.
• Throughput: 224 output tokens per second across five longer generations (capped at 512 tokens).

Strategic Implications for Infrastructure

Hetzner's entry into inference suggests a strategy to leverage its existing cost-efficiency in data center operations to commoditize open-weight model hosting. Because open-weight inference is a commodity market, the primary competitive advantages are the ability to operate GPU hardware cheaply and maintain high hardware utilization.

GPU Utilization and Revenue

An inference API allows Hetzner to share GPU capacity across multiple users, which is more efficient than renting bare-metal GPUs to single customers. This allows the company to turn spare GPU capacity into revenue more effectively.

The Hardware Bottleneck

A critical question remains regarding whether Hetzner will invest in high-density multi-GPU systems. Their current public dedicated GPU lineup consists of workstation-class GPUs:

• NVIDIA RTX 4000 SFF Ada Generation (20 GB VRAM)
• NVIDIA RTX PRO 6000 Blackwell Max-Q (96 GB VRAM)

While these are sufficient for small to medium models like Qwen, they are inadequate for massive models (e.g., GLM-5 with 754 billion parameters) that require B200/B300-class hardware and high-speed interconnects. The long-term viability of Hetzner as a major inference provider depends on whether they move beyond workstation GPUs to enterprise-grade GPU clusters.

Community Perspectives and Market Context

Industry observers and users on Hacker News have highlighted several key points regarding this experiment:

• EU Sovereignty and Compliance: There is significant interest in a highly respected, EU-native inference provider to satisfy GDPR and regulatory requirements, as most current offerings are based in the US or China.
• Market Commoditization: Some users expect this to lead to a "reseller" model for inference, similar to traditional web hosting, making AI compute more affordable for smaller players.
• Regional Specialization: One theory suggests a emerging division of labor where the US leads in proprietary models, China leads in open-weight models, and Europe focuses on the most efficient inference services.

"It would certainly be interesting to have a highly respected EU-native inference provider, if only to make the regulatory gods happy"

"Will this be the new division of labor? Americans - best proprietary models Chinese - best open weight models Europeans - best / most efficient inference service"