Hugging Face Models on Foundry Managed Compute

Hugging Face Models on Foundry Managed Compute

TL;DR

Microsoft has announced Foundry Managed Compute and the integration of Hugging Face models on Foundry. This allows enterprises to deploy a curated catalog of open-weight models from the Hugging Face ecosystem onto a managed GPU platform-as-a-service with one click, inheriting the same security, governance, and observability as frontier models on the Foundry platform.

Microsoft Foundry and Managed Compute

Microsoft Foundry is a platform designed for building and operating agentic AI applications. It provides a unified endpoint and SDKs (Python, C#, JavaScript, and Java) to access a wide range of models from providers including Microsoft, OpenAI, Anthropic, Meta, Mistral, DeepSeek, and Hugging Face.

Foundry Agent Service

Foundry includes the Foundry Agent Service, which provides:

  • Multi-agent orchestration with built-in memory.
  • Knowledge grounding via Foundry IQ.
  • Connectable tools using agentic protocols for enterprise data access.
  • Observability loops, including end-to-end tracing, real-time monitoring, continuous evaluations, and a prompt optimizer.

Enterprise Guardrails and Security

Developers have access to integrated content safety filters, task-adherence guardrails, an AI Red Teaming Agent for adversarial testing, unified RBAC, private networking, and Azure Policy integration.

Foundry Managed Compute

Managed Compute is a managed GPU platform-as-a-service for open-source and custom models. It abstracts the underlying GPU topology, allowing developers to deploy based on model parameters, context length, and optimization goals (latency vs. throughput). Microsoft automatically handles container updates, runtime upgrades, and security patches for supported runtimes, including vLLM, SGLang, TensorRT-LLM, NIM, TEI, and llama.cpp.

The Hugging Face Collection on Foundry

Integrating Hugging Face models into Foundry provides an operational layer for open-weight models, enabling enterprises to leverage the flexibility of open AI while maintaining corporate security standards.

Key Capabilities of Open Models

Open-weight models allow for deep customization (fine-tuning, distillation, quantization, LoRA), precise version control, and cost shaping by paying for accelerators by the hour and scaling to zero when idle.

The Curation Pipeline

To ensure enterprise readiness, Hugging Face and Microsoft use a systematic five-step curation process:

  1. Identification: Trending models are selected based on community signals and customer demand.
  2. Compliance and Security Screening: Licenses are reviewed against Microsoft's enterprise distribution policy, and repositories are inspected to remove trust_remote_code patterns.
  3. Runtime Building: Microsoft builds, scans for CVEs, and signs inference container images.
  4. Secure Weight Storage: Weights are pulled from Hugging Face and stored in Microsoft-managed Azure storage to eliminate the need for outbound network access to the Hugging Face Hub during deployment.
  5. Validation: Every model, runtime, and accelerator combination is tested for API conformance and performance (latency, throughput, TTFT) before being published to the catalog.

Model Runtimes and Modalities

Foundry supports a diverse array of modalities—including text, vision, audio, and multimodal—using specialized runtimes matched to the model architecture:

  • vLLM: The default high-throughput engine for open LLMs. Due to Hugging Face's contribution to vLLM, Transformers library models can often be served on Foundry the same day they land on Hugging Face.
  • SGLang: Optimized for language and multi-modal models, specifically supporting structured outputs (JSON, regex) essential for agentic workloads.
  • Text Embeddings Inference (TEI): Used for embedding, reranker, and sequence-classification models to optimize RAG and semantic search.
  • llama.cpp: Provides a path for GGUF-quantized models on CPUs or small GPUs for cost-optimized deployments.
  • TensorRT-LLM and NIM: Utilized on NVIDIA hardware for superior latency and throughput for specific model families.
  • hf-serve: Hugging Face's multi-model inference server for architectures outside the LLM/embedding fast paths, such as vision and audio.

Deployment and Scoring Workflow

Deploying an open-weight model from the Hugging Face Collection involves a five-step process: browsing the catalog, choosing a deployment template, configuring instance count, deploying (via portal, CLI, SDK, or REST), and scoring via the unified Foundry endpoint.

Deployment Templates

Deployment templates are versioned assets that pin the runtime, accelerator family, context length, and tuning settings. For example, a qwen3-32b model may offer templates for different NVIDIA A100 or H100 configurations and context lengths (40K vs 128K).

Integration

Deployments are reachable through the unified Foundry endpoint using the OpenAI SDK. These models can be slotted into Foundry Agents as admin-connected models, sharing the same authentication, endpoint, and observability as frontier models.

Sources