Hugging Face Secrets Management Scaling with Infisical

Hugging Face has migrated its secrets management to Infisical to centralize sensitive configuration data across its multi-cloud infrastructure. This transition addresses the risks of secret sprawl and manual rotation burdens while providing a developer-friendly workflow that integrates with Kubernetes, CI/CD pipelines, and local development environments.

Centralizing Multi-Cloud Secrets Management

Hugging Face transitioned from an AWS-only setup to a multi-cloud environment including Azure and GCP. To avoid the complexity of heavyweight solutions like HashiCorp Vault, the team adopted Infisical to provide a multi-cloud abstraction and a centralized system for managing credentials.

This migration solved several critical infrastructure challenges:

  • Secret Sprawl: Inconsistent management across different environments increased security risks.
  • Permission Complexity: Scaling the team required tighter role-based access controls (RBAC) integrated with Okta SSO.
  • Local Development Friction: The use of traditional .env files compromised security and slowed down developer productivity.
  • Manual Rotation Burden: A previous security incident involving exposed credentials highlighted the need for automated secret rotation.

Kubernetes Integration and Automation

Kubernetes serves as the core of Hugging Face's production environment. The team utilizes the Infisical Kubernetes Operator to automate the synchronization of secrets between the Infisical platform and Kubernetes objects.

Automated Secret Syncing

The Infisical Operator monitors for changes in the Infisical platform and propagates updates to corresponding Kubernetes secrets. This is achieved through the InfisicalSecret Custom Resource Definition (CRD), which maps a specific Infisical secret ID to a target Kubernetes secret name.

Deployment Control

While the Infisical Operator can automatically trigger container reloads when secrets change, Hugging Face engineers typically opt for manual redeployments. This decision is necessary to maintain precise control over deployments in an environment handling over 10 million requests per minute across numerous replicas.

Local Development and CI/CD Workflows

To eliminate insecure local .env files, Hugging Face uses the Infisical CLI to inject secrets directly into development environments. This ensures that local configurations remain consistent with production standards and reduces onboarding friction for new engineers.

For CI/CD, Infisical is integrated into GitHub Actions using OIDC authentication and Terraform. By utilizing self-hosted runners within a secure environment, the team ensures that every deployment adheres to production-grade security standards.

Security and Access Management

Security is reinforced through a fine-grained RBAC system integrated with Okta. Permissions are automatically mapped from Okta groups, allowing developers to maintain administrative rights over their specific projects while restricting read/write access for frontend and backend teams.

Additionally, the platform provides secure credential sharing for ML/AI researchers and simplifies auditing and secret rotation processes, which are critical for rapid incident response.

Technical Outcomes

Centralizing secrets management with Infisical resulted in several key operational improvements:

  • Reduced Configuration Time: Self-serve workflows eliminated the need for engineers to manually configure environment secrets.
  • Enhanced Incident Response: Automated audits and fine-grained access controls enabled a "shift left" approach to security.
  • Infrastructure Consistency: Uniform integration across cloud providers, Kubernetes clusters, and CI/CD pipelines eliminated discrepancies in secret management.

As Adrien Carreira, Head of Infrastructure at Hugging Face, stated:

"Infisical provided all the functionality and security settings we needed to boost our security posture and save engineering time. Whether you're working locally, running kubernetes clusters in production, or operating secrets within CI/CD pipelines, Infisical has a seamless prebuilt workflow."

Sources