Hugging Face Production Infrastructure Alerting Strategies

Hugging Face employs a robust monitoring and alerting system to ensure the stability and scalability of its production infrastructure. By implementing targeted alerts for network traffic, log pipelines, and orchestration API health, the team identifies potential issues before they escalate into major incidents.

High NAT Gateway Throughput and Cost Optimization

Hugging Face uses a NAT (Network Address Translation) gateway to centralize outbound traffic from private networks to the public internet, providing a strategic vantage point for security and cost analysis. To manage cloud expenses, the team monitors network traffic volume using a static threshold alert that triggers when throughput surpasses predefined limits.

This alerting mechanism serves two primary functions:

  • Early Warning System: It detects unusual traffic spikes that may indicate unexpected system behavior.
  • Infrastructure Review: It prompts regular reviews of traffic trends to align infrastructure growth with evolving needs.

In practice, this alert has helped the team optimize configurations when third-party security and autoscaling tools increased telemetry data egress. It also identifies instances where traffic mistakenly avoids low-cost private paths. For example, fetching objects directly from LFS repository storage is more cost-effective than using CDN-hosted assets. To resolve such issues, Hugging Face leverages DNS overrides via the CDKTF AWS provider to route traffic through private network paths.

End-to-End Log Archival Validation

Hugging Face utilizes a sophisticated logging pipeline to capture Hub model usage data. The pipeline flows from Filebeat (log collection) to Logstash (transformation and enrichment) to Elasticsearch (storage and analysis), and finally to AWS object storage (long-term archival in Parquet format) for querying via Amazon Athena.

Despite its design, the pipeline is susceptible to failures including:

  • Elasticsearch Backpressure: High ingress or intensive querying can lead to log rejection or delays.
  • Schema Mismatches: Significant changes in application log field types can cause auto-schema detection to fail, leading to write errors.
  • Memory Exhaustion: Limited memory resources for Logstash and Filebeat can lead to Out-of-Memory (OOM) crashes during backpressure events.
  • Archival Failures: Resource-intensive archival jobs can fail due to node capacity limits or unusually large log entries.

To mitigate these risks, Hugging Face implemented an alert that validates the end-to-end log flow by comparing the number of requests received by the Application Load Balancer (ALB) with the number of logs successfully archived. This ratio ensures that the volume of data entering the system matches the volume archived, allowing the team to detect log loss during infrastructure refactoring or archival job failures.

Kubernetes API Health and Rate Limiting

Because the Kubernetes API acts as the central orchestration point for container management, Hugging Face monitors API error rates and rate limiting metrics to prevent cascading failures. The infrastructure heavily relies on the kube-rs library, which provides Rust-based abstractions for reflectors, controllers, watchers, and finalizers.

Key applications of kube-rs at Hugging Face include:

  • Automated Certificate Management: Using the kube::api:: module to manage HTTPS certificates for custom domains in the Spaces product.
  • Custom Controllers: Using the kube::runtime:: module to build controllers for billing management, where watchers and finalizers track customer pods for accurate resource billing.
  • Maintenance Operations: Facilitating node draining and termination during infrastructure updates.

Monitoring the Kubernetes API is critical because disruptions can impact the management of customer pods and cloud networking resources. For instance, the team used these alerts to identify a third-party tool bug that repeatedly requested a node be drained, triggering API rate limiting before the issue could degrade user experience.

Detecting Silent Cluster Metric Failures

To manage a dynamic environment where clusters are frequently created and destroyed, Hugging Face uses a specific Prometheus query to detect clusters that are not sending metrics to the central Grafana Mimir store.

The alert monitors the container_network_transmit_packets_total metric from the local Prometheus instance. The query triggers if:

  1. A new cluster is added but has not yet sent any metrics.
  2. A cluster has existed for more than 48 hours but has never sent any metrics.

This is achieved by comparing the current rate of packet transmission against a historical rate from 48 hours prior. If both the current and historical rates are zero, the alert fires, notifying the team of crashes in the metrics infrastructure or failures during cluster setup.

Sources