Hugging Face and Protect AI Security Partnership: 6-Month Progress Report

Hugging Face and Protect AI partnered in October 2024 to integrate Protect AI's Guardian scanning technology into the Hugging Face Hub. This partnership aims to secure the machine learning (ML) supply chain by providing developers with inline security alerts and comprehensive vulnerability reports via Insights DB.

Scanning Scale and Impact

As of April 1, 2025, the partnership has achieved the following scale in model security scanning:

  • Total Models Scanned: 4.47 million unique model versions across 1.41 million repositories.
  • Threats Identified: 352,000 unsafe or suspicious issues found across 51,700 models.
  • Performance: 226 million requests served in the last 30 days with an average response time of 7.94 ms.

New Threat Detection Modules

Protect AI has launched four new detection modules to expand the range of covered model file formats and identify sophisticated obfuscation techniques:

  1. PAIT-ARV-100: Detects archive slips that can write to the file system at load time.
  2. PAIT-JOBLIB-101: Detects suspicious code execution in Joblib models at load time.
  3. PAIT-TF-200: Identifies architectural backdoors within TensorFlow SavedModels.
  4. PAIT-LMAFL-300: Detects malicious code execution during inference in Llamafile formats.

Additionally, Guardian now detects the high-severity CVE-2025-1550 vulnerability in Keras.

Zero Trust Approach to Model Security

Guardian employs a zero trust approach, treating arbitrary code execution as inherently unsafe regardless of the user's intent. This strategy is necessary because attackers often use obfuscation—such as compression, encoding, and serialization—to hide malicious payloads within seemingly benign scripts or framework extensibility components. By flagging execution risks as "suspicious" on InsightsDB, Guardian mitigates risks that would be otherwise undetectable through simple payload inspection.

Common Attack Themes in ML Models

Through research and the huntr bug bounty program—which has provided over 200 reports—Protect AI has identified several recurring attack vectors:

Library-Dependent Attack Chains

These attacks leverage functions from common ML libraries (e.g., PyTorch, Numpy, Pandas) present in the user's environment. The impact is proportional to the library's popularity; for instance, PyTorch's torchvision.io functions can be exploited to overwrite or delete files on a victim's system.

Payload Obfuscation

Attackers use compression and serialization to bypass scanners. Formats like Joblib and Keras can embed compressed payloads or nested archives, exposing users to TarSlip or ZipSlip vulnerabilities, which can lead to Denial of Service or Arbitrary Code Execution via path traversal.

Framework-Extensibility Vulnerabilities

ML frameworks provide mechanisms like custom layers and configuration-based code loading that create attack vectors. CVE-2025-1550 in Keras demonstrates how custom layers can be exploited for arbitrary code execution despite existing security features.

Attack Vector Chaining

Sophisticated attackers combine multiple vulnerabilities (e.g., combining an obfuscated payload with a framework extension mechanism) to create complex compromise pathways that appear benign when viewed in isolation.

Enhanced Detection Capabilities

Guardian has evolved its detection capabilities to address these threats:

  • Deep Structure Analysis: Scanners for PyTorch and Pickle now examine execution paths to identify malicious patterns triggered by library dependencies.
  • Multi-layered Analysis: Guardian decompresses nested archives and examines compressed payloads to uncover hidden malicious code in formats like Joblib and Keras.
  • Framework Analysis: Detection modules now analyze ML framework extension mechanisms to block dangerous implementations and identified CVE-2025-1550 before public disclosure.
  • Expanded Format Support: Coverage now includes Joblib, TensorFlow (for architectural backdoors), and Llamafile.

Community-Driven Security

Protect AI's threat research is augmented by huntr, the AI/ML bug bounty program. With over 17,000 security researchers, the community has contributed over 200 reports on model file vulnerabilities, which are automatically integrated into the Guardian scanning process on the Hugging Face Hub.

Sources