Snorkel AI and Hugging Face Integration for Enterprise Foundation Models

TL;DR

Snorkel AI and Hugging Face have partnered to integrate Hugging Face Inference Endpoints into the Snorkel Flow platform. This collaboration allows enterprises to access a vast library of open-source foundation models and adapt them to specific business use cases using Snorkel's data-centric programmatic labeling, reducing the cost and complexity of hosting and fine-tuning large-scale models.

Adapting Foundation Models via Snorkel Flow

Snorkel Flow enables enterprises to adapt foundation models for specific production use cases through an iterative "detect and correct" development process. This workflow begins with inspecting the "out of the box" predictions of a selected foundation model on company data, which serve as initial training labels.

Users identify error modes in the model and correct them using programmatic labeling, which involves updating training labels via heuristics or prompts. The base foundation model is then fine-tuned on these updated labels and re-evaluated. This cycle continues until the model reaches the required quality for deployment.

Integration of Hugging Face Models and Infrastructure

Snorkel AI leverages Hugging Face to provide users with access to more than 150,000 open-source models, including domain-specific models like BioBERT and SciBERT. This variety allows users to select base models that best align with their specific business needs, providing a jump-start for initial predictions and fine-tuning.

To manage the infrastructure and costs associated with these models, Snorkel AI utilizes Hugging Face Inference Endpoints. This service provides several key technical advantages:

  • Rapid Deployment: Users can create a model API in a few clicks for immediate use.
  • Cost Efficiency: The "pause and resume" capability allows Snorkel to activate model APIs only when a client needs them and put them to sleep during inactivity.
  • Flexible Configuration: The service offers options for cloud provider selection and security levels to meet enterprise requirements.

Enterprise Implications for AI Development

This partnership addresses the primary barriers enterprises face when adopting foundation models: the high cost of self-hosting, the lack of resources to train models from scratch, and the risks associated with using off-the-shelf models for production without governance.

By combining Snorkel's automated data labeling with Hugging Face's open-source ecosystem, companies can build data-centric AI applications without needing the massive datasets typically required for manual fine-tuning. As Clement Delangue, co-founder and CEO of Hugging Face, stated:

"With Snorkel AI and Hugging Face Inference Endpoints, companies will accelerate their data-centric AI applications with open source at the core."

This integration allows organizations to maintain control over the customer experience and the specific solution tailored to their use case while leveraging the open-source community's advancements.

Sources