spaCy Integration with Hugging Face Hub

Hugging Face has integrated the spaCy library into the Hugging Face Hub, enabling users to share, discover, and deploy spaCy NLP pipelines using a standardized infrastructure. This integration streamlines the path from prototyping to production by providing a centralized repository for spaCy models and simplified deployment options.

Simplified Model Distribution and Installation

Users can now install spaCy models directly from the Hugging Face Hub using pip install. This removes the need for manual downloads and simplifies dependency management. Models can be loaded into spaCy using standard methods:

# Using spacy.load()
import spacy
nlp = spacy.load("en_core_web_sm")

# Importing as module
import en_core_web_sm
nlp = en_core_web_sm.load()

To facilitate this, the Hub provides a "Use in spaCy" button on model repositories, which generates a working code snippet for installation and loading.

Model Discovery and Hub Capabilities

The Hugging Face Hub now hosts over 60 canonical models from the spaCy organization, based on the spaCy 3.1 release. In addition to official models, community-contributed spaCy models are discoverable via a dedicated filter on the Hub.

Interactive Widgets and Inference API

The integration includes specialized widgets for Named Entity Recognition (NER), allowing users to test models interactively in the browser. While NER is supported out-of-the-box via the Hosted Inference API, support for Part-of-Speech (POS) tagging and text classification is planned for future updates.

Users can also interact with these models via HTTP requests for production use cases:

curl -X POST --data '{"inputs": "Hello, this is Omar"}' https://api-inference.huggingface.co/models/spacy/en_core_web_sm

Streamlined Model Sharing via spacy-huggingface-hub

Developers can upload their own spaCy pipelines to the Hub using the spacy-huggingface-hub library. This library extends the spaCy CLI with a huggingface-hub push command, automating the upload of the packaged model, metadata, and the generation of a model card.

The workflow for sharing a model is as follows:

  1. Authenticate via huggingface-cli login.
  2. Package the model using python -m spacy package.
  3. Push the package to the Hub using python -m spacy huggingface-hub push.

Infrastructure for Library Integration

This integration was made possible through the huggingface_hub library, which provides the necessary API and widget framework to support various libraries. Hugging Face provides a guide for other library maintainers wishing to integrate their tools into the Hub.

Sources