Active Learning with AutoNLP and Prodigy

Hugging Face has demonstrated a workflow for implementing active learning in Machine Learning, using AutoNLP for automated model training and Prodigy for iterative data labeling. This pipeline allows users to create state-of-the-art (SOTA) models by focusing human effort on data labeling rather than model architecture and hyperparameter tuning.

Automated Model Training with AutoNLP

AutoNLP is a framework designed to build SOTA deep learning models with minimal coding. It leverages Hugging Face's existing ecosystem, including the transformers, datasets, and inference-api libraries.

Key capabilities of AutoNLP include:

  • Broad Task Support: It supports binary classification, regression, multi-class classification, token classification (such as NER or part-of-speech tagging), question answering, and summarization.
  • Multilingual Support: It supports languages including English, French, German, Spanish, Hindi, and Dutch, with options for custom models and tokenizers for unsupported languages.
  • Automated Optimization: AutoNLP automatically handles model selection, hyperparameter tuning, and the training of multiple model candidates to identify the best performer.

Real-time Data Annotation with Prodigy

Prodigy, developed by Explosion, is a commercial web-based annotation tool that enables real-time data labeling. While primarily used for NLP tasks like NER and text classification, it also supports Computer Vision and custom task creation. In the active learning pipeline, Prodigy serves as the interface where humans iteratively label raw text to create the training sets required by AutoNLP.

Case Study: BBC News Classification and NER

To demonstrate the pipeline, Hugging Face applied these tools to the BBC News Classification dataset from Kaggle, which contains news articles categorized into five classes: business, entertainment, politics, sport, and tech.

Multi-class Classification

Using AutoNLP, a classification model was trained on the existing categories. The process involved uploading the dataset and allowing AutoNLP to train 15 different models. The resulting best model achieved an accuracy of 98.67%.

Named Entity Recognition (NER) via Active Learning

Because the BBC dataset lacked entity labels, an active learning process was implemented to add NER capabilities (labels: PERSON, ORG, PRODUCT, LOCATION):

  1. Initial Labeling: A small set of approximately 20 samples was labeled in Prodigy. An initial AutoNLP model showed 86% accuracy but 0 precision and recall, indicating the model had not yet learned the patterns.
  2. Iterative Growth: As the labeled dataset grew to 70 samples, accuracy rose to 92%, with precision at 0.52 and recall at 0.42.
  3. Performance Breakthrough: At approximately 150 samples, the model reached 95.7% accuracy, 0.64 precision, and 0.76 recall, showing a significant improvement in detecting entities in unseen text.
  4. Final Optimization: After labeling approximately 250 samples, the process reached a plateau of optimal results with ~95.9% accuracy, 0.73 precision, and 0.79 recall.

Technical Implementation and Workflow

Since Prodigy does not export data directly in AutoNLP format, a custom Python script using spacy and prodigy.components.db was used to convert Prodigy annotations into a JSONL file. This file is then uploaded to AutoNLP under the "Token Classification" task.

This workflow shifts the technical burden from model engineering to data quality. By using AutoNLP, developers avoid the need to manually select optimizers, schedulers, or perform manual pre-processing and post-processing, allowing them to focus entirely on the quality of the labeled data.

Sources