Opinion Classification with Kili and HuggingFace AutoTrain

Hugging Face has detailed a workflow for building an active learning pipeline for text classification by integrating Kili, a data-centric AI training platform, with HuggingFace AutoTrain. This approach allows developers to iteratively label data and train models with minimal coding, significantly reducing the time required to move from raw data to a production-ready classifier.

Active Learning Pipeline Overview

Active learning is an iterative process where labeled data is added to a dataset and the model is retrained to improve performance. In this implementation, the pipeline is applied to approximately 40,000 user reviews of the Medium app from the Google Play Store to categorize opinions and perform sentiment analysis.

Data Annotation with Kili

Kili is used as the end-to-end platform for creating high-quality training data. The workflow involves:

  • Project Configuration: Creating multi-class text classification projects via a web interface or Python API.
  • Label Definition: For the Medium review dataset, four primary categories were defined: Subscription, Content, Interface, and User Experience. Additional labels for "Other" and "Multi-label" were used to handle edge cases.
  • API Integration: The Kili Python API allows for programmatic project creation, data uploading in batches of 100, and updating asset properties (e.g., moving samples to a "To Review" status to correct mislabeling or bias).
  • Labeling Interface: Kili provides built-in keyboard shortcuts and a streamlined UI to accelerate the annotation process.

Automated Modeling with AutoTrain

AutoTrain automates the machine learning pipeline, including data cleaning, model selection, and hyper-parameter optimization. It is built on the transformers, datasets, and inference-api libraries.

  • Capabilities: AutoTrain supports binary and multi-label text classification, token classification, extractive question answering, text summarization, and text scoring across multiple languages.
  • Performance: In the provided example, AutoTrain achieved nearly 89% accuracy in approximately 20 minutes of training, with the entire setup process taking roughly 30 minutes.

Manual Modeling and Hyper-parameter Optimization

To provide a comparison, a manual modeling approach was implemented using the Hugging Face Trainer API and Ray Tune for hyper-parameter optimization.

Technical Implementation

  • Base Model: The cardiffnlp/twitter-roberta-base-sentiment model was selected for fine-tuning.
  • Optimization Stack: The pipeline utilized the Async Successive Halving Algorithm (ASHA) as the scheduler and HyperOpt as the search algorithm.
  • Dataset Handling: A custom TextClassificationDataset class was created to map labels to indices and handle tokenization via the AutoTokenizer.

Results and Observations

Manual tuning with 20 and 40 trials showed that model performance is highly sensitive to dataset quality. The author noted that introducing bias during the labeling phase caused performance drops, which were corrected as sample variance increased in later dataset versions.

Final Analysis and Insights

By applying the fine-tuned model to the full dataset and combining it with sentiment analysis, the following insights were derived regarding the Medium mobile application:

  • Subscription: Most reviews regarding subscriptions are negative, indicating that paid content is generally unwelcome in mobile apps.
  • Interface: A significant number of negative reviews target the interface, particularly in version 4.5, suggesting bugs or user confusion regarding specific features.
  • Content and Experience: Users generally express positive sentiment toward the articles and their overall experience with the platform.

Conclusion

Integrating Kili's annotation tools with AutoTrain provides a highly efficient path to deploying text classifiers. While manual tuning via Ray Tune offers more control, AutoTrain's ability to automate model selection and hyper-parameter optimization makes it significantly faster for establishing a baseline and iterating on data-centric AI projects.

Sources