Hugging Face Expert Acceleration Program: Witty Works Case Study

Witty Works collaborated with the Hugging Face Expert Acceleration Program to transition from a rule-based linguistic analysis tool to a context-dependent ML classifier for inclusive language. This shift allowed the writing assistant to accurately identify non-inclusive words based on their surrounding context rather than relying solely on vocabulary lists.

Transitioning from Linguistic Analysis to Contextual Classification

Witty Works initially developed its writing assistant using transfer learning with pre-trained spaCy models. This basic approach utilized lemmatization, linguistic analysis, and the extraction of features such as part-of-speech tags, gender, and word dependencies to highlight non-inclusive words against a knowledge base of approximately 2,300 words and idioms in English and German.

While this method was effective for 85% of the vocabulary, it failed to handle context-dependent words. For example, the word "fossil" in "Fossil fuels are not renewable resources" is inclusive, whereas in "He is an old fossil," it is non-inclusive. To solve this, Witty Works moved toward Hugging Face transformers to understand the semantic context of words.

Implementing Few-Shot Learning with SetFit

Witty Works faced a significant data bottleneck: fine-tuning a vanilla BERT model would have required hundreds of annotated samples per category, a process that was too costly and time-consuming. To resolve this, Hugging Face experts recommended the following technical shifts:

From Word Embeddings to Sentence Embeddings

Instead of extracting token embeddings for specific words, the team switched to contextualized sentence embeddings using a Sentence Transformers architecture. This approach uses Siamese and triplet network structures to ensure semantically similar sentences are closer in the vector space, allowing the resulting embeddings to serve as input for classical classifiers like K-Nearest Neighbors (KNN) or logistic regression.

Few-Shot Fine-Tuning with SetFit

Witty Works adopted SetFit (Sentence Transformer Fine-tuning), a framework that combines contrastive learning and semantic sentence similarity. This allowed the team to achieve high accuracy with only 15-20 labeled sentences per specific word, compared to the 100-200 previously estimated.

Model Selection and Deployment

To ensure low latency for a real-time browser extension, Witty Works and Hugging Face tested multiple sentence transformer models. They selected mpnet-base-v2 combined with logistic regression and KNN.

Key deployment outcomes include:

  • Infrastructure: The model was deployed on Azure.
  • Performance: The model achieved an accuracy of 0.92.
  • Efficiency: The reduction in training data minimized the manual review effort required to actively manage bias in the training set.

Expert Insights on ML Workflows

The collaboration highlighted a specific iterative methodology for building ML workflows. As Julien Simon, Chief Evangelist at Hugging Face, described it:

"The Hugging way to build workflows: find open-source pre-trained models, evaluate them right away, see what works, see what does not. By iterating, you start learning things immediately"

For Witty Works, this approach reduced the financial and temporal costs of data annotation and deployment while providing a technical sparring partner for their startup's development process.

Sources