Supercharging Customer Service with Machine Learning

Hugging Face has demonstrated a workflow for automating customer service by using machine learning to filter and prioritize unsatisfied customer feedback. By mapping the business problem to a text classification task and fine-tuning a pretrained transformer model, organizations can significantly reduce human effort while ensuring urgent complaints are addressed.

Mapping Customer Service to NLP Tasks

To automate the prioritization of customer messages, the problem is modeled as a text classification task. Specifically, the goal is to classify incoming messages into one of five sentiment categories: very unsatisfied, unsatisfied, neutral, satisfied, or very satisfied.

This approach allows customer support teams to focus exclusively on the most unsatisfied customers, aiming to answer 100% of those urgent messages while filtering out neutral or positive feedback.

Dataset Selection and Curation

Model performance is heavily dependent on the quality and relevance of the training data. While internal company data is recommended for real-world applications, Hugging Face utilized the amazon_reviews_multi dataset for this simulation.

Dataset Evaluation Criteria

  • Quality: The data must correspond to the expected real-world use case and remain diverse and unbiased.
  • Size: Larger datasets generally lead to better performance.

Among several options, amazon_reviews_multi was selected because it provides granular sentiment labels (1-5 stars) that map directly to the five required sentiment categories, whereas other datasets like Amazon polarity only offered binary labels.

Model Selection and Fine-Tuning

For this implementation, the microsoft/deberta-v3-base model was selected. DeBERTa is a top-ranking model on benchmarks such as GLUE and SuperGLUE, which evaluate text classification capabilities.

Technical Implementation Workflow

  1. Preprocessing: Using the datasets library, the review_body is tokenized and limited to a maximum length of 128 tokens. Labels (1-5) are shifted to a 0-4 range for compatibility with the model.
  2. Training: The model is fine-tuned using the Hugging Face Trainer API with the following hyperparameters:
    • Epochs: 2
    • Learning Rate: 2e-5
    • Warmup Steps: 200
    • Evaluation Strategy: Every 5,000 steps
  3. Metrics: Initial evaluation used standard accuracy. The model achieved approximately 61.8% accuracy on the validation set after 50,000 steps.

Use-Case Specific Evaluation

Standard accuracy can be misleading if the business goal is specific. To better measure the model's utility for customer service, Hugging Face implemented a custom metric focusing on the "very unsatisfied" class.

By treating both "very unsatisfied" and "unsatisfied" labels as targets for response, the model achieved the following results on the test set:

  • Recall for very unsatisfied customers: ~95% (the percentage of very unsatisfied messages correctly caught).
  • False Positive Rate: ~12% (the percentage of satisfied messages incorrectly labeled as unsatisfied).

Business Impact Simulation

In a scenario with 10,000 daily messages (500 of which are very negative), this system would reduce the human workload from 10,000 messages to approximately 1,700 messages. This represents an 83% reduction in human effort while only missing 5% of the most urgent complaints.

Production Optimization

Once a model reaches acceptable performance, Hugging Face recommends several paths for production optimization:

  • Hardware and Precision: Utilizing optimized GPUs and lowering precision to float16.
  • Accelerator Libraries: Using ONNX Runtime, quantization, or inference servers like NVIDIA Triton.
  • Hugging Face Optimum: An open-source library designed to simplify the optimization of Transformers models.
  • Inference API: A plug-and-play solution for serving machine learning tasks in production without requiring deep technical infrastructure knowledge.

Sources