Sentiment Analysis on Encrypted Data with Homomorphic Encryption

TL;DR

Hugging Face has demonstrated a method for performing sentiment analysis on encrypted data using Fully Homomorphic Encryption (FHE) via the Concrete-ML library. By combining BERT transformer embeddings with an XGBoost classifier, the system can provide sentiment predictions without ever decrypting the user's input data on the server.

Privacy-Preserving Inference with Homomorphic Encryption

Homomorphic encryption allows computations to be performed on encrypted data without requiring prior decryption. This is critical for sensitive applications, such as analyzing private messages, where the user's data must remain confidential even during the inference process.

To implement this, Hugging Face utilized the Concrete-ML library, which enables data scientists to deploy machine learning models in FHE settings without requiring deep expertise in cryptography.

Technical Architecture: Transformers and XGBoost

The system employs a two-stage pipeline to transform raw text into a sentiment prediction while maintaining encryption compatibility.

1. Text Representation via Transformers

Because raw text is not suitable for FHE, the system uses a transformer to generate a hidden representation (embedding) of the text.

  • Model Used: A BERT transformer fine-tuned on the Stanford Sentiment Treebank dataset (cardiffnlp/twitter-roberta-base-sentiment-latest).
  • Process: The text is tokenized and passed through the transformer. The system extracts the last hidden layer state and averages the representations of all tokens to create a single text-level vector of 768 dimensions.
  • Client-Side Execution: This transformation from text to tensor must occur on the client machine, as the encryption is applied to the resulting transformer representation.

2. Classification via XGBoost

The 768-dimensional vector is then fed into an XGBoost classifier, which is more compatible with FHE than deep neural networks.

  • Training: The model was trained using a public Twitter airline sentiment dataset, utilizing GridSearchCV to optimize hyperparameters.
  • Optimal Parameters: The best performing model used max_depth: 1, n_bits: 3, and n_estimators: 50.
  • Performance: The resulting model achieved an accuracy of 85.04% on the test set.

FHE Execution and Performance

Once the XGBoost model is trained, it is compiled into an FHE inference engine using Concrete-ML.

  • Inference Consistency: Predictions made over encrypted data (execute_in_fhe=True) are identical to predictions made on clear text.
  • Execution Time: FHE inference for a single tweet takes approximately 4.4 seconds on a 16-core CPU.
  • Compilation Time: The model compilation process takes approximately 9.3 seconds.

Deployment Workflow

The deployment follows a client-server protocol to ensure data privacy:

  1. Key Generation: The client generates a private and public key pair.
  2. Encryption: The client encodes, quantizes, and encrypts the transformer representation of the text using the public key.
  3. Server-Side Prediction: The server receives the encrypted data and performs the prediction using the public evaluation key. The server never sees the decrypted input.
  4. Decryption: The server sends the encrypted prediction back to the client, who decrypts it using their private key.

This architecture was implemented as a demo using Gradio for the client application, FastAPI for the server, and Uvicorn as the ASGI server, hosted on Hugging Face Spaces.

Sources