Running Privacy-Preserving Inferences on Hugging Face Endpoints
Zama and Hugging Face have integrated Concrete ML—a privacy-preserving machine learning framework—with Hugging Face Endpoints. This integration allows developers to deploy pre-compiled models that utilize Fully Homomorphic Encryption (FHE), enabling direct computation over encrypted data without the need for the private key, ensuring that user data remains private during inference.
FHE-Powered Inference via Hugging Face Endpoints
Users can deploy FHE-friendly models directly from the Hugging Face Hub using the "Inference Endpoint (dedicated)" option. Because Concrete ML models currently do not support GPUs, deployment requires selecting a CPU-backed machine (up to eight vCPUs are currently available).
Client-Side Workflow
To interact with a deployed FHE endpoint, users must perform the following steps:
- Clone the Model Repository: Users clone the specific model repository from Hugging Face to their local machine.
- Environment Setup: Users install Concrete ML and dependencies locally using Python 3.10 (required for compatibility with Hugging Face Endpoints).
- Execution: Using a provided script (e.g.,
play_with_endpoint.py), users quantize and encrypt their input data locally before sending it to the endpoint. - Decryption: The endpoint returns an encrypted prediction, which the user then decrypts and dequantizes locally to reveal the final result.
Technical Implementation via Custom Handlers
This functionality is made possible through Hugging Face's custom inference handlers. Zama implemented a handler.py file that defines a custom __call__ method for the EndpointHandler. This allows the endpoint to support specific FHE operations:
save_key: Saves FHE evaluation keys.append_key: Saves large FHE evaluation keys in smaller pieces.inference: Executes the FHE inference on the encrypted data.
Model Performance and Availability
Zama has provided several pre-compiled example models to demonstrate different ML tasks and their respective execution times on Hugging Face CPU Endpoints:
| Model Kind | Dataset | Execution Time on HF Endpoint |
|---|---|---|
| Logistic Regression | Synthetic | 0.4 sec |
| DecisionTree | Spam | 2.0 sec |
| QNN | Iris | 3.7 sec |
| CNN | MNIST | 24 sec |
Limitations and Future Directions
While the integration enables privacy-preserving inference, several technical constraints exist:
- Volatile Key Storage: Evaluation keys are stored in the endpoint's RAM. If the endpoint restarts, keys are lost and must be re-sent.
- Scaling: Because RAM is not shared between machines, handling massive traffic across multiple instances is challenging.
- Hardware Constraints: The current limit of eight vCPUs for CPU-backed endpoints may restrict high-load applications.
Preparing Custom FHE Models
Developers can create their own pre-compiled models by forking Zama's existing repositories and modifying creating_models.py to suit their specific dataset and task. The process involves training the model with Concrete ML to generate client.zip, server.zip, and versions.json files, which are then uploaded to the Hugging Face Hub with concrete-ml and FHE tags for discoverability.
Future Enhancements
Zama and Hugging Face aim to improve the experience by providing more powerful CPU machines, deeper integration of Concrete ML into the Hugging Face interface (such as a dedicated "Private-Preserving Inference Endpoint" button), and non-volatile, shared state storage for FHE inference keys across multiple server machines.