dkozlov/awesome-knowledge-distillation
Awesome Knowledge Distillation
What it solves
Knowledge distillation is a technique used to transfer the "intelligence" or learned patterns from a large, complex model (the teacher) into a smaller, more efficient model (the student). This allows developers to create compact models that are faster and lighter, making them suitable for deployment on devices with limited computing power while retaining much of the original model's performance.
How it works
In this process, a large model is trained on a dataset. Once trained, its outputs—which often contain subtle, "dark knowledge" about how the model relates different categories—are used to guide the training of a smaller model. The student model learns to mimic the teacher's behavior, effectively compressing the teacher's knowledge into a more streamlined architecture.
Who it’s for
This collection is for machine learning researchers and engineers looking to optimize deep learning models for production environments, mobile devices, or resource-constrained hardware.
Highlights
- A comprehensive, curated list of academic papers on knowledge distillation.
- Covers foundational research dating back to 1990.
- Includes advanced topics like adversarial training, multi-task learning, and model compression.
- Features specific applications for computer vision, natural language processing (BERT), and speech synthesis.
Related
- Project
- Project
- Project
- Project