Teaching is Hard: How to Train Small Models and Outperforming Large Counterparts | Towards Data Science

Distilling the knowledge of a large model is complex but a new method shows incredible performances