39 citations · 135 across the 14 of their papers we have counts for
8 papers · 1 filter
Supervision Complexity and its Role in Knowledge Distillation
Hrayr Harutyunyan, Ankit Singh Rawat, Aditya Krishna Menon +2
Despite the popularity and efficacy of knowledge distillation, there is limited understanding of why it helps. In order to study the generalization behavior of a distilled student,…
EmbedDistill: A Geometric Knowledge Distillation for Information Retrieval
Seungyeon Kim, Ankit Singh Rawat, Manzil Zaheer +6
Large neural models (such as Transformers) achieve state-of-the-art performance for information retrieval (IR). In this paper, we aim to improve distillation methods that pave the…
Teacher Guided Training: An Efficient Framework for Knowledge Transfer
Manzil Zaheer, Ankit Singh Rawat, Seungyeon Kim +5
The remarkable performance gains realized by large pretrained models, e.g., GPT-3, hinge on the massive amounts of data they are exposed to during training. Analogously, distilling…
Balancing Robustness and Sensitivity using Feature Contrastive Learning
Seungyeon Kim, Daniel Glasner, Srikumar Ramalingam +3
It is generally believed that robust training of extremely large networks is critical to their success in real-world applications. However, when taken to the extreme, methods that…
On the Reproducibility of Neural Network Predictions
Srinadh Bhojanapalli, Kimberly Wilber, Andreas Veit +4
Standard training techniques for neural networks involve multiple sources of randomness, e.g., initialization, mini-batch ordering and in some cases data augmentation. Given that n…
Evaluations and Methods for Explanation through Robustness Analysis
Cheng-Yu Hsieh, Chih-Kuan Yeh, Xuanqing Liu +4
Feature based explanations, that provide importance of each feature towards the model prediction, is arguably one of the most intuitive ways to explain a model. In this paper, we e…
Why distillation helps: a statistical perspective
Aditya Krishna Menon, Ankit Singh Rawat, Sashank J. Reddi +2
Knowledge distillation is a technique for improving the performance of a simple "student" model by replacing its one-hot training labels with a distribution over labels obtained fr…
Matrix Approximation under Local Low-Rank Assumption
Joonseok Lee, Seungyeon Kim, Guy Lebanon +1
Matrix approximation is a common tool in machine learning for building accurate prediction models for recommendation systems, text mining, and computer vision. A prevalent assumpti…