39 citations · 135 across the 14 of their papers we have counts for
15 papers
Supervision Complexity and its Role in Knowledge Distillation
Hrayr Harutyunyan, Ankit Singh Rawat, Aditya Krishna Menon +2
Despite the popularity and efficacy of knowledge distillation, there is limited understanding of why it helps. In order to study the generalization behavior of a distilled student,…
EmbedDistill: A Geometric Knowledge Distillation for Information Retrieval
Seungyeon Kim, Ankit Singh Rawat, Manzil Zaheer +6
Large neural models (such as Transformers) achieve state-of-the-art performance for information retrieval (IR). In this paper, we aim to improve distillation methods that pave the…
Teacher Guided Training: An Efficient Framework for Knowledge Transfer
Manzil Zaheer, Ankit Singh Rawat, Seungyeon Kim +5
The remarkable performance gains realized by large pretrained models, e.g., GPT-3, hinge on the massive amounts of data they are exposed to during training. Analogously, distilling…
Balancing Robustness and Sensitivity using Feature Contrastive Learning
Seungyeon Kim, Daniel Glasner, Srikumar Ramalingam +3
It is generally believed that robust training of extremely large networks is critical to their success in real-world applications. However, when taken to the extreme, methods that…
On the Reproducibility of Neural Network Predictions
Srinadh Bhojanapalli, Kimberly Wilber, Andreas Veit +4
Standard training techniques for neural networks involve multiple sources of randomness, e.g., initialization, mini-batch ordering and in some cases data augmentation. Given that n…
Semantic Label Smoothing for Sequence to Sequence Problems
Michal Lukasik, Himanshu Jain, Aditya Krishna Menon +4
Label smoothing has been shown to be an effective regularization strategy in classification, that prevents overfitting and helps in label de-noising. However, extending such method…
Evaluations and Methods for Explanation through Robustness Analysis
Cheng-Yu Hsieh, Chih-Kuan Yeh, Xuanqing Liu +4
Feature based explanations, that provide importance of each feature towards the model prediction, is arguably one of the most intuitive ways to explain a model. In this paper, we e…
Why distillation helps: a statistical perspective
Aditya Krishna Menon, Ankit Singh Rawat, Sashank J. Reddi +2
Knowledge distillation is a technique for improving the performance of a simple "student" model by replacing its one-hot training labels with a distribution over labels obtained fr…
Why are Adaptive Methods Good for Attention Models?
Jingzhao Zhang, Sai Praneeth Karimireddy, Andreas Veit +4
While stochastic gradient descent (SGD) is still the \emph{de facto} algorithm in deep learning, adaptive methods like Clipped SGD/Adam have been observed to outperform SGD across…
Fast Spammer Detection Using Structural Rank
Seungyeon Kim, Haesun Park, Guy Lebanon
Comments for a product or a news article are rapidly growing and became a medium of measuring quality products or services. Consequently, spammers have been emerged in this area to…
The Manifold of Human Emotions
Seungyeon Kim, Fuxin Li, Guy Lebanon +1
Sentiment analysis predicts the presence of positive or negative emotions in a text document. In this paper, we consider higher dimensional extensions of the sentiment concept, whi…
Matrix Approximation under Local Low-Rank Assumption
Joonseok Lee, Seungyeon Kim, Guy Lebanon +1
Matrix approximation is a common tool in machine learning for building accurate prediction models for recommendation systems, text mining, and computer vision. A prevalent assumpti…
Cumulative Revision Map
Seungyeon Kim, Joshua V. Dillon, Guy Lebanon
Unlike static documents, version-controlled documents are edited by one or more authors over a certain period of time. Examples include large scale computer code, papers authored b…
Beyond Sentiment: The Manifold of Human Emotions
Seungyeon Kim, Fuxin Li, Guy Lebanon +1
Sentiment analysis predicts the presence of positive or negative emotions in a text document. In this paper we consider higher dimensional extensions of the sentiment concept, whic…
Local Space-Time Smoothing for Version Controlled Documents
Seungyeon Kim, Guy Lebanon
Unlike static documents, version controlled documents are continuously edited by one or more authors. Such collaborative revision process makes traditional modeling and visualizati…