220 citations · 2.3k across the 62 of their papers we have counts for
10 papers · 1 filter
When in Doubt, Summon the Titans: Efficient Inference with Large Models
Ankit Singh Rawat, Manzil Zaheer, Aditya Krishna Menon +2
Scaling neural networks to "large" sizes, with billions of parameters, has been shown to yield impressive results on many challenging problems. However, the inference cost incurred…
Leveraging redundancy in attention with Reuse Transformers
Srinadh Bhojanapalli, Ayan Chakrabarti, Andreas Veit +5
Pairwise dot product-based attention allows Transformers to exchange information between tokens in an input-dependent way, and is key to their success across diverse applications i…
Batch Active Learning at Scale
Gui Citovsky, Giulia DeSalvo, Claudio Gentile +4
The ability to train complex and highly effective models often requires an abundance of training data, which can easily become a bottleneck in cost, time, and computational resourc…
Teacher's pet: understanding and mitigating biases in distillation
Michal Lukasik, Srinadh Bhojanapalli, Aditya Krishna Menon +1
Knowledge distillation is widely used as a means of improving the performance of a relatively simple student model using the predictions from a complex teacher model. Several works…
Eigen Analysis of Self-Attention and its Reconstruction from Partial Computation
Srinadh Bhojanapalli, Ayan Chakrabarti, Himanshu Jain +3
State-of-the-art transformer models use pairwise dot-product based self-attention, which comes at a computational cost quadratic in the input sequence length. In this paper, we inv…
Scaling Hierarchical Agglomerative Clustering to Billion-sized Datasets
Baris Sumengen, Anand Rajagopalan, Gui Citovsky +6
Hierarchical Agglomerative Clustering (HAC) is one of the oldest but still most widely used clustering methods. However, HAC is notoriously hard to scale to large data sets as the…