5 citations · 11 across the 10 of their papers we have counts for
11 papers
Attention to Mamba: A Recipe for Cross-Architecture Distillation
Abhinav Moudgil, Ningyuan Huang, Eeshan Gunesh Dhekane +3
State Space Models (SSMs) such as Mamba have become a popular alternative to Transformer models, due to their reduced memory consumption and higher throughput at generation compare…
SelfReflect: Can LLMs Communicate Their Internal Answer Distribution?
Michael Kirchhof, Luca Füger, Adam Goliński +4
The common approach to communicate a large language model's (LLM) uncertainty is to add a percentage number or a hedging word to its response. But is this all we can do? Instead of…
Theory, Analysis, and Best Practices for Sigmoid Self-Attention
Jason Ramapuram, Federico Danieli, Eeshan Dhekane +8
Attention is a key part of the transformer architecture. It is a sequence-to-sequence mapping that transforms each sequence element into a weighted sum of values. The weights are t…
Poly-View Contrastive Learning
Amitis Shidani, Devon Hjelm, Jason Ramapuram +3
Contrastive learning typically matches pairs of related views among a number of unrelated negative views. Views can be generated (e.g. by augmentations) or be observed. We investig…
Bootstrap Your Own Variance
Polina Turishcheva, Jason Ramapuram, Sinead Williamson +3
Understanding model uncertainty is important for many applications. We propose Bootstrap Your Own Variance (BYOV), combining Bootstrap Your Own Latent (BYOL), a negative-free Self-…
How to Scale Your EMA
Dan Busbridge, Jason Ramapuram, Pierre Ablin +4
Preserving training dynamics across batch sizes is an important tool for practical machine learning as it enables the trade-off between batch size and wall-clock time. This trade-o…
Elastic Weight Consolidation Improves the Robustness of Self-Supervised Learning Methods under Transfer
Andrius Ovsianas, Jason Ramapuram, Dan Busbridge +2
Self-supervised representation learning (SSL) methods provide an effective label-free initial condition for fine-tuning downstream tasks. However, in numerous realistic scenarios,…
Iterated learning for emergent systematicity in VQA
Ankit Vani, Max Schwarzer, Yuchen Lu +2
Although neural module networks have an architectural bias towards compositionality, they require gold standard layouts to generalize systematically in practice. When instead learn…
Transfer Learning by Modeling a Distribution over Policies
Disha Shrivastava, Eeshan Gunesh Dhekane, Riashat Islam
Exploration and adaptation to new tasks in a transfer learning setup is a central challenge in reinforcement learning. In this work, we build on the idea of modeling a distribution…
Hierarchical Importance Weighted Autoencoders
Chin-Wei Huang, Kris Sankaran, Eeshan Dhekane +2
Importance weighted variational inference (Burda et al., 2015) uses multiple i.i.d. samples to have a tighter variational lower bound. We believe a joint proposal has the potential…
Learning Affective Correspondence between Music and Image
Gaurav Verma, Eeshan Gunesh Dhekane, Tanaya Guha
We introduce the problem of learning affective correspondence between audio (music) and visual data (images). For this task, a music clip and an image are considered similar (havin…