565 citations · 1.1k across the 21 of their papers we have counts for
7 papers · 1 filter
Simplifying and Understanding State Space Models with Diagonal Linear RNNs
Ankit Gupta, Harsh Mehta, Jonathan Berant
Sequence models based on linear state spaces (SSMs) have recently emerged as a promising choice of architecture for modeling long range dependencies across various modalities. Howe…
Differentially Private Image Classification from Features
Harsh Mehta, Walid Krichene, Abhradeep Thakurta +2
Leveraging transfer learning has recently been shown to be an effective strategy for training large models with Differential Privacy (DP). Moreover, somewhat surprisingly, recent w…
Convexifying Transformers: Improving optimization and understanding of transformer networks
Tolga Ergen, Behnam Neyshabur, Harsh Mehta
Understanding the fundamental mechanism behind the success of transformer networks is still an open problem in the deep learning literature. Although their remarkable performance h…
Long Range Language Modeling via Gated State Spaces
Harsh Mehta, Ankit Gupta, Ashok Cutkosky +1
State space models have shown to be effective at modeling long range dependencies, specially on sequence classification tasks. In this work we focus on autoregressive sequence mode…
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao +448
Language models demonstrate both quantitative improvement and new qualitative capabilities with increasing scale. Despite their potentially transformative impact, these new capabil…
Large Scale Transfer Learning for Differentially Private Image Classification
Harsh Mehta, Abhradeep Thakurta, Alexey Kurakin +1
Differential Privacy (DP) provides a formal framework for training machine learning models with individual example level privacy. In the field of deep learning, Differentially Priv…
Transformer Memory as a Differentiable Search Index
Yi Tay, Vinh Q. Tran, Mostafa Dehghani +10
In this paper, we demonstrate that information retrieval can be accomplished with a single Transformer, in which all information about the corpus is encoded in the parameters of th…