665 citations · 2.3k across the 48 of their papers we have counts for
4 papers · 1 filter
Hash Layers For Large Sparse Models
Stephen Roller, Sainbayar Sukhbaatar, Arthur Szlam +1
We investigate the training of sparse layers that use different parameters for different inputs based on hashing in large Transformer models. Specifically, we modify the feedforwar…
Not All Memories are Created Equal: Learning to Forget by Expiring
Sainbayar Sukhbaatar, Da Ju, Spencer Poff +4
Attention mechanisms have shown promising results in sequence modeling tasks that require long-term memory. Recent work investigated mechanisms to reduce the computational cost of…
Staircase Attention for Recurrent Processing of Sequences
Da Ju, Stephen Roller, Sainbayar Sukhbaatar +1
Attention mechanisms have become a standard tool for sequence modeling tasks, in particular by stacking self-attention layers over the entire input sequence as in the Transformer a…
Memory-Augmented Reinforcement Learning for Image-Goal Navigation
Lina Mezghani, Sainbayar Sukhbaatar, Thibaut Lavril +4
In this work, we present a memory-augmented approach for image-goal navigation. Earlier attempts, including RL-based and SLAM-based approaches have either shown poor generalization…