5.2k citations · 10k across the 79 of their papers we have counts for
12 papers · 1 filter
Large Language Model Programs
Imanol Schlag, Sainbayar Sukhbaatar, Asli Celikyilmaz +4
In recent years, large pre-trained language models (LLMs) have demonstrated the ability to follow instructions and perform novel tasks from a few examples. The possibility to param…
Learning to Reason and Memorize with Self-Notes
Jack Lanchantin, Shubham Toshniwal, Jason Weston +2
Large language models have been shown to struggle with multi-step reasoning, and do not retain previous reasoning steps for future use. We propose a simple method for solving both…
Hash Layers For Large Sparse Models
Stephen Roller, Sainbayar Sukhbaatar, Arthur Szlam +1
We investigate the training of sparse layers that use different parameters for different inputs based on hashing in large Transformer models. Specifically, we modify the feedforwar…
Not All Memories are Created Equal: Learning to Forget by Expiring
Sainbayar Sukhbaatar, Da Ju, Spencer Poff +4
Attention mechanisms have shown promising results in sequence modeling tasks that require long-term memory. Recent work investigated mechanisms to reduce the computational cost of…
Staircase Attention for Recurrent Processing of Sequences
Da Ju, Stephen Roller, Sainbayar Sukhbaatar +1
Attention mechanisms have become a standard tool for sequence modeling tasks, in particular by stacking self-attention layers over the entire input sequence as in the Transformer a…
Neural Text Generation with Unlikelihood Training
Sean Welleck, Ilia Kulikov, Stephen Roller +3
Neural text generation is a key tool in natural language applications, but it is well known there are major problems at its core. In particular, standard likelihood training and de…