2.7k citations · 6.8k across the 39 of their papers we have counts for
4 papers · 1 filter
Not All Memories are Created Equal: Learning to Forget by Expiring
Sainbayar Sukhbaatar, Da Ju, Spencer Poff +4
Attention mechanisms have shown promising results in sequence modeling tasks that require long-term memory. Recent work investigated mechanisms to reduce the computational cost of…
Training with Quantization Noise for Extreme Model Compression
Angela Fan, Pierre Stock, Benjamin Graham +4
We tackle the problem of producing compact models, maximizing their accuracy for a given model size. A standard solution is to train networks with Quantization Aware Training, wher…
Addressing Some Limitations of Transformers with Feedback Memory
Angela Fan, Thibaut Lavril, Edouard Grave +2
Transformers have been successfully applied to sequential, auto-regressive tasks despite being feedforward networks. Unlike recurrent neural networks, Transformers use attention to…
Reducing Transformer Depth on Demand with Structured Dropout
Angela Fan, Edouard Grave, Armand Joulin
Overparameterized transformer networks have obtained state of the art results in various natural language processing tasks, such as machine translation, language modeling, and ques…