1.2k citations · 3.4k across the 52 of their papers we have counts for
5 papers · 1 filter
Long Range Arena: A Benchmark for Efficient Transformers
Yi Tay, Mostafa Dehghani, Samira Abnar +7
Transformers do not scale very well to long sequence lengths largely because of quadratic self-attention complexity. In the recent months, a wide spectrum of efficient, fast Transf…
Efficient Transformers: A Survey
Yi Tay, Mostafa Dehghani, Dara Bahri +1
Transformer model architectures have garnered immense interest lately due to their effectiveness across a range of domains like language, vision and reinforcement learning. In the…
IDF++: Analyzing and Improving Integer Discrete Flows for Lossless Compression
Rianne van den Berg, Alexey A. Gritsenko, Mostafa Dehghani +2
In this paper we analyse and improve integer discrete flows for lossless compression. Integer discrete flows are a recently proposed class of models that learn invertible transform…
Transferring Inductive Biases through Knowledge Distillation
Samira Abnar, Mostafa Dehghani, Willem Zuidema
Having the right inductive biases can be crucial in many tasks or scenarios where data or computing resources are a limiting factor, or where training data is not perfectly represe…
MetNet: A Neural Weather Model for Precipitation Forecasting
Casper Kaae Sønderby, Lasse Espeholt, Jonathan Heek +6
Weather forecasting is a long standing scientific challenge with direct social and economic impact. The task is suitable for deep neural networks due to vast amounts of continuousl…