263 citations · 1.1k across the 15 of their papers we have counts for
9 papers · 1 filter
Brainformers: Trading Simplicity for Efficiency
Yanqi Zhou, Nan Du, Yanping Huang +12
Transformers are central to recent successes in natural language processing and computer vision. Transformers have a mostly uniform backbone where layers alternate between feed-for…
Unified Functional Hashing in Automatic Machine Learning
Ryan Gillard, Stephen Jonany, Yingjie Miao +7
The field of Automatic Machine Learning (AutoML) has recently attained impressive results, including the discovery of state-of-the-art machine learning solutions, such as neural im…
The Carbon Footprint of Machine Learning Training Will Plateau, Then Shrink
David Patterson, Joseph Gonzalez, Urs Hölzle +7
Machine Learning (ML) workloads have rapidly grown in importance, but raised concerns about their carbon footprint. Four best practices can reduce ML training energy by up to 100x…
Primer: Searching for Efficient Transformers for Language Modeling
David R. So, Wojciech Mańke, Hanxiao Liu +3
Large Transformer models have been central to recent advances in natural language processing. The training and inference costs of these models, however, have grown rapidly and beco…
Pay Attention to MLPs
Hanxiao Liu, Zihang Dai, David R. So +1
Transformers have become one of the most important architectural innovations in deep learning and have enabled many breakthroughs over the past few years. Here we propose a simple…
Carbon Emissions and Large Neural Network Training
David Patterson, Joseph Gonzalez, Quoc Le +6
The computation demand for machine learning (ML) has grown rapidly recently, which comes with a number of costs. Estimating the energy cost helps measure its environmental impact a…