347 citations · 402 across the 10 of their papers we have counts for
4 papers · 1 filter
Towards Structured Dynamic Sparse Pre-Training of BERT
Anastasia Dietrich, Frithjof Gressmann, Douglas Orr +3
Identifying algorithms for computational efficient unsupervised training of large language models is an important and active area of research. In this work, we develop and study a…
GroupBERT: Enhanced Transformer Architecture with Efficient Grouped Structures
Ivan Chelombiev, Daniel Justus, Douglas Orr +4
Attention based language models have become a critical component in state-of-the-art natural language processing systems. However, these models have significant computational requi…
Proxy-Normalizing Activations to Match Batch Normalization while Removing Batch Dependence
Antoine Labatie, Dominic Masters, Zach Eaton-Rosen +1
We investigate the reasons for the performance degradation incurred with batch-independent normalization. We find that the prototypical techniques of layer normalization and instan…
Making EfficientNet More Efficient: Exploring Batch-Independent Normalization, Group Convolutions and Reduced Resolution Training
Dominic Masters, Antoine Labatie, Zach Eaton-Rosen +1
Much recent research has been dedicated to improving the efficiency of training and inference for image classification. This effort has commonly focused on explicitly improving the…