94 citations · 245 across the 21 of their papers we have counts for
6 papers · 1 filter
Large-Scale Deep Learning Optimizations: A Comprehensive Survey
Xiaoxin He, Fuzhao Xue, Xiaozhe Ren +1
Deep learning have achieved promising results on a wide spectrum of AI applications. Larger datasets and models consistently yield better performance. However, we generally spend l…
NumGPT: Improving Numeracy Ability of Generative Pre-trained Models
Zhihua Jin, Xin Jiang, Xingbo Wang +4
Existing generative pre-trained language models (e.g., GPT) focus on modeling the language structure and semantics of general texts. However, those models do not consider the numer…
EfficientBERT: Progressively Searching Multilayer Perceptron via Warm-up Knowledge Distillation
Chenhe Dong, Guangrun Wang, Hang Xu +3
Pre-trained language models have shown remarkable results on various NLP tasks. Nevertheless, due to their bulky size and slow inference speed, it is hard to deploy them on edge de…
AutoBERT-Zero: Evolving BERT Backbone from Scratch
Jiahui Gao, Hang Xu, Han Shi +5
Transformer-based pre-trained language models like BERT and its variants have recently achieved promising performance in various natural language processing (NLP) tasks. However, t…
PanGu-: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation
Wei Zeng, Xiaozhe Ren, Teng Su +35
Large-scale Pretrained Language Models (PLMs) have become the new paradigm for Natural Language Processing (NLP). PLMs with hundreds of billions parameters such as GPT-3 have demon…
SparseBERT: Rethinking the Importance Analysis in Self-attention
Han Shi, Jiahui Gao, Xiaozhe Ren +4
Transformer-based models are popularly used in natural language processing (NLP). Its core component, self-attention, has aroused widespread interest. To understand the self-attent…