299 citations · 1.1k across the 29 of their papers we have counts for
4 papers · 1 filter
Consistent Diffusion Language Models
Hasan Amin, Yuan Gao, Yaser Souri +4
Diffusion language models (DLMs) are an attractive alternative to autoregressive models because they promise sublinear-time, parallel generation, yet practical gains remain elusive…
TorchScale: Transformers at Scale
Shuming Ma, Hongyu Wang, Shaohan Huang +8
Large Transformers have achieved state-of-the-art performance across many tasks. Most open-source libraries on scaling Transformers focus on improving training or inference with be…
Foundation Transformers
Hongyu Wang, Shuming Ma, Shaohan Huang +12
A big convergence of model architectures across language, vision, speech, and multimodal is emerging. However, under the same name "Transformers", the above areas use different imp…
METRO: Efficient Denoising Pretraining of Large Scale Autoencoding Language Models with Model Generated Signals
Payal Bajaj, Chenyan Xiong, Guolin Ke +7
We present an efficient method of pretraining large-scale autoencoding language models using training signals generated by an auxiliary model. Originated in ELECTRA, this training…