Maximizing Efficiency of Language Model Pre-training for Learning Representation

Junmo Kang, Suwon Shin, Jeonghwan Kim +2

Pre-trained language models in the past years have shown exponential growth in model parameters and compute time. ELECTRA is a novel approach for improving the compute efficiency o…