Scaling Expert Language Models with Unsupervised Domain Discovery

Suchin Gururangan, Margaret Li, Mike Lewis +4

Large language models are typically trained densely: all parameters are updated with respect to all inputs. This requires synchronization of billions of parameters across thousands…