Methods of improving LLM training stability

Oleg Rybakov, Mike Chrzanowski, Peter Dykas +2

Training stability of large language models(LLMs) is an important research topic. Reproducing training instabilities can be costly, so we use a small language model with 830M param…