HLAT: High-quality Large Language Model Pre-trained on AWS Trainium

Haozheng Fan, Hao Zhou, Guangtai Huang +6

Getting large language models (LLMs) to perform well on the downstream tasks requires pre-training over trillions of tokens. This typically demands a large number of powerful compu…