Fewer Truncations Improve Language Modeling

Hantian Ding, Zijian Wang, Giovanni Paolini +4

In large language model training, input documents are typically concatenated together and then split into sequences of equal length to avoid padding tokens. Despite its efficiency,…