2 papers
cs.CL2024
LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Models
Chi Han, Qifan Wang, Hao Peng +4
Today's large language models (LLMs) typically train on short text segments (e.g., <4K tokens) due to the quadratic complexity of their Transformer architectures. As a result, thei…
cs.CL2024
Representation Deficiency in Masked Language Modeling
Yu Meng, Jitin Krishnan, Sinong Wang +6
Masked Language Modeling (MLM) has been one of the most prominent approaches for pretraining bidirectional text encoders due to its simplicity and effectiveness. One notable concer…