4 citations · 4 across the 2 of their papers we have counts for
2 papers
cs.CL2024★ 4 cited
Data Engineering for Scaling Language Models to 128K Context
Yao Fu, Rameswar Panda, Xinyao Niu +4
We study the continual pretraining recipe for scaling language models' context lengths to 128K, with a focus on data engineering. We hypothesize that long context modeling, in part…
cs.CL2023
FiLM: Fill-in Language Models for Any-Order Generation
Tianxiao Shen, Hao Peng, Ruoqi Shen +3
Language models have become the backbone of today's AI systems. However, their predominant left-to-right generation limits the use of bidirectional context, which is essential for…