41 papers
Cracks in the Foundation: Seemingly Minor Architectural Choices Impact Long Context Extension
Amanda Bertsch, Luca Soldaini, Matthew R. Gormley +4
One might imagine that architectural variations within the dense transformer paradigm have a limited effect on accuracy. However, we demonstrate that this is not the case in the lo…
Pretraining Data Can Be Poisoned through Computational Propaganda
Victoria Graf, Hannaneh Hajishirzi, Noah A. Smith +2
The paper shows that language model pretraining data can be poisoned through publicly editable web discussion pages, and introduces a method called HalfLife to estimate how much ma…
Rethinking the Evaluation of Harness Evolution for Agents
Yike Wang, Huaisheng Zhu, Zhengyu Hu +7
The paper reexamines how automatic harness evolution for large language model agents is evaluated, comparing it to simple test‑time scaling baselines and finding that it offers lim…
Tmax: A simple recipe for terminal agents
Hamish Ivison, Junjie Oscar Yin, Rulin Shao +3
Terminal-using agents have quickly become the most popular downstream application of language models (LMs). Despite their prevalence, relatively little academic work has examined R…
Olmo Hybrid: From Theory to Practice and Back
William Merrill, Yanhong Li, Tyler Romero +19
Recent work has demonstrated the potential of non-transformer language models, especially linear recurrent neural networks (RNNs) and hybrid models that mix recurrence and attentio…
RLVE: Scaling Up Reinforcement Learning for Language Models with Adaptive Verifiable Environments
Zhiyuan Zeng, Hamish Ivison, Yiping Wang +14
We introduce Reinforcement Learning (RL) with Adaptive Verifiable Environments (RLVE), an approach using verifiable environments that procedurally generate problems and provide alg…