3 citations · 7 across the 5 of their papers we have counts for
6 papers
Fork-and-Flush: Escaping Idea Basins in Autoresearch Agents
Ziyang Cai, Christos Ziakas, Vasilis Kontonis +5
Autoresearch agents tackle open-ended problems by repeatedly proposing candidate solutions, evaluating them, and using feedback to guide subsequent experiments. We show that indepe…
Full-bandwidth transformer
Xi Wang, Ziyang Cai, Zheng Zhan +5
Autoregressive transformers compute along two axes: horizontally across generated tokens, and vertically through model depth. Dense attention gives each token broad horizontal acce…
Hierarchical Latent Prediction for Language Models
Chang Shi, Tim Pearce, Manan Tomar +2
While standard Next-Token Prediction (NTP) lays the foundation of language model pre- training, its teacher-forced training paradigm may not be optimal for long-horizon reasoning a…
Scaling Laws for Pre-training Agents and World Models
Tim Pearce, Tabish Rashid, Dave Bignell +3
The performance of embodied agents has been shown to improve by increasing model parameters, dataset size, and compute. This has been demonstrated in domains from robotics to video…
Reconciling Kaplan and Chinchilla Scaling Laws
Tim Pearce, Jinyeop Song
Kaplan et al. [2020] (`Kaplan') and Hoffmann et al. [2022] (`Chinchilla') studied the scaling behavior of transformers trained on next-token language prediction. These studies prod…
Diffusion for World Modeling: Visual Details Matter in Atari
Eloi Alonso, Adam Jelley, Vincent Micheli +4
World models constitute a promising approach for training reinforcement learning agents in a safe and sample-efficient manner. Recent world models predominantly operate on sequence…