From the 1 of 18 linked papers with an AI index.
3 papers · 1 filter
Can Scale Save Us From Plasticity Loss in Large Language Models?
J. Fernando Hernandez-Garcia, Tomás Figliolia, Beren Millidge
The loss of plasticity - the ability of a network to learn new information after having already learned older information - is a fundamental challenge in creating artificial neural…
ZAYA1-8B Technical Report
Robert Washbourne, Rishi Iyer, Tomas Figliolia +15
We present ZAYA1-8B, a reasoning-focused mixture-of-experts (MoE) model with 700M active and 8B total parameters, built on Zyphra's MoE++ architecture. ZAYA1-8B's core pretraining,…
Exploring Action-Centric Representations Through the Lens of Rate-Distortion Theory
Miguel de Llanza Varona, Christopher L. Buckley, Beren Millidge
Organisms have to keep track of the information in the environment that is relevant for adaptive behaviour. Transmitting information in an economical and efficient way becomes cruc…