16 papers
Convergence of Continual Learning in Homogeneous Deep Networks
Matan Schliserman, Gon Buzaglo, Itay Evron +1
We characterize weakly regularized continual classification in homogeneous models as sequential projections onto task margin sets. This result generalizes prior analyses restricted…
Workspace Optimization: How to Train Your Agent
Elad Sarafian, Gal Kaplun, Ron Banner +2
Modern agents built on frontier language models often cannot adapt their weights. What, then, remains trainable? We argue it is the agent's \emph{workspace}, the structured externa…
Retrieval from Within: An Intrinsic Capability of Attention-Based Models
Elad Hoffer, Yochai Blau, Edan Kinderman +3
Retrieval-augmented generation (RAG) typically treats retrieval and generation as separate systems. We ask whether an attention-based encoder-decoder can instead retrieve directly…
Normalized Architectures are Natively 4-Bit
Maxim Fishman, Brian Chmiel, Ron Banner +2
Training large language models at 4-bit precision is critical for efficiency. We show that nGPT, an architecture that constrains weights and hidden representations to the unit hype…
From Continual Learning to SGD and Back: Better Rates for Continual Linear Models
Itay Evron, Ran Levinstein, Matan Schliserman +4
We study the common continual learning setup where an overparameterized model is sequentially fitted to a set of jointly realizable tasks. We analyze forgetting, defined as the los…
Block Sparse Flash Attention
Daniel Ohayon, Itay Lamprecht, Itay Hubara +3
Modern large language models increasingly require long contexts for reasoning and multi-document tasks, but attention's quadratic complexity creates a severe computational bottlene…