1 citations · 3 across the 19 of their papers we have counts for
6 papers · 1 filter
UNREAL: Unifying Retrieval and Long-Context with a Single Model
Edan Kinderman, Elad Hoffer, Yochai Blau +4
Long-context inference and Retrieval-Augmented Generation (RAG) handle evidence selection at vastly different scales, from a single long prompt to an entire corpus. We ask whether…
More Value per Key: Asymmetric Sparse Attention for Faster LLM Decoding
Noam Elata, Itay Lamprecht, Mikey Shechter +3
Autoregressive generation in Large Language Models (LLMs) is constrained by the memory and computational demands of attention mechanisms. Sparse attention methods mitigate this cos…
Convergence of Continual Learning in Homogeneous Deep Networks
Matan Schliserman, Gon Buzaglo, Itay Evron +1
We characterize weakly regularized continual classification in homogeneous models as sequential projections onto task margin sets. This result generalizes prior analyses restricted…
Workspace Optimization: How to Train Your Agent
Elad Sarafian, Gal Kaplun, Ron Banner +2
Modern agents built on frontier language models often cannot adapt their weights. What, then, remains trainable? We argue it is the agent's \emph{workspace}, the structured externa…
Retrieval from Within: An Intrinsic Capability of Attention-Based Models
Elad Hoffer, Yochai Blau, Edan Kinderman +3
Retrieval-augmented generation (RAG) typically treats retrieval and generation as separate systems. We ask whether an attention-based encoder-decoder can instead retrieve directly…
Normalized Architectures are Natively 4-Bit
Maxim Fishman, Brian Chmiel, Ron Banner +2
Training large language models at 4-bit precision is critical for efficiency. We show that nGPT, an architecture that constrains weights and hidden representations to the unit hype…