29 citations · 94 across the 25 of their papers we have counts for
7 papers · 1 filter
Titan-Llama: Retrofitting Pre-trained LLMs with External Neural Memory Modules
Sarah Pan, Ali Behrouz
Transformers are efficient to pre-train at scale because attention permits parallel computation across sequence positions, but long-context inference requires increasingly large KV…
Self-Preservation in Language Models
Dongkyu Cho, Rumi Chunara, Ali Behrouz
Continual learning (CL) seeks to adapt pretrained models while preserving prior capabilities. Yet the tasks supplied for preservation cover only part of what a model knows, while f…
Proteus: Incremental Memory Activation for Long-Context Sequence Modeling
Reza Bayat, Ali Behrouz, Vahab Mirrokni +1
The quadratic cost of attention-based sequence models for long contexts has motivated a growing line of research on memory-based models that can compress context into a compact sta…
Online Neural Space Time Memory for Dynamic Novel View Synthesis
Baback Elmieh, Lynn Tsai, Zeman Li +8
Online novel view synthesis from multi-view streaming videos faces a fundamental trade-off: maintaining a persistent, long-horizon memory to reconstruct temporarily occluded region…
Language Models Need Sleep: Learning to Self-Modify and Consolidate Memories
Ali Behrouz, Farnoosh Hashemi, Adel Javanmard +1
The past few decades have witnessed significant advances in the design of machine learning algorithms, from early studies on task-specific shallow models to more general deep Large…
Tapered Language Models
Reza Bayat, Ali Behrouz, Aaron Courville
Modern language models, including transformer, recurrent, and memory-based variants, share a common chassis: a stack of identical layers in which parameters are allocated uniformly…
Memory Caching: RNNs with Growing Memory
Ali Behrouz, Zeman Li, Yuan Deng +3
Transformers have been established as the de-facto backbones for most recent advances in sequence modeling, mainly due to their growing memory capacity that scales with the context…