29 citations · 94 across the 25 of their papers we have counts for
29 papers
Titan-Llama: Retrofitting Pre-trained LLMs with External Neural Memory Modules
Sarah Pan, Ali Behrouz
Transformers are efficient to pre-train at scale because attention permits parallel computation across sequence positions, but long-context inference requires increasingly large KV…
Self-Preservation in Language Models
Dongkyu Cho, Rumi Chunara, Ali Behrouz
Continual learning (CL) seeks to adapt pretrained models while preserving prior capabilities. Yet the tasks supplied for preservation cover only part of what a model knows, while f…
Proteus: Incremental Memory Activation for Long-Context Sequence Modeling
Reza Bayat, Ali Behrouz, Vahab Mirrokni +1
The quadratic cost of attention-based sequence models for long contexts has motivated a growing line of research on memory-based models that can compress context into a compact sta…
Online Neural Space Time Memory for Dynamic Novel View Synthesis
Baback Elmieh, Lynn Tsai, Zeman Li +8
Online novel view synthesis from multi-view streaming videos faces a fundamental trade-off: maintaining a persistent, long-horizon memory to reconstruct temporarily occluded region…
Language Models Need Sleep: Learning to Self-Modify and Consolidate Memories
Ali Behrouz, Farnoosh Hashemi, Adel Javanmard +1
The past few decades have witnessed significant advances in the design of machine learning algorithms, from early studies on task-specific shallow models to more general deep Large…
Tapered Language Models
Reza Bayat, Ali Behrouz, Aaron Courville
Modern language models, including transformer, recurrent, and memory-based variants, share a common chassis: a stack of identical layers in which parameters are allocated uniformly…
Memory Caching: RNNs with Growing Memory
Ali Behrouz, Zeman Li, Yuan Deng +3
Transformers have been established as the de-facto backbones for most recent advances in sequence modeling, mainly due to their growing memory capacity that scales with the context…
Nested Learning: The Illusion of Deep Learning Architectures
Ali Behrouz, Meisam Razaviyayn, Peilin Zhong +1
Despite the recent progresses, particularly in developing Language Models, there are fundamental challenges and unanswered questions about how such models can continually learn/mem…
Trellis: Learning to Compress Key-Value Memory in Attention Models
Mahdi Karami, Ali Behrouz, Praneeth Kacham +1
Transformers, while powerful, suffer from quadratic computational complexity and the ever-growing Key-Value (KV) cache of the attention mechanism. This paper introduces Trellis, a…
MS-SSM: A Multi-Scale State Space Model for Efficient Sequence Modeling
Mahdi Karami, Ali Behrouz, Peilin Zhong +2
State-space models (SSMs) have recently attention as an efficient alternative to computationally expensive attention-based models for sequence modeling. They rely on linear recurre…
TNT: Improving Chunkwise Training for Test-Time Memorization
Zeman Li, Ali Behrouz, Yuan Deng +5
Recurrent neural networks (RNNs) with deep test-time memorization modules, such as Titans and TTT, represent a promising, linearly-scaling paradigm distinct from Transformers. Whil…
Hydra: Dual Exponentiated Memory for Multivariate Time Series Analysis
Asal Meskin, Alireza Mirrokni, Ali Najar +1
In recent years, effectively modeling multivariate time series has gained significant popularity, mainly due to its wide range of applications, ranging from healthcare to financial…
ATLAS: Learning to Optimally Memorize the Context at Test Time
Ali Behrouz, Zeman Li, Praneeth Kacham +5
Transformers have been established as the most popular backbones in sequence modeling, mainly due to their effectiveness in in-context retrieval tasks and the ability to learn at s…
It's All Connected: A Journey Through Test-Time Memorization, Attentional Bias, Retention, and Online Optimization
Ali Behrouz, Meisam Razaviyayn, Peilin Zhong +1
Designing efficient and effective architectural backbones has been in the core of research efforts to enhance the capability of foundation models. Inspired by the human cognitive p…
Titans: Learning to Memorize at Test Time
Ali Behrouz, Peilin Zhong, Vahab Mirrokni
Over more than a decade there has been an extensive research effort on how to effectively utilize recurrent models and attention. While recurrent models aim to compress the data in…
Best of Both Worlds: Advantages of Hybrid Graph Sequence Models
Ali Behrouz, Ali Parviz, Mahdi Karami +3
Modern sequence models (e.g., Transformers, linear RNNs, etc.) emerged as dominant backbones of recent deep learning frameworks, mainly due to their efficiency, representational po…
A Unified Core Structure in Multiplex Networks: From Finding the Densest Subgraph to Modeling User Engagement
Farnoosh Hashemi, Ali Behrouz
In many complex systems, the interactions between objects span multiple aspects. Multiplex networks are accurate paradigms to model such systems, where each edge is associated with…
Chimera: Effectively Modeling Multivariate Time Series with 2-Dimensional State Space Models
Ali Behrouz, Michele Santacatterina, Ramin Zabih
Modeling multivariate time series is a well-established problem with a wide range of applications from healthcare to financial markets. Traditional State Space Models (SSMs) are cl…
MambaMixer: Efficient Selective State Space Models with Dual Token and Channel Selection
Ali Behrouz, Michele Santacatterina, Ramin Zabih
Recent advances in deep learning have mainly relied on Transformers due to their data dependency and ability to learn at scale. The attention module in these architectures, however…
Graph Mamba: Towards Learning on Graphs with State Space Models
Ali Behrouz, Farnoosh Hashemi
Graph Neural Networks (GNNs) have shown promising potential in graph representation learning. The majority of GNNs define a local message-passing mechanism, propagating information…
Generalized Densest Subgraph in Multiplex Networks
Ali Behrouz, Farnoosh Hashemi
Finding dense subgraphs of a large network is a fundamental problem in graph mining that has been studied extensively both for its theoretical richness and its many practical appli…
CAT-Walk: Inductive Hypergraph Learning via Set Walks
Ali Behrouz, Farnoosh Hashemi, Sadaf Sadeghian +1
Temporal hypergraphs provide a powerful paradigm for modeling time-dependent, higher-order interactions in complex systems. Representation learning for hypergraphs is essential for…
CS-TGN: Community Search via Temporal Graph Neural Networks
Farnoosh Hashemi, Ali Behrouz, Milad Rezaei Hajidehi
Searching for local communities is an important research challenge that allows for personalized community discovery and supports advanced data analysis in various complex networks,…
Anomaly Detection in Multiplex Dynamic Networks: from Blockchain Security to Brain Disease Prediction
Ali Behrouz, Margo Seltzer
The problem of identifying anomalies in dynamic networks is a fundamental task with a wide range of applications. However, it raises critical challenges due to the complex nature o…