29 citations · 94 across the 25 of their papers we have counts for
18 papers · 1 filter
Proteus: Incremental Memory Activation for Long-Context Sequence Modeling
Reza Bayat, Ali Behrouz, Vahab Mirrokni +1
The quadratic cost of attention-based sequence models for long contexts has motivated a growing line of research on memory-based models that can compress context into a compact sta…
Language Models Need Sleep: Learning to Self-Modify and Consolidate Memories
Ali Behrouz, Farnoosh Hashemi, Adel Javanmard +1
The past few decades have witnessed significant advances in the design of machine learning algorithms, from early studies on task-specific shallow models to more general deep Large…
Tapered Language Models
Reza Bayat, Ali Behrouz, Aaron Courville
Modern language models, including transformer, recurrent, and memory-based variants, share a common chassis: a stack of identical layers in which parameters are allocated uniformly…
Memory Caching: RNNs with Growing Memory
Ali Behrouz, Zeman Li, Yuan Deng +3
Transformers have been established as the de-facto backbones for most recent advances in sequence modeling, mainly due to their growing memory capacity that scales with the context…
Nested Learning: The Illusion of Deep Learning Architectures
Ali Behrouz, Meisam Razaviyayn, Peilin Zhong +1
Despite the recent progresses, particularly in developing Language Models, there are fundamental challenges and unanswered questions about how such models can continually learn/mem…
Trellis: Learning to Compress Key-Value Memory in Attention Models
Mahdi Karami, Ali Behrouz, Praneeth Kacham +1
Transformers, while powerful, suffer from quadratic computational complexity and the ever-growing Key-Value (KV) cache of the attention mechanism. This paper introduces Trellis, a…
MS-SSM: A Multi-Scale State Space Model for Efficient Sequence Modeling
Mahdi Karami, Ali Behrouz, Peilin Zhong +2
State-space models (SSMs) have recently attention as an efficient alternative to computationally expensive attention-based models for sequence modeling. They rely on linear recurre…
TNT: Improving Chunkwise Training for Test-Time Memorization
Zeman Li, Ali Behrouz, Yuan Deng +5
Recurrent neural networks (RNNs) with deep test-time memorization modules, such as Titans and TTT, represent a promising, linearly-scaling paradigm distinct from Transformers. Whil…
Hydra: Dual Exponentiated Memory for Multivariate Time Series Analysis
Asal Meskin, Alireza Mirrokni, Ali Najar +1
In recent years, effectively modeling multivariate time series has gained significant popularity, mainly due to its wide range of applications, ranging from healthcare to financial…
It's All Connected: A Journey Through Test-Time Memorization, Attentional Bias, Retention, and Online Optimization
Ali Behrouz, Meisam Razaviyayn, Peilin Zhong +1
Designing efficient and effective architectural backbones has been in the core of research efforts to enhance the capability of foundation models. Inspired by the human cognitive p…
Titans: Learning to Memorize at Test Time
Ali Behrouz, Peilin Zhong, Vahab Mirrokni
Over more than a decade there has been an extensive research effort on how to effectively utilize recurrent models and attention. While recurrent models aim to compress the data in…
Best of Both Worlds: Advantages of Hybrid Graph Sequence Models
Ali Behrouz, Ali Parviz, Mahdi Karami +3
Modern sequence models (e.g., Transformers, linear RNNs, etc.) emerged as dominant backbones of recent deep learning frameworks, mainly due to their efficiency, representational po…
Chimera: Effectively Modeling Multivariate Time Series with 2-Dimensional State Space Models
Ali Behrouz, Michele Santacatterina, Ramin Zabih
Modeling multivariate time series is a well-established problem with a wide range of applications from healthcare to financial markets. Traditional State Space Models (SSMs) are cl…
MambaMixer: Efficient Selective State Space Models with Dual Token and Channel Selection
Ali Behrouz, Michele Santacatterina, Ramin Zabih
Recent advances in deep learning have mainly relied on Transformers due to their data dependency and ability to learn at scale. The attention module in these architectures, however…
Graph Mamba: Towards Learning on Graphs with State Space Models
Ali Behrouz, Farnoosh Hashemi
Graph Neural Networks (GNNs) have shown promising potential in graph representation learning. The majority of GNNs define a local message-passing mechanism, propagating information…
CAT-Walk: Inductive Hypergraph Learning via Set Walks
Ali Behrouz, Farnoosh Hashemi, Sadaf Sadeghian +1
Temporal hypergraphs provide a powerful paradigm for modeling time-dependent, higher-order interactions in complex systems. Representation learning for hypergraphs is essential for…
Anomaly Detection in Multiplex Dynamic Networks: from Blockchain Security to Brain Disease Prediction
Ali Behrouz, Margo Seltzer
The problem of identifying anomalies in dynamic networks is a fundamental task with a wide range of applications. However, it raises critical challenges due to the complex nature o…
Fast Optimization of Weighted Sparse Decision Trees for use in Optimal Treatment Regimes and Optimal Policy Design
Ali Behrouz, Mathias Lecuyer, Cynthia Rudin +1
Sparse decision trees are one of the most common forms of interpretable models. While recent advances have produced algorithms that fully optimize sparse decision trees for predict…