8 papers
How Can Mamba Learn In Context with Outliers and Generalize Provably?
Hongkang Li, Songtao Lu, Xiaodong Cui +2
The Mamba model has gained significant attention for its computational advantages over Transformer-based models, while achieving comparable performance across a wide range of langu…
Transformers Learn the Optimal DDPM Denoiser for Multi-Token GMMs
Hongkang Li, Hancheng Min, Rene Vidal
Transformer-based diffusion models have demonstrated remarkable performance at generating high-quality samples. However, our theoretical understanding of the reasons for this succe…
Visual prompting reimagined: The power of the Activation Prompts
Yihua Zhang, Hongkang Li, Yuguang Yao +5
Visual prompting (VP) has emerged as a popular method to repurpose pretrained vision models for adaptation to downstream tasks. Unlike conventional model fine-tuning techniques, VP…
A Theoretical Analysis of Mamba's Training Dynamics: Filtering Relevant Features for Generalization in State Space Models
Mugunthan Shandirasegaran, Hongkang Li, Songyang Zhang +2
The recent empirical success of Mamba and other selective state space models (SSMs) has renewed interest in non-attention architectures for sequence modeling, yet their theoretical…
Theoretical Learning Performance of Graph Neural Networks: The Impact of Jumping Connections and Layer-wise Sparsification
Jiawei Sun, Hongkang Li, Meng Wang
Jumping connections enable Graph Convolutional Networks (GCNs) to overcome over-smoothing, while graph sparsification reduces computational demands by selecting a sub-matrix of the…
Merging Smarter, Generalizing Better: Enhancing Model Merging on OOD Data
Bingjie Zhang, Hongkang Li, Changlong Shi +5
Multi-task learning (MTL) concurrently trains a model on diverse task datasets to exploit common features, thereby improving overall performance across the tasks. Recent studies ha…