1 citations · 4 across the 17 of their papers we have counts for
17 papers
Transformers Learn the Optimal DDPM Denoiser for Multi-Token GMMs
Hongkang Li, Hancheng Min, Rene Vidal
Transformer-based diffusion models have demonstrated remarkable performance at generating high-quality samples. However, our theoretical understanding of the reasons for this succe…
Visual prompting reimagined: The power of the Activation Prompts
Yihua Zhang, Hongkang Li, Yuguang Yao +5
Visual prompting (VP) has emerged as a popular method to repurpose pretrained vision models for adaptation to downstream tasks. Unlike conventional model fine-tuning techniques, VP…
A Theoretical Analysis of Mamba's Training Dynamics: Filtering Relevant Features for Generalization in State Space Models
Mugunthan Shandirasegaran, Hongkang Li, Songyang Zhang +2
The recent empirical success of Mamba and other selective state space models (SSMs) has renewed interest in non-attention architectures for sequence modeling, yet their theoretical…
How Can Mamba Learn In Context with Outliers and Generalize Provably?
Hongkang Li, Songtao Lu, Xiaodong Cui +2
The Mamba model has gained significant attention for its computational advantages over Transformer-based models, while achieving comparable performance across a wide range of langu…
Theoretical Learning Performance of Graph Neural Networks: The Impact of Jumping Connections and Layer-wise Sparsification
Jiawei Sun, Hongkang Li, Meng Wang
Jumping connections enable Graph Convolutional Networks (GCNs) to overcome over-smoothing, while graph sparsification reduces computational demands by selecting a sub-matrix of the…
Merging Smarter, Generalizing Better: Enhancing Model Merging on OOD Data
Bingjie Zhang, Hongkang Li, Changlong Shi +5
Multi-task learning (MTL) concurrently trains a model on diverse task datasets to exploit common features, thereby improving overall performance across the tasks. Recent studies ha…
When is Task Vector Provably Effective for Model Editing? A Generalization Analysis of Nonlinear Transformers
Hongkang Li, Yihua Zhang, Shuai Zhang +3
Task arithmetic refers to editing the pre-trained model by adding a weighted sum of task vectors, each of which is the weight update from the pre-trained model to fine-tuned models…
Training Nonlinear Transformers for Chain-of-Thought Inference: A Theoretical Generalization Analysis
Hongkang Li, Songtao Lu, Pin-Yu Chen +2
Chain-of-Thought (CoT) is an efficient prompting method that enables the reasoning ability of large language models by augmenting the query using multiple examples with multiple in…
Learning on Transformers is Provable Low-Rank and Sparse: A One-layer Analysis
Hongkang Li, Meng Wang, Shuai Zhang +2
Efficient training and inference algorithms, such as low-rank adaption and model pruning, have shown impressive performance for learning Transformer-based large foundation models.…
What Improves the Generalization of Graph Transformers? A Theoretical Dive into the Self-attention and Positional Encoding
Hongkang Li, Meng Wang, Tengfei Ma +3
Graph Transformers, which incorporate self-attention and positional encoding, have recently emerged as a powerful architecture for various graph learning tasks. Despite their impre…
How does promoting the minority fraction affect generalization? A theoretical study of the one-hidden-layer neural network on group imbalance
Hongkang Li, Shuai Zhang, Yihua Zhang +3
Group imbalance has been a known problem in empirical risk minimization (ERM), where the achieved high average accuracy is accompanied by low accuracy in a minority group. Despite…
How Do Nonlinear Transformers Learn and Generalize in In-Context Learning?
Hongkang Li, Meng Wang, Songtao Lu +2
Transformer-based large language models have displayed impressive in-context learning capabilities, where a pre-trained model can handle new tasks without fine-tuning by simply aug…
On the Convergence and Sample Complexity Analysis of Deep Q-Networks with -Greedy Exploration
Shuai Zhang, Hongkang Li, Meng Wang +6
This paper provides a theoretical understanding of Deep Q-Network (DQN) with the -greedy exploration in deep reinforcement learning. Despite the tremendous empirical a…
How Can Context Help? Exploring Joint Retrieval of Passage and Personalized Context
Hui Wan, Hongkang Li, Songtao Lu +2
The integration of external personalized context information into document-grounded conversational systems has significant potential business value, but has not been well-studied.…
A Theoretical Understanding of Shallow Vision Transformers: Learning, Generalization, and Sample Complexity
Hongkang Li, Meng Wang, Sijia Liu +1
Vision Transformers (ViTs) with self-attention modules have recently achieved great empirical success in many vision tasks. Due to non-convex interactions across layers, however, t…
Learning and generalization of one-hidden-layer neural networks, going beyond standard Gaussian data
Hongkang Li, Shuai Zhang, Meng Wang
This paper analyzes the convergence and generalization of training a one-hidden-layer neural network when the input features follow the Gaussian mixture model consisting of a finit…
Generalization Guarantee of Training Graph Convolutional Networks with Graph Topology Sampling
Hongkang Li, Meng Wang, Sijia Liu +2
Graph convolutional networks (GCNs) have recently achieved great empirical success in learning graph-structured data. To address its scalability issue due to the recursive embeddin…