2 citations · 2 across the 13 of their papers we have counts for
17 papers
Learning without Overwriting: A Theory of Self-Distillation and Supervised Fine-Tuning in Continual Reasoning
Shinichi Uemura, Taiji Suzuki
On-policy self-distillation (OPSD) of large language models (LLMs) has demonstrated the ability to improve reasoning capabilities while preserving previously acquired knowledge. De…
DPRM: A Plug-in Doob h transform-induced Token-Ordering Module for Discrete Diffusion Models
Dake Bu, Wei Huang, Andi Han +5
Discrete diffusion models admit many token orders, yet most systems rely on confidence-based decoding. Confidence is a strong and efficient heuristic, but it can be myopic because…
On the Learning Dynamics of Two-layer Linear Networks with Label Noise SGD
Tongcheng Zhang, Zhanpeng Zhou, Mingze Wang +4
One crucial factor behind the success of deep learning lies in the implicit bias induced by noise inherent in gradient-based training algorithms. Motivated by empirical observation…
Provable Benefit of Curriculum in Transformer Tree-Reasoning Post-Training
Dake Bu, Wei Huang, Andi Han +4
Recent curriculum techniques in the post-training stage of LLMs have been empirically observed to outperform non-curriculum approaches in improving reasoning performance, yet a pri…
Distributional Biases in Post-Training: A Markovian Analysis of Reasoning Trajectories
Dake Bu, Wei Huang, Andi Han +5
Foundation models exhibit broad knowledge but limited task-specific reasoning, motivating post-training strategies such as RL with verifiable rewards (RLVR) and test-time scaling (…
How Does Label Noise Gradient Descent Improve Generalization in the Low SNR Regime?
Wei Huang, Andi Han, Yujin Song +4
The capacity of deep learning models is often large enough to both learn the underlying statistical signal and overfit to noise in the training set. This noise memorization can be…
Mamba Can Learn Low-Dimensional Targets In-Context via Test-Time Feature Learning
Junsoo Oh, Wei Huang, Taiji Suzuki
Mamba, a recently proposed linear-time sequence model, has attracted significant attention for its computational efficiency and strong empirical performance. However, a rigorous th…
Trained Mamba Emulates Online Gradient Descent in In-Context Linear Regression
Jiarui Jiang, Wei Huang, Miao Zhang +2
State-space models (SSMs), particularly Mamba, emerge as an efficient Transformer alternative with linear complexity for long-sequence modeling. Recent empirical works demonstrate…
Test time training enhances in-context learning of nonlinear functions
Kento Kuwataka, Taiji Suzuki
Test-time training (TTT) enhances model performance by explicitly updating designated parameters prior to each prediction to adapt to the test data. While TTT has demonstrated cons…
Generalization Bound of Gradient Flow through Training Trajectory and Data-dependent Kernel
Yilan Chen, Zhichao Wang, Wei Huang +3
Gradient-based optimization methods have shown remarkable empirical success, yet their theoretical generalization properties remain only partially understood. In this paper, we est…
On the Role of Label Noise in the Feature Learning Process
Andi Han, Wei Huang, Zhanpeng Zhou +5
Deep learning with noisy labels presents significant challenges. In this work, we theoretically characterize the role of label noise from a feature learning perspective. Specifical…
On the Comparison between Multi-modal and Single-modal Contrastive Learning
Wei Huang, Andi Han, Yongqiang Chen +3
Multi-modal contrastive learning with language supervision has presented a paradigm shift in modern machine learning. By pre-training on a web-scale dataset, multi-modal contrastiv…
Provably Transformers Harness Multi-Concept Word Semantics for Efficient In-Context Learning
Dake Bu, Wei Huang, Andi Han +4
Transformer-based large language models (LLMs) have displayed remarkable creative prowess and emergence capabilities. Existing empirical studies have revealed a strong connection b…
On the Optimization and Generalization of Two-layer Transformers with Sign Gradient Descent
Bingrui Li, Wei Huang, Andi Han +4
The Adam optimizer is widely used for transformer optimization in practice, which makes understanding the underlying optimization mechanisms an important problem. However, due to t…
Unveil Benign Overfitting for Transformer in Vision: Training Dynamics, Convergence, and Generalization
Jiarui Jiang, Wei Huang, Miao Zhang +2
Transformers have demonstrated great power in the recent development of large foundational models. In particular, the Vision Transformer (ViT) has brought revolutionary changes to…
Provably Neural Active Learning Succeeds via Prioritizing Perplexing Samples
Dake Bu, Wei Huang, Taiji Suzuki +4
Neural Network-based active learning (NAL) is a cost-effective data selection technique that utilizes neural networks to select and train on a small subset of samples. While existi…
Convergence Error Analysis of Reflected Gradient Langevin Dynamics for Globally Optimizing Non-Convex Constrained Problems
Kanji Sato, Akiko Takeda, Reiichiro Kawai +1
Gradient Langevin dynamics and a variety of its variants have attracted increasing attention owing to their convergence towards the global optimal solution, initially in the uncons…