12 papers · 1 filter
Bayesian Optimality of In-Context Learning with Selective State Spaces
Di Zhang, Jiaqi Xing
We propose Bayesian optimal sequential prediction as a new principle for understanding in-context learning (ICL). Unlike interpretations framing Transformers as performing implicit…
Bigger Is Safer: Provable Robustness in In-Context Learning Scales with Capacity
Di Zhang, Ningxu Zhang, Zimeng Liu
In-context learning (ICL) allows large language models to adapt to new tasks from a few examples without updating their parameters. Existing theories explain ICL by assuming the te…
Exact Flow Linear Attention: Exact Solution from Continuous-Time Dynamics
Jingdi Lei, Di Zhang, Soujanya Poria
In this paper, we introduce Exact Flow Linear Attention~(EFLA), an exact-flow formulation of delta-rule linear attention. We show that the delta-rule update can be interpreted as a…
Fast Bayesian Updates via Harmonic Representations
Di Zhang
Bayesian inference, while foundational to probabilistic reasoning, is often hampered by the computational intractability of posterior distributions, particularly through the challe…
Learning Geometry: A Framework for Building Adaptive Manifold Models through Metric Optimization
Di Zhang
This paper proposes a novel paradigm for machine learning that moves beyond traditional parameter optimization. Unlike conventional approaches that search for optimal parameters wi…
The Neural Differential Manifold: An Architecture with Explicit Geometric Structure
Di Zhang
This paper introduces the Neural Differential Manifold (NDM), a novel neural network architecture that explicitly incorporates geometric structure into its fundamental design. Depa…
On the Optimal Representation Efficiency of Barlow Twins: An Information-Geometric Interpretation
Di Zhang
Self-supervised learning (SSL) has achieved remarkable success by learning meaningful representations without labeled data. However, a unified theoretical framework for understandi…
A Frequency-Domain Analysis of the Multi-Armed Bandit Problem: A New Perspective on the Exploration-Exploitation Trade-off
Di Zhang
The stochastic multi-armed bandit (MAB) problem is one of the most fundamental models in sequential decision-making, with the core challenge being the trade-off between exploration…
PolyKAN: A Polyhedral Analysis Framework for Provable and Approximately Optimal KAN Compression
Di Zhang
Kolmogorov-Arnold Networks (KANs) have emerged as a promising alternative to traditional Multi-Layer Perceptrons (MLPs), offering enhanced interpretability and a solid mathematical…
Towards Stable and Effective Reinforcement Learning for Mixture-of-Experts
Di Zhang, Xun Wu, Shaohan Huang +6
Recent advances in reinforcement learning (RL) have substantially improved the training of large-scale language models, leading to significant gains in generation quality and reaso…
AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
Can Jin, Yang Zhou, Qixin Zhang +8
Test-time scaling strategies for Large Language Models predominantly rely on either reinforcement learning with sparse outcome rewards or search-based methods guided by static Proc…
The Stochastic Conjugate Subgradient Algorithm For Kernel Support Vector Machines
Di Zhang, Suvrajeet Sen
Stochastic First-Order (SFO) methods have been a cornerstone in addressing a broad spectrum of modern machine learning (ML) challenges. However, their efficacy is increasingly ques…