6 citations · 11 across the 12 of their papers we have counts for
12 papers
InfoFlow: A Framework for Multi-Layer Transformer Analysis
Penghao Yu, Haotian Jiang, Zeyu Bao +1
While the approximation properties of single-layer Transformer architectures have been studied in recent works, a rigorous theoretical understanding of the multi-layer setting rema…
The Effect of Attention Head Count on Transformer Approximation
Penghao Yu, Haotian Jiang, Zeyu Bao +2
Transformer has become the dominant architecture for sequence modeling, yet a detailed understanding of how its structural parameters influence expressive power remains limited. In…
Performance-Efficiency Tradeoffs in Transformers: An Approximation Theory Perspective
Ruoxi Yu, Haotian Jiang, Jingpu Cheng +3
Transformers have achieved remarkable successes across a wide range of applications, yet the theoretical foundation of their model efficiency remains underexplored. In this work, w…
The Effect of Depth on the Expressivity of Deep Linear State-Space Models
Zeyu Bao, Penghao Yu, Haotian Jiang +1
Deep state-space models (SSMs) have gained increasing popularity in sequence modelling. While there are numerous theoretical investigations of shallow SSMs, how the depth of the SS…
Numerical Investigation of Sequence Modeling Theory using Controllable Memory Functions
Haotian Jiang, Zeyu Bao, Shida Wang +1
The evolution of sequence modeling architectures, from recurrent neural networks and convolutional models to Transformers and structured state-space models, reflects ongoing effort…
StableSSM: Alleviating the Curse of Memory in State-space Models through Stable Reparameterization
Shida Wang, Qianxiao Li
In this paper, we investigate the long-term memory learning capabilities of state-space models (SSMs) from the perspective of parameterization. We prove that state-space models wit…
Forward and Inverse Approximation Theory for Linear Temporal Convolutional Networks
Haotian Jiang, Qianxiao Li
We present a theoretical analysis of the approximation properties of convolutional architectures when applied to the modeling of temporal sequences. Specifically, we prove an appro…
Inverse Approximation Theory for Nonlinear Recurrent Neural Networks
Shida Wang, Zhong Li, Qianxiao Li
We prove an inverse approximation theorem for the approximation of nonlinear sequence-to-sequence relationships using recurrent neural networks (RNNs). This is a so-called Bernstei…
Approximation Rate of the Transformer Architecture for Sequence Modeling
Haotian Jiang, Qianxiao Li
The Transformer architecture is widely applied in sequence modeling applications, yet the theoretical understanding of its working principles remains limited. In this work, we inve…
A Brief Survey on the Approximation Theory for Sequence Modelling
Haotian Jiang, Qianxiao Li, Zhong Li +1
We survey current developments in the approximation theory of sequence modelling in machine learning. Particular emphasis is placed on classifying existing results for various mode…
Approximation Theory of Convolutional Architectures for Time Series Modelling
Haotian Jiang, Zhong Li, Qianxiao Li
We study the approximation properties of convolutional architectures applied to time series modelling, which can be formulated mathematically as a functional approximation problem.…
On the Curse of Memory in Recurrent Neural Networks: Approximation and Optimization Analysis
Zhong Li, Jiequn Han, Weinan E +1
We study the approximation properties and optimization dynamics of recurrent neural networks (RNNs) when applied to learn input-output relationships in temporal data. We consider t…