4 papers
Sinkhorn doubly stochastic attention rank decay analysis
Michela Lapenna, Rita Fioresi, Bahman Gharesifard
The self-attention mechanism is central to the success of Transformer architectures. However, standard row-stochastic attention has been shown to suffer from significant signal deg…
On the Topology of Neural Network Superlevel Sets
Bahman Gharesifard
We show that neural networks with activations satisfying a Riccati-type ordinary differential equation condition, an assumption arising in recent universal approximation results in…
Localmax dynamics for attention in transformers and its asymptotic behavior
Henri Cimetière, Maria Teresa Chiri, Bahman Gharesifard
We introduce a new discrete-time attention model, termed the localmax dynamics, which interpolates between the classic softmax dynamics and the hardmax dynamics, where only the tok…
Neural Lyapunov Function Approximation with Self-Supervised Reinforcement Learning
Luc McCutcheon, Bahman Gharesifard, Saber Fallah
Control Lyapunov functions are traditionally used to design a controller which ensures convergence to a desired state, yet deriving these functions for nonlinear systems remains a…