4 papers · 2 filters
Convergent Stochastic Training of Multi-Headed Attention and Understanding LoRA
Zhengkai Sun, Dibyakanti Kumar, Alejandro F Frangi +2
Transformers have revolutionized machine learning and deploying attention layers in the model is increasingly standard across a myriad of applications. Further, for large models, i…
On the Generalization Behavior of Deep Residual Networks From a Dynamical System Perspective
Jinshu Huang, Mingfei Sun, Chunlin Wu
Deep neural networks (DNNs) have significantly advanced machine learning, with model depth playing a central role in their successes. The dynamical system modeling approach has rec…
Rank-1 Approximation of Inverse Fisher for Natural Policy Gradients in Deep Reinforcement Learning
Yingxiao Huo, Satya Prakash Dash, Radu Stoican +2
Natural gradients have long been studied in deep reinforcement learning due to their fast convergence properties and covariant weight updates. However, computing natural gradients…
Gradient Regularized Natural Gradients
Satya Prakash Dash, Hossein Abdi, Wei Pan +2
Gradient regularization (GR) has been shown to improve the generalizability of trained models. While Natural Gradient Descent has been shown to accelerate optimization in the initi…