56 citations · 283 across the 30 of their papers we have counts for
8 papers · 1 filter
Efficient and Reliable Overlay Networks for Decentralized Federated Learning
Yifan Hua, Kevin Miller, Andrea L. Bertozzi +2
We propose near-optimal overlay networks based on -regular expander graphs to accelerate decentralized federated learning (DFL) and improve its generalization. In DFL a massive…
How Does Momentum Benefit Deep Neural Networks Architecture Design? A Few Case Studies
Bao Wang, Hedi Xia, Tan Nguyen +1
We present and review an algorithmic and theoretical framework for improving neural network architecture design via momentum. As case studies, we consider how momentum can improve…
Training Deep Neural Networks with Adaptive Momentum Inspired by the Quadratic Optimization
Tao Sun, Huaming Ling, Zuoqiang Shi +2
Heavy ball momentum is crucial in accelerating (stochastic) gradient-based optimization algorithms for machine learning. Existing heavy ball momentum is usually weighted by a unifo…
Heavy Ball Neural Ordinary Differential Equations
Hedi Xia, Vai Suliafu, Hangjie Ji +4
We propose heavy ball neural ordinary differential equations (HBNODEs), leveraging the continuous limit of the classical momentum accelerated gradient descent, to improve neural OD…
AIR-Net: Adaptive and Implicit Regularization Neural Network for Matrix Completion
Zhemin Li, Tao Sun, Hongxia Wang +1
The explicit low-rank regularization, e.g., nuclear norm regularization, has been widely used in imaging sciences. However, it has been found that implicit regularization outperfor…
FMMformer: Efficient and Flexible Transformer via Decomposed Near-field and Far-field Attention
Tan M. Nguyen, Vai Suliafu, Stanley J. Osher +2
We propose FMMformers, a class of efficient and flexible transformers inspired by the celebrated fast multipole method (FMM) for accelerating interacting particle simulation. FMM d…