56 citations · 283 across the 30 of their papers we have counts for
22 papers · 1 filter
Momentum Transformer: Closing the Performance Gap Between Self-attention and Its Linearization
Tan Nguyen, Richard G. Baraniuk, Robert M. Kirby +2
Transformers have achieved remarkable success in sequence modeling and beyond but suffer from quadratic computational and memory complexities with respect to the length of the inpu…
Learning POD of Complex Dynamics Using Heavy-ball Neural ODEs
Justin Baker, Elena Cherkaev, Akil Narayan +1
Proper orthogonal decomposition (POD) allows reduced-order modeling of complex dynamical systems at a substantial level, while maintaining a high degree of accuracy in modeling the…
glassoformer: a query-sparse transformer for post-fault power grid voltage prediction
Yunling Zheng, Carson Hu, Guang Lin +3
We propose GLassoformer, a novel and efficient transformer architecture leveraging group Lasso regularization to reduce the number of queries of the standard self-attention mechani…
How Does Momentum Benefit Deep Neural Networks Architecture Design? A Few Case Studies
Bao Wang, Hedi Xia, Tan Nguyen +1
We present and review an algorithmic and theoretical framework for improving neural network architecture design via momentum. As case studies, we consider how momentum can improve…
Training Deep Neural Networks with Adaptive Momentum Inspired by the Quadratic Optimization
Tao Sun, Huaming Ling, Zuoqiang Shi +2
Heavy ball momentum is crucial in accelerating (stochastic) gradient-based optimization algorithms for machine learning. Existing heavy ball momentum is usually weighted by a unifo…
Heavy Ball Neural Ordinary Differential Equations
Hedi Xia, Vai Suliafu, Hangjie Ji +4
We propose heavy ball neural ordinary differential equations (HBNODEs), leveraging the continuous limit of the classical momentum accelerated gradient descent, to improve neural OD…
AIR-Net: Adaptive and Implicit Regularization Neural Network for Matrix Completion
Zhemin Li, Tao Sun, Hongxia Wang +1
The explicit low-rank regularization, e.g., nuclear norm regularization, has been widely used in imaging sciences. However, it has been found that implicit regularization outperfor…
FMMformer: Efficient and Flexible Transformer via Decomposed Near-field and Far-field Attention
Tan M. Nguyen, Vai Suliafu, Stanley J. Osher +2
We propose FMMformers, a class of efficient and flexible transformers inspired by the celebrated fast multipole method (FMM) for accelerating interacting particle simulation. FMM d…
Robust Certification for Laplace Learning on Geometric Graphs
Matthew Thorpe, Bao Wang
Graph Laplacian (GL)-based semi-supervised learning is one of the most used approaches for classifying nodes in a graph. Understanding and certifying the adversarial robustness of…
Stochastic Gradient Descent with Nonlinear Conjugate Gradient-Style Adaptive Momentum
Bao Wang, Qiang Ye
Momentum plays a crucial role in stochastic gradient-based optimization algorithms for accelerating or improving training deep neural networks (DNNs). In deep learning practice, th…
Differentially Private Federated Learning with Laplacian Smoothing
Zhicong Liang, Bao Wang, Quanquan Gu +2
Federated learning aims to protect data privacy by collaboratively learning a model without sharing private data among users. However, an adversary may still be able to infer the p…
Sparsity Meets Robustness: Channel Pruning for the Feynman-Kac Formalism Principled Robust Deep Neural Nets
Thu Dinh, Bao Wang, Andrea L. Bertozzi +1
Deep neural nets (DNNs) compression is crucial for adaptation to mobile devices. Though many successful algorithms exist to compress naturally trained DNNs, developing efficient an…
Scheduled Restart Momentum for Accelerated Stochastic Gradient Descent
Bao Wang, Tan M. Nguyen, Andrea L. Bertozzi +2
Stochastic gradient descent (SGD) with constant momentum and its variants such as Adam are the optimization algorithms of choice for training deep neural networks (DNNs). Since DNN…
Laplacian Smoothing Stochastic Gradient Markov Chain Monte Carlo
Bao Wang, Difan Zou, Quanquan Gu +1
As an important Markov Chain Monte Carlo (MCMC) method, stochastic gradient Langevin dynamics (SGLD) algorithm has achieved great success in Bayesian learning and posterior samplin…
Graph Interpolating Activation Improves Both Natural and Robust Accuracies in Data-Efficient Deep Learning
Bao Wang, Stanley J. Osher
Improving the accuracy and robustness of deep neural nets (DNNs) and adapting them to small training data are primary tasks in deep learning research. In this paper, we replace the…
DP-LSSGD: A Stochastic Optimization Method to Lift the Utility in Privacy-Preserving ERM
Bao Wang, Quanquan Gu, March Boedihardjo +2
Machine learning (ML) models trained by differentially private stochastic gradient descent (DP-SGD) have much lower utility than the non-private ones. To mitigate this degradation,…
A Deterministic Gradient-Based Approach to Avoid Saddle Points
Lisa Maria Kreusser, Stanley J. Osher, Bao Wang
Loss functions with a large number of saddle points are one of the major obstacles for training modern machine learning models efficiently. First-order methods such as gradient des…
Mathematical Analysis of Adversarial Attacks
Zehao Dou, Stanley J. Osher, Bao Wang
In this paper, we analyze efficacy of the fast gradient sign method (FGSM) and the Carlini-Wagner's L2 (CW-L2) attack. We prove that, within a certain regime, the untargeted FGSM c…
ResNets Ensemble via the Feynman-Kac Formalism to Improve Natural and Robust Accuracies
Bao Wang, Binjie Yuan, Zuoqiang Shi +1
Empirical adversarial risk minimization (EARM) is a widely used mathematical framework to robustly train deep neural nets (DNNs) that are resistant to adversarial attacks. However,…
Adversarial Defense via Data Dependent Activation Function and Total Variation Minimization
Bao Wang, Alex T. Lin, Wei Zhu +3
We improve the robustness of Deep Neural Net (DNN) to adversarial attacks by using an interpolating function as the output activation. This data-dependent activation remarkably imp…
Graph-Based Deep Modeling and Real Time Forecasting of Sparse Spatio-Temporal Data
Bao Wang, Xiyang Luo, Fangbo Zhang +3
We present a generic framework for spatio-temporal (ST) data modeling, analysis, and forecasting, with a special focus on data that is sparse in both space and time. Our multi-scal…
Deep Neural Nets with Interpolating Function as Output Activation
Bao Wang, Xiyang Luo, Zhen Li +3
We replace the output layer of deep neural nets, typically the softmax function, by a novel interpolating function. And we propose end-to-end training and testing algorithms for th…