AdaTerm: Adaptive T-Distribution Estimated Robust Moments for Noise-Robust Stochastic Gradient Optimization

Abstract

With the increasing practicality of deep learning applications, practitioners are inevitably faced with datasets corrupted by noise from various sources such as measurement errors, mislabeling, and estimated surrogate inputs/outputs that can adversely impact the optimization results. It is a common practice to improve the optimization algorithm's robustness to noise, since this algorithm is ultimately in charge of updating the network parameters. Previous studies revealed that the first-order moment used in Adam-like stochastic gradient descent optimizers can be modified based on the Student's t-distribution. While this modification led to noise-resistant updates, the other associated statistics remained unchanged, resulting in inconsistencies in the assumed models. In this paper, we propose AdaTerm, a novel approach that incorporates the Student's t-distribution to derive not only the first-order moment but also all the associated statistics. This provides a unified treatment of the optimization process, offering a comprehensive framework under the statistical model of the t-distribution for the first time. The proposed approach offers several advantages over previously proposed approaches, including reduced hyperparameters and improved robustness and adaptability. This noise-adaptive behavior contributes to AdaTerm's exceptional learning performance, as demonstrated through various optimization problems with different and/or unknown noise ratios. Furthermore, we introduce a new technique for deriving a theoretical regret bound without relying on AMSGrad, providing a valuable contribution to the field

27 pages; Final version accepted by Elsevier Neurocomputing Journal (2023-08; https://doi.org/10.1016/j.neucom.2023.126692)

References in corpus (7)

• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• On the Convergence of Adam and Beyond
• Understanding and Improving Layer Normalization
• Unbiasing Truncated Backpropagation Through Time
• Proximal Policy Optimization with Adaptive Threshold for Symmetric Relative Density Ratio
• Strength of Minibatch Noise in SGD
• Training trajectories, mini-batch losses and the curious role of the learning rate

Cited by in corpus (9)

• Multiplicative update rules for accelerating deep learning training and increasing robustness
• Real-time Sampling-based Model Predictive Control based on Reverse Kullback-Leibler Divergence and Its Adaptive Acceleration
• Consolidated Adaptive T-soft Update for Deep Reinforcement Learning
• Revisiting Experience Replayable Conditions
• Improvements of Dark Experience Replay and Reservoir Sampling towards Better Balance between Consolidation and Plasticity
• LiRA: Light-Robust Adversary for Model-based Reinforcement Learning in Real World
• Variational Adaptive Noise and Dropout towards Stable Recurrent Neural Networks
• Weber-Fechner Law in Temporal Difference learning derived from Control as Inference
• Skin-Machine Interface with Multimodal Contact Motion Classifier