On the Convergence of Adam and Beyond

Abstract

Several recently proposed stochastic optimization methods that have been successfully used in training deep networks such as RMSProp, Adam, Adadelta, Nadam are based on using gradient updates scaled by square roots of exponential moving averages of squared past gradients. In many applications, e.g. learning with large output spaces, it has been empirically observed that these algorithms fail to converge to an optimal solution (or a critical point in nonconvex settings). We show that one cause for such failures is the exponential moving average used in the algorithms. We provide an explicit example of a simple convex optimization setting where Adam does not converge to the optimal solution, and describe the precise problems with the previous analysis of Adam algorithm. Our analysis suggests that the convergence issues can be fixed by endowing such algorithms with `long-term memory' of past gradients, and propose new variants of the Adam algorithm which not only fix the convergence issues but often also lead to improved empirical performance.

arXiv commentAppeared in ICLR 2018

Cited by in corpus (488)

• E(3)-Equivariant Graph Neural Networks for Data-Efficient and Accurate Interatomic Potentials
• PhysNet: A Neural Network for Predicting Energies, Forces, Dipole Moments and Partial Charges
• On the Variance of the Adaptive Learning Rate and Beyond
• Improving Generalization Performance by Switching from Adam to SGD
• SpookyNet: Learning Force Fields with Electronic Degrees of Freedom and Nonlocal Effects
• A Survey of Convolutional Neural Networks: Analysis, Applications, and Prospects
• Robust Motion In-betweening
• A Comparison of Optimization Algorithms for Deep Learning
• Deep Image Translation with an Affinity-Based Change Prior for Unsupervised Multimodal Change Detection
• Fake News Detection on Social Media using Geometric Deep Learning
• On Empirical Comparisons of Optimizers for Deep Learning
• Flipout: Efficient Pseudo-Independent Weight Perturbations on Mini-Batches
• Adafactor: Adaptive Learning Rates with Sublinear Memory Cost
• How Does Learning Rate Decay Help Modern Neural Networks?
• Convolutional neural networks: a magic bullet for gravitational-wave detection?
• Dense Dilated Convolutions Merging Network for Land Cover Classification
• Riemannian Adaptive Optimization Methods
• Improved Techniques for Training Score-Based Generative Models
• Optimization for deep learning: theory and algorithms
• DGCN: Diversified Recommendation with Graph Convolutional Networks
• Adaptive Federated Optimization
• AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradients
• Heterogeneous Domain Generalization via Domain Mixup
• Two-Stage Mesh Deep Learning for Automated Tooth Segmentation and Landmark Localization on 3D Intraoral Scans
• Iterative Joint Image Demosaicking and Denoising using a Residual Denoising Network
• Omni-Scale Feature Learning for Person Re-Identification
• Real-Time Detection of Gravitational Waves from Binary Neutron Stars using Artificial Neural Networks
• Recent Advances in Object Detection in the Age of Deep Convolutional Neural Networks
• Review: Deep Learning in Electron Microscopy
• Gaussian Moments as Physically Inspired Molecular Descriptors for Accurate and Scalable Machine Learning Potentials
• Online Learning: A Modern Introduction Using Convex Optimization
• Federated Learning with Matched Averaging
• A Comparison of Various Classical Optimizers for a Variational Quantum Linear Solver
• A Deep Generative Model for Molecule Optimization via One Fragment Modification
• Torchreid: A Library for Deep Learning Person Re-Identification in Pytorch
• Variational Quantum Boltzmann Machines
• Efficient Two-Stream Network for Violence Detection Using Separable Convolutional LSTM
• Deep Generalized Max Pooling
• On the Convergence Proof of AMSGrad and a New Version
• Mime: Mimicking Centralized Stochastic Algorithms in Federated Learning
• On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimization
• Deep Learning-based Extreme Heatwave Forecast
• Compositional Embeddings Using Complementary Partitions for Memory-Efficient Recommendation Systems
• Quasi-hyperbolic momentum and Adam for deep learning
• Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networks
• Transferable E(3) equivariant parameterization for Hamiltonian of molecules and solids
• Stochastic Conjugate Gradient Algorithm with Variance Reduction
• Why gradient clipping accelerates training: A theoretical justification for adaptivity
• Stochastic Gradient Methods with Layer-wise Adaptive Moments for Training of Deep Networks
• Convergence guarantees for RMSProp and ADAM in non-convex optimization and an empirical comparison to Nesterov acceleration
• AdamP: Slowing Down the Slowdown for Momentum Optimizers on Scale-invariant Weights
• Deep Polynomial Neural Networks
• An FE-DMN method for the multiscale analysis of fiber reinforced plastic components
• On the Convergence of Adaptive Gradient Methods for Nonconvex Optimization
• Dive into Deep Learning
• Parameter-Efficient Person Re-identification in the 3D Space
• Attention-based Audio-Visual Fusion for Robust Automatic Speech Recognition
• Feature-Critic Networks for Heterogeneous Domain Generalization
• A Deep Generative Model for Graph Layout
• Information Scrambling in Quantum Neural Networks
• A Similarity Measure for Material Appearance
• Large Kernel Distillation Network for Efficient Single Image Super-Resolution
• T-LEAP: Occlusion-robust pose estimation of walking cows using temporal information
• Nostalgic Adam: Weighting more of the past gradients when designing the adaptive learning rate
• Detection and Parameter Estimation of Gravitational Waves from Binary Neutron-Star Mergers in Real LIGO Data using Deep Learning
• On Symmetric Losses for Learning from Corrupted Labels
• Fast simulation of muons produced at the SHiP experiment using Generative Adversarial Networks
• AdaTask: A Task-aware Adaptive Learning Rate Approach to Multi-task Learning
• GOT: An Optimal Transport framework for Graph comparison
• Long short-term memory and learning-to-learn in networks of spiking neurons
• Orbital Optimization in Selected Configuration Interaction Methods
• Adam revisited: a weighted past gradients perspective
• Solving Stochastic Compositional Optimization is Nearly as Easy as Solving Stochastic Optimization
• Disentangling Disentanglement in Variational Autoencoders
• OnsagerNet: Learning Stable and Interpretable Dynamics using a Generalized Onsager Principle
• Full-sky Cosmic Microwave Background Foreground Cleaning Using Machine Learning
• Learning Generalisable Omni-Scale Representations for Person Re-Identification
• Accelerating SGD with momentum for over-parameterized learning
• A Survey of Optimization Methods from a Machine Learning Perspective
• Light Field Image Quality Assessment With Auxiliary Learning Based on Depthwise and Anglewise Separable Convolutions
• Multiscale reweighted stochastic embedding (MRSE): Deep learning of collective variables for enhanced sampling
• A Deep Learning Algorithm for High-Dimensional Exploratory Item Factor Analysis
• HyperE2VID: Improving Event-Based Video Reconstruction via Hypernetworks
• Fluctuation-dissipation relations for stochastic gradient descent
• A Selective Overview of Deep Learning
• Adaptivity without Compromise: A Momentumized, Adaptive, Dual Averaged Gradient Method for Stochastic Optimization
• Parameter Efficient Multimodal Transformers for Video Representation Learning
• Principled Weight Initialization for Hypernetworks
• Learning to Select, Track, and Generate for Data-to-Text
• Deep Residual Autoencoders for Expectation Maximization-inspired Dictionary Learning
• Three-body problem -- from Newton to supercomputer plus machine learning
• Universal Adversarial Examples and Perturbations for Quantum Classifiers
• Deep Gamblers: Learning to Abstain with Portfolio Theory
• Variational Bayes survival analysis for unemployment modelling
• Descending through a Crowded Valley - Benchmarking Deep Learning Optimizers
• Sequential Training of Neural Networks with Gradient Boosting
• On Statistical Bias In Active Learning: How and When To Fix It
• Distributed Training with Heterogeneous Data: Bridging Median- and Mean-Based Algorithms
• Equivariant Transformer Networks
• A Simple Convergence Proof of Adam and Adagrad
• Neural-Network Assisted Study of Nitrogen Atom Dynamics on Amorphous Solid Water. I. Adsorption & Desorption
• Mixed Dimension Embeddings with Application to Memory-Efficient Recommendation Systems
• Nemesyst: A Hybrid Parallelism Deep Learning-Based Framework Applied for Internet of Things Enabled Food Retailing Refrigeration Systems
• Densely connected neural networks for nonlinear regression
• Momentum-Based Variance Reduction in Non-Convex SGD
• Budgeted Training: Rethinking Deep Neural Network Training Under Resource Constraints
• QUOTIENT: Two-Party Secure Neural Network Training and Prediction
• Laplacian Smoothing Gradient Descent
• An Adaptive and Momental Bound Method for Stochastic Learning
• A Sufficient Condition for Convergences of Adam and RMSProp
• Towards Understanding Label Smoothing
• AdaShift: Decorrelation and Convergence of Adaptive Learning Rate Methods
• An Empirical Comparison of Optimizers for Quantum Machine Learning with SPSA-based Gradients
• AdaSwarm: Augmenting Gradient-Based optimizers in Deep Learning with Swarm Intelligence
• A deep learning driven pseudospectral PCE based FFT homogenization algorithm for complex microstructures
• Deep Neural Networks for Marine Debris Detection in Sonar Images
• AdaTerm: Adaptive T-Distribution Estimated Robust Moments for Noise-Robust Stochastic Gradient Optimization
• Group Convolutional Neural Networks Improve Quantum State Accuracy
• SE(3)-equivariant prediction of molecular wavefunctions and electronic densities
• Variational Bayesian Approximation of Inverse Problems using Sparse Precision Matrices
• Liberty or Depth: Deep Bayesian Neural Nets Do Not Need Complex Weight Posterior Approximations
• AdaX: Adaptive Gradient Descent with Exponential Long Term Memory
• DeepSym: Deep Symbol Generation and Rule Learning from Unsupervised Continuous Robot Interaction for Planning
• Riemannian adaptive stochastic gradient algorithms on matrix manifolds
• Reservoirs learn to learn
• Painless Stochastic Gradient: Interpolation, Line-Search, and Convergence Rates
• Deep Frank-Wolfe For Neural Network Optimization
• SGD Converges to Global Minimum in Deep Learning via Star-convex Path
• Segmentation of Skin Lesions and their Attributes Using Multi-Scale Convolutional Neural Networks and Domain Specific Augmentations
• XNAS: Neural Architecture Search with Expert Advice
• Polysemous Visual-Semantic Embedding for Cross-Modal Retrieval
• Data efficiency and extrapolation trends in neural network interatomic potentials
• FedH2L: Federated Learning with Model and Statistical Heterogeneity
• Local Adaptivity in Federated Learning: Convergence and Consistency
• Decompose to manipulate: Manipulable Object Synthesis in 3D Medical Images with Structured Image Decomposition
• ToyADMOS: A Dataset of Miniature-Machine Operating Sounds for Anomalous Sound Detection
• STEM: A Stochastic Two-Sided Momentum Algorithm Achieving Near-Optimal Sample and Communication Complexities for Federated Learning
• First-order Convergence Theory for Weakly-Convex-Weakly-Concave Min-max Problems
• Towards Unified INT8 Training for Convolutional Neural Network
• Towards Better Understanding of Adaptive Gradient Algorithms in Generative Adversarial Nets
• A Better Alternative to Error Feedback for Communication-Efficient Distributed Learning
• Generative Latent Flow
• AUNet: Attention-guided dense-upsampling networks for breast mass segmentation in whole mammograms
• Learning from Positive and Unlabeled Data with Arbitrary Positive Shift
• Bayesian Neural Networks for Fast SUSY Predictions
• Does Adam optimizer keep close to the optimal point?
• Quantum Generative Adversarial Networks For Anomaly Detection In High Energy Physics
• Mixing ADAM and SGD: a Combined Optimization Method
• Evaluating uncertainties in electrochemical impedance spectra of solid oxide fuel cells
• On the Convergence of AdaBound and its Connection to SGD
• From FiLM to Video: Multi-turn Question Answering with Multi-modal Context
• Structure Probing Neural Network Deflation
• SRNet: Improving Generalization in 3D Human Pose Estimation with a Split-and-Recombine Approach
• CoolMomentum: A Method for Stochastic Optimization by Langevin Dynamics with Simulated Annealing
• Second-Order Neural Dependency Parsing with Message Passing and End-to-End Training
• Towards Robust, Locally Linear Deep Networks
• Competence-based Curriculum Learning for Neural Machine Translation
• Classification of Upper Arm Movements from EEG signals using Machine Learning with ICA Analysis
• Adaptive Gradient Descent without Descent
• A Weakly Supervised Approach for Estimating Spatial Density Functions from High-Resolution Satellite Imagery
• Siamese Capsule Networks
• An Information-Theoretic Optimality Principle for Deep Reinforcement Learning
• Fact-based Text Editing
• Estimation of discrete choice models with hybrid stochastic adaptive batch size algorithms
• Extraction of hierarchical functional connectivity components in human brain using resting-state fMRI
• Learning to Optimize in Model Predictive Control
• Robust Deep Reinforcement Learning through Adversarial Loss
• Towards Practical Adam: Non-Convexity, Convergence Theory, and Mini-Batch Acceleration
• Detection of Illicit Drug Trafficking Events on Instagram: A Deep Multimodal Multilabel Learning Approach
• Unbiased MLMC stochastic gradient-based optimization of Bayesian experimental designs
• Temporal-Clustering Invariance in Irregular Healthcare Time Series
• Effective Federated Adaptive Gradient Methods with Non-IID Decentralized Data
• Heterogeneous Molecular Graph Neural Networks for Predicting Molecule Properties
• LaProp: Separating Momentum and Adaptivity in Adam
• Weakly-supervised Learning For Catheter Segmentation in 3D Frustum Ultrasound
• Neural Pitch-Shifting and Time-Stretching with Controllable LPCNet
• Microplankton life histories revealed by holographic microscopy and deep learning
• A High Probability Analysis of Adaptive SGD with Momentum
• The Role of Memory in Stochastic Optimization
• WeMix: How to Better Utilize Data Augmentation
• TensorBNN: Bayesian Inference for Neural Networks using Tensorflow
• Reward Estimation for Variance Reduction in Deep Reinforcement Learning
• P-CRITICAL: A Reservoir Autoregulation Plasticity Rule for Neuromorphic Hardware
• SCG-Net: Self-Constructing Graph Neural Networks for Semantic Segmentation
• A Second look at Exponential and Cosine Step Sizes: Simplicity, Adaptivity, and Performance
• Adam${}^{+}$: A Stochastic Method with Adaptive Variance Reduction
• Replay attack detection with complementary high-resolution information using end-to-end DNN for the ASVspoof 2019 Challenge
• RawNet: Advanced end-to-end deep neural network using raw waveforms for text-independent speaker verification
• Learning Generalizable Visual Representations via Interactive Gameplay
• FlowSeq: Non-Autoregressive Conditional Sequence Generation with Generative Flow
• Anomaly Detection for Industrial Control Systems Using Sequence-to-Sequence Neural Networks
• Secure Quantized Training for Deep Learning
• Machine learning with tree tensor networks, CP rank constraints, and tensor dropout
• Kalman Gradient Descent: Adaptive Variance Reduction in Stochastic Optimization
• Multiplicative update rules for accelerating deep learning training and increasing robustness
• Deep reinforcement learning approach to MIMO precoding problem: Optimality and Robustness
• Newton-type Methods for Minimax Optimization
• Hyperbolic Deep Neural Networks: A Survey
• MintNet: Building Invertible Neural Networks with Masked Convolutions
• Mutually exciting point process graphs for modelling dynamic networks
• Building Computationally Efficient and Well-Generalizing Person Re-Identification Models with Metric Learning
• Automatic differentiation and the optimization of differential equation models in biology
• Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling
• Deep Fictitious Play for Stochastic Differential Games
• Stochastic Sign Descent Methods: New Algorithms and Better Theory
• Compositional ADAM: An Adaptive Compositional Solver
• AdaS: Adaptive Scheduling of Stochastic Gradients
• LiftFormer: 3D Human Pose Estimation using attention models
• Accelerated Large Batch Optimization of BERT Pretraining in 54 minutes
• Explicitly Conditioned Melody Generation: A Case Study with Interdependent RNNs
• Non-asymptotic Convergence of Adam-type Reinforcement Learning Algorithms under Markovian Sampling
• Scalable Psychological Momentum Forecasting in Esports
• Non-Convergence and Limit Cycles in the Adam optimizer
• Classification from Positive, Unlabeled and Biased Negative Data
• Joint Online Parameter Estimation and Optimal Sensor Placement for the Partially Observed Stochastic Advection-Diffusion Equation
• A Deep Generative Model for Reordering Adjacency Matrices
• Real-time Denoising and Dereverberation with Tiny Recurrent U-Net
• TurboSVM-FL: Boosting Federated Learning through SVM Aggregation for Lazy Clients
• An Adaptive Gradient Method with Energy and Momentum
• Online Smoothing for Diffusion Processes Observed with Noise
• Stochastic Gradient Descent with Nonlinear Conjugate Gradient-Style Adaptive Momentum
• Multivariate, Multistep Forecasting, Reconstruction and Feature Selection of Ocean Waves via Recurrent and Sequence-to-Sequence Networks
• Many-Joint Robot Arm Control with Recurrent Spiking Neural Networks
• SUMO: Unbiased Estimation of Log Marginal Probability for Latent Variable Models
• Large-Scale Deep Learning Optimizations: A Comprehensive Survey
• Towards Efficient and Unbiased Implementation of Lipschitz Continuity in GANs
• Positive-Negative Momentum: Manipulating Stochastic Gradient Noise to Improve Generalization
• Human Pose Estimation from Ambiguous Pressure Recordings with Spatio-temporal Masked Transformers
• Private Stochastic Non-Convex Optimization: Adaptive Algorithms and Tighter Generalization Bounds
• Universal Stagewise Learning for Non-Convex Problems with Convergence on Averaged Solutions
• Competing Against Equilibria in Zero-Sum Games with Evolving Payoffs
• Neural Density Estimation and Likelihood-free Inference
• Deep Neural Network Approach to Forward-Inverse Problems
• TAdam: A Robust Stochastic Gradient Optimizer
• SceneGraphFusion: Incremental 3D Scene Graph Prediction from RGB-D Sequences
• Graph Attention Network based Pruning for Reconstructing 3D Liver Vessel Morphology from Contrasted CT Images
• NASI: Label- and Data-agnostic Neural Architecture Search at Initialization
• Oblique Decision Trees from Derivatives of ReLU Networks
• Adaptive Temporal Difference Learning with Linear Function Approximation
• Disentangling Adaptive Gradient Methods from Learning Rates
• Autoencoder-Based Incremental Class Learning without Retraining on Old Data
• Adaptive First-and Zeroth-order Methods for Weakly Convex Stochastic Optimization Problems
• Preictal Period Optimization for Deep Learning-Based Epileptic Seizure Prediction
• Improved RawNet with Feature Map Scaling for Text-independent Speaker Verification using Raw Waveforms
• Aggregated Momentum: Stability Through Passive Damping
• Linear Convergence of Adaptive Stochastic Gradient Descent
• Rayleigh-Gauss-Newton optimization with enhanced sampling for variational Monte Carlo
• Proximal Adam: Robust Adaptive Update Scheme for Constrained Optimization
• BiAdam: Fast Adaptive Bilevel Optimization Methods
• A Qualitative Study of the Dynamic Behavior for Adaptive Gradient Algorithms
• Convergence Analysis of a Momentum Algorithm with Adaptive Step Size for Non Convex Optimization
• A General Family of Stochastic Proximal Gradient Methods for Deep Learning
• Stochastic Gradient Descent with Polyak's Learning Rate
• A Decentralized Adaptive Momentum Method for Solving a Class of Min-Max Optimization Problems
• Escaping Saddle Points Faster with Stochastic Momentum
• ARC: A Vision-based Automatic Retail Checkout System
• Versatile and Robust Transient Stability Assessment via Instance Transfer Learning
• Robust and efficient algorithms for high-dimensional black-box quantum optimization
• Hessian based analysis of SGD for Deep Nets: Dynamics and Generalization
• Training Neural Networks for and by Interpolation
• FNet II: Spectral Classification of Quasars, Galaxies, Stars, and broad absorption line (BAL) Quasars
• Deep learning insights into non-universality in the halo mass function
• Understanding the Generalization of Adam in Learning Neural Networks with Proper Regularization
• DeepOBS: A Deep Learning Optimizer Benchmark Suite
• Near-optimal Local Convergence of Alternating Gradient Descent-Ascent for Minimax Optimization
• The Role of Momentum Parameters in the Optimal Convergence of Adaptive Polyak's Heavy-ball Methods
• Stochastic Trust Region Methods with Trust Region Radius Depending on Probabilistic Models
• Phase-aware Single-stage Speech Denoising and Dereverberation with U-Net
• On the Effectiveness of Regularization Against Membership Inference Attacks
• Distance-Based Regularisation of Deep Networks for Fine-Tuning
• On the Convergence of Decentralized Adaptive Gradient Methods
• Conjugate-gradient-based Adam for stochastic optimization and its application to deep learning
• A new regret analysis for Adam-type algorithms
• AdaSGD: Bridging the gap between SGD and Adam
• Convergence of a Relaxed Variable Splitting Method for Learning Sparse Neural Networks via ${\ell }_1,{\ell }_0$, and transformed-${\ell }_1$ Penalties
• Estimating initial conditions for dynamical systems with incomplete information
• Second-Order Semantic Dependency Parsing with End-to-End Neural Networks
• Learning by Turning: Neural Architecture Aware Optimisation
• BigSurvSGD: Big Survival Data Analysis via Stochastic Gradient Descent
• Projection-Free Adaptive Gradients for Large-Scale Optimization
• Exploiting Adam-like Optimization Algorithms to Improve the Performance of Convolutional Neural Networks
• FCM-RDpA: TSK Fuzzy Regression Model Construction Using Fuzzy C-Means Clustering, Regularization, DropRule, and Powerball AdaBelief
• Gradient descent with momentum --- to accelerate or to super-accelerate?
• Joint Constrained Learning for Event-Event Relation Extraction
• AdaGDA: Faster Adaptive Gradient Descent Ascent Methods for Minimax Optimization
• Adaptive Step Sizes in Variance Reduction via Regularization
• Learning and Segmenting Dense Voxel Embeddings for 3D Neuron Reconstruction
• Adaptivity and Optimality: A Universal Algorithm for Online Convex Optimization
• An Element-Wise Weights Aggregation Method for Federated Learning
• Deep Discriminative Representation Learning with Attention Map for Scene Classification
• Non-Asymptotic Analysis of Stochastic Approximation Algorithms for Streaming Data
• Convergence of adaptive algorithms for weakly convex constrained optimization
• Early Detection of Fake News by Utilizing the Credibility of News, Publishers, and Users Based on Weakly Supervised Learning
• MixML: A Unified Analysis of Weakly Consistent Parallel Learning
• Recursive Estimation for Sparse Gaussian Process Regression
• Convergence Analyses of Online ADAM Algorithm in Convex Setting and Two-Layer ReLU Neural Network
• Stability of SGD: Tightness Analysis and Improved Bounds
• BCQQ: Batch-Constraint Quantum Q-Learning with Cyclic Data Re-uploading
• Sinkhorn Natural Gradient for Generative Models
• Learning from Imperfect Annotations
• Adaptive Serverless Learning
• The Nonlinearity Coefficient - A Practical Guide to Neural Architecture Design
• On the Distributional Properties of Adaptive Gradients
• Exploring Severe Occlusion: Multi-Person 3D Pose Estimation with Gated Convolution
• Attentive Filtering Networks for Audio Replay Attack Detection
• Explainable AI for ML jet taggers using expert variables and layerwise relevance propagation
• FaSTExt: Fast and Small Text Extractor
• Riemannian Adaptive Optimization Algorithm and Its Application to Natural Language Processing
• Orthogonal Over-Parameterized Training
• On the Universality of the Double Descent Peak in Ridgeless Regression
• Better SGD using Second-order Momentum
• Momentum Accelerates the Convergence of Stochastic AUPRC Maximization
• High-probability Bounds for Non-Convex Stochastic Optimization with Heavy Tails
• Universal Dependency Parsing with a General Transition-Based DAG Parser
• Multi-Channel and Multi-Microphone Acoustic Echo Cancellation Using A Deep Learning Based Approach
• Probing What Different NLP Tasks Teach Machines about Function Word Comprehension
• High-order Semantic Role Labeling
• Expectigrad: Fast Stochastic Optimization with Robust Convergence Properties
• Exploiting the Full Capacity of Deep Neural Networks while Avoiding Overfitting by Targeted Sparsity Regularization
• Unsupervised Disentanglement GAN for Domain Adaptive Person Re-Identification
• Representing Partial Programs with Blended Abstract Semantics
• Variational Representations and Neural Network Estimation of Rényi Divergences
• The Development of Spatial Attention U-Net for The Recovery of Ionospheric Measurements and The Extraction of Ionospheric Parameters
• ${\mathcal{C}}^k$-continuous Spline Approximation with TensorFlow Gradient Descent Optimizers
• Learning IMM Filter Parameters from Measurements using Gradient Descent
• Doubly Adaptive Scaled Algorithm for Machine Learning Using Second-Order Information
• Improving Human Text Comprehension through Semi-Markov CRF-based Neural Section Title Generation
• Scalable and Accurate Dialogue State Tracking via Hierarchical Sequence Generation
• Cross-lingual Entity Alignment with Incidental Supervision
• An Imitation Game for Learning Semantic Parsers from User Interaction
• Adaptively Preconditioned Stochastic Gradient Langevin Dynamics
• Mapping in a cycle: Sinkhorn regularized unsupervised learning for point cloud shapes
• Differentiable Visual Computing
• Importance Weighted Hierarchical Variational Inference
• A Fully Stochastic Second-Order Trust Region Method
• Rethinking movie genre classification with fine-grained semantic clustering
• FastAdaBelief: Improving Convergence Rate for Belief-based Adaptive Optimizers by Exploiting Strong Convexity
• SAdam: A Variant of Adam for Strongly Convex Functions
• On the Trend-corrected Variant of Adaptive Stochastic Optimization Methods
• On the One-sided Convergence of Adam-type Algorithms in Non-convex Non-concave Min-max Optimization
• Characterizing the Value of Information in Medical Notes
• Optimizer Fusion: Efficient Training with Better Locality and Parallelism
• Unsupervised Learning of Camera Pose with Compositional Re-estimation
• Tensor-Train Networks for Learning Predictive Modeling of Multidimensional Data
• An Orthogonal-SGD based Learning Approach for MIMO Detection under Multiple Channel Models
• "What Are You Trying to Do?" Semantic Typing of Event Processes
• Quantized Adam with Error Feedback
• Learning Partial Differential Equations in Reproducing Kernel Hilbert Spaces
• Adaptive Gradient Methods Can Be Provably Faster than SGD after Finite Epochs
• HBONet: Harmonious Bottleneck on Two Orthogonal Dimensions
• Gravity Optimizer: a Kinematic Approach on Optimization in Deep Learning
• Time-Delay Momentum: A Regularization Perspective on the Convergence and Generalization of Stochastic Momentum for Deep Learning
• Improving Irregularly Sampled Time Series Learning with Dense Descriptors of Time
• Adam with Bandit Sampling for Deep Learning
• Learning scale-variant features for robust iris authentication with deep learning based ensemble framework
• A study on the role of subsidiary information in replay attack spoofing detection
• An Optimistic Acceleration of AMSGrad for Nonconvex Optimization
• Approximated Orthonormal Normalisation in Training Neural Networks
• Reverse engineering learned optimizers reveals known and novel mechanisms
• Applying Cyclical Learning Rate to Neural Machine Translation
• Full Workspace Generation of Serial-link Manipulators by Deep Learning based Jacobian Estimation
• Microscopy Image Restoration using Deep Learning on W2S
• Deep Reinforced Attention Learning for Quality-Aware Visual Recognition
• PoseConvGRU: A Monocular Approach for Visual Ego-motion Estimation by Learning
• Incorporating the Barzilai-Borwein Adaptive Step Size into Sugradient Methods for Deep Network Training
• Parallel and distributed asynchronous adaptive stochastic gradient methods
• Adaptive Differentially Private Empirical Risk Minimization
• Learn Faster and Forget Slower via Fast and Stable Task Adaptation
• Robustness via Cross-Domain Ensembles
• Choice Set Optimization Under Discrete Choice Models of Group Decisions
• A Survey on Large-scale Machine Learning
• Exemplar-based synthesis of geology using kernel discrepancies and generative neural networks
• Learning Bilingual Word Embeddings Using Lexical Definitions
• Constructing unbiased gradient estimators with finite variance for conditional stochastic optimization
• Stochastic Gradient Methods with Block Diagonal Matrix Adaptation
• DEAM: Adaptive Momentum with Discriminative Weight for Stochastic Optimization
• L2M: Practical posterior Laplace approximation with optimization-driven second moment estimation
• AdaL: Adaptive Gradient Transformation Contributes to Convergences and Generalizations
• Anatomical and Diagnostic Bayesian Segmentation in Prostate MRI $-$Should Different Clinical Objectives Mandate Different Loss Functions?
• Data Augmentation for Text Generation Without Any Augmented Data
• Compressed Communication for Distributed Training: Adaptive Methods and System
• Loss Landscape Dependent Self-Adjusting Learning Rates in Decentralized Stochastic Gradient Descent
• PCLs: Geometry-aware Neural Reconstruction of 3D Pose with Perspective Crop Layers
• A Stochastic Objective-Function-Free Adaptive Regularization Method with Optimal Complexity
• On Adaptive Stochastic Optimization for Streaming Data: A Newton's Method with O(dN) Operations
• Competition analysis on the over-the-counter credit default swap market
• Accelerated Almost-Sure Convergence Rates for Nonconvex Stochastic Gradient Descent using Stochastic Learning Rates
• Momentum Centering and Asynchronous Update for Adaptive Gradient Methods
• Implicit Quantile Neural Networks for Jet Simulation and Correction
• Communication-Compressed Adaptive Gradient Method for Distributed Nonconvex Optimization
• Learning to Be Cautious
• Solving Differential Equations via Continuous-Variable Quantum Computers
• CADA: Communication-Adaptive Distributed Adam
• Estimating Reproducible Functional Networks Associated with Task Dynamics using Unsupervised LSTMs
• Universal Online Convex Optimization Meets Second-order Bounds
• Polygonal Unadjusted Langevin Algorithms: Creating stable and efficient adaptive algorithms for neural networks
• $k$-Neighbor Based Curriculum Sampling for Sequence Prediction
• Statistical Measures For Defining Curriculum Scoring Function
• On the Last Iterate Convergence of Momentum Methods
• Generalizing Adversarial Examples by AdaBelief Optimizer
• Training Aware Sigmoidal Optimizer
• Probabilistic Federated Learning of Neural Networks Incorporated with Global Posterior Information
• Universal Representation Learning of Knowledge Bases by Jointly Embedding Instances and Ontological Concepts
• Spatio-Temporal Neural Network for Fitting and Forecasting COVID-19
• Hierarchical Multitask Learning Approach for BERT
• An Approximation Algorithm for Optimal Subarchitecture Extraction
• Extraction of Hierarchical Functional Connectivity Components in human brain using Adversarial Learning
• Distributed Stochastic Non-Convex Optimization: Momentum-Based Variance Reduction
• Segment Aggregation for short utterances speaker verification using raw waveforms
• StRDAN: Synthetic-to-Real Domain Adaptation Network for Vehicle Re-Identification
• Discrete Point Flow Networks for Efficient Point Cloud Generation
• Road Mapping In LiDAR Images Using A Joint-Task Dense Dilated Convolutions Merging Network
• Dense Dilated Convolutions Merging Network for Semantic Mapping of Remote Sensing Images
• CProp: Adaptive Learning Rate Scaling from Past Gradient Conformity
• RawNet: Fast End-to-End Neural Vocoder
• SPI-Optimizer: an integral-Separated PI Controller for Stochastic Optimization
• Universal Dependency Parsing from Scratch
• Multi-vision Attention Networks for On-line Red Jujube Grading
• Positive-Unlabeled Classification under Class Prior Shift and Asymmetric Error
• Power Gradient Descent
• Deep Pepper: Expert Iteration based Chess agent in the Reinforcement Learning Setting
• Predictive Local Smoothness for Stochastic Gradient Methods
• Batch Uniformization for Minimizing Maximum Anomaly Score of DNN-based Anomaly Detection in Sounds
• An Adaptive Remote Stochastic Gradient Method for Training Neural Networks
• Improving Visual Recognition using Ambient Sound for Supervision
• Variance Regularization for Accelerating Stochastic Optimization
• Hierarchical BiGraph Neural Network as Recommendation Systems
• Flexible numerical optimization with ensmallen
• Semi-Implicit Back Propagation
• MaxVA: Fast Adaptation of Step Sizes by Maximizing Observed Variance of Gradients
• Adaptive Hierarchical Hyper-gradient Descent
• A New Adaptive Gradient Method with Gradient Decomposition
• Learning Metrics from Mean Teacher: A Supervised Learning Method for Improving the Generalization of Speaker Verification System
• A Modular Analysis of Provable Acceleration via Polyak's Momentum: Training a Wide ReLU Network and a Deep Linear Network
• Neograd: Near-Ideal Gradient Descent
• How much progress have we made in neural network training? A New Evaluation Protocol for Benchmarking Optimizers
• Adaptive Gradient Method with Resilience and Momentum
• Neural Network Training Techniques Regularize Optimization Trajectory: An Empirical Study
• Local Convergence of Adaptive Gradient Descent Optimizers
• Neural representation and generation for RNA secondary structures
• Parabolic Approximation Line Search for DNNs
• Introducing an experimental distortion-tolerant speech encryption scheme for secure voice communication
• AsymptoticNG: A regularized natural gradient optimization algorithm with look-ahead strategy
• Adam Induces Implicit Weight Sparsity in Rectifier Neural Networks
• The Effect of Efficient Messaging and Input Variability on Neural-Agent Iterated Language Learning
• Efficient Semi-Implicit Variational Inference
• Adaptive Learning Rate and Momentum for Training Deep Neural Networks
• Identifying Illicit Drug Dealers on Instagram with Large-scale Multimodal Data Fusion
• Convergence of a Relaxed Variable Splitting Coarse Gradient Descent Method for Learning Sparse Weight Binarized Activation Neural Networks
• Learning Robust Hierarchical Patterns of Human Brain across Many fMRI Studies
• Practical Bayesian Learning of Neural Networks via Adaptive Optimisation Methods
• A Comprehensive Study on Optimization Strategies for Gradient Descent In Deep Learning
• ConCAD: Contrastive Learning-based Cross Attention for Sleep Apnea Detection
• Local AdaGrad-Type Algorithm for Stochastic Convex-Concave Optimization
• Rapidly Adapting Moment Estimation
• A Neural Frequency-Severity Model and Its Application to Insurance Claims
• GoPRONTO: a Feedback-based Framework for Nonlinear Optimal Control
• A theoretical and empirical study of new adaptive algorithms with additional momentum steps and shifted updates for stochastic non-convex optimization
• Intuitive Shape Editing in Latent Space
• Active Learning for Argument Mining: A Practical Approach
• Frequency-aware SGD for Efficient Embedding Learning with Provable Benefits
• Solving reward-collecting problems with UAVs: a comparison of online optimization and Q-learning
• Accelerating Distributed SGD for Linear Regression using Iterative Pre-Conditioning
• On Generalization of Adaptive Methods for Over-parameterized Linear Regression
• Learning Constraints and Descriptive Segmentation for Subevent Detection
• Implicit Quantile Networks For Emulation in Jet Physics
• A Study of Policy Gradient on a Class of Exactly Solvable Models
• Machine learning analysis of Photometric data from the Dark Energy Survey
• Automatic Live Music Song Identification Using Multi-level Deep Sequence Similarity Learning
• Learning to Represent Bilingual Dictionaries
• 2nd-order Updates with 1st-order Complexity
• Scaling transition from momentum stochastic gradient descent to plain stochastic gradient descent
• Bayesian filtering unifies adaptive and non-adaptive neural network optimization methods
• ${\bar{G}}_{mst}$:An Unbiased Stratified Statistic and a Fast Gradient Optimization Algorithm Based on It
• Understanding Modern Techniques in Optimization: Frank-Wolfe, Nesterov's Momentum, and Polyak's Momentum
• Do place cells dream of conditional probabilities? Learning Neural Nyström representations
• The Number of Steps Needed for Nonconvex Optimization of a Deep Learning Optimizer is a Rational Function of Batch Size
• Toward Communication Efficient Adaptive Gradient Method
• Tom: Leveraging trend of the observed gradients for faster convergence
• Learning Only from Relevant Keywords and Unlabeled Documents
• A Search-based Neural Model for Biomedical Nested and Overlapping Event Detection
• Automatic and Simultaneous Adjustment of Learning Rate and Momentum for Stochastic Gradient Descent
• Discovering Invariances in Healthcare Neural Networks
• PDE-Inspired Algorithms for Semi-Supervised Learning on Point Clouds
• A straightforward line search approach on the expected empirical loss for stochastic deep learning problems
• Retrofitting Contextualized Word Embeddings with Paraphrases
• BAMSProd: A Step towards Generalizing the Adaptive Optimization Methods to Deep Binary Model
• QuantNet: Learning to Quantize by Learning within Fully Differentiable Framework