An overview of gradient descent optimization algorithms

Abstract

Gradient descent optimization algorithms, while increasingly popular, are often used as black-box optimizers, as practical explanations of their strengths and weaknesses are hard to come by. This article aims to provide the reader with intuitions with regard to the behaviour of different algorithms that will allow her to put them to use. In the course of this overview, we look at different variants of gradient descent, summarize challenges, introduce the most common optimization algorithms, review architectures in a parallel and distributed setting, and investigate additional strategies for optimizing gradient descent.

Added derivations of AdaMax and Nadam

References in corpus (1)

• ADADELTA: An Adaptive Learning Rate Method

Cited by in corpus (624)

• MultiResUNet : Rethinking the U-Net Architecture for Multimodal Biomedical Image Segmentation
• Machine Learning Force Fields
• Adaptive activation functions accelerate convergence in deep and physics-informed neural networks
• Machine learning in cardiovascular flows modeling: Predicting arterial blood pressure from non-invasive 4D flow MRI data using physics-informed neural networks
• Regularized Deep Networks in Intelligent Transportation Systems: A Taxonomy and a Case Study
• Recent Advances in Recurrent Neural Networks
• A Fourth-Generation High-Dimensional Neural Network Potential with Accurate Electrostatics Including Non-local Charge Transfer
• A Review on Deep Learning in UAV Remote Sensing
• Hyper-Parameter Optimization: A Review of Algorithms and Applications
• Extended dynamic mode decomposition with dictionary learning: a data-driven adaptive spectral decomposition of the Koopman operator
• Deep Autoencoder based Energy Method for the Bending, Vibration, and Buckling Analysis of Kirchhoff Plates
• The Challenge of Machine Learning in Space Weather Nowcasting and Forecasting
• Train longer, generalize better: closing the generalization gap in large batch training of neural networks
• Efficient training of physics-informed neural networks via importance sampling
• Physically Interpretable Neural Networks for the Geosciences: Applications to Earth System Variability
• Machine learning approximation algorithms for high-dimensional fully nonlinear partial differential equations and second-order backward stochastic differential equations
• Combining Planning and Deep Reinforcement Learning in Tactical Decision Making for Autonomous Driving
• Deep learning electromagnetic inversion with convolutional neural networks
• Forecasting day-ahead electricity prices in Europe: the importance of considering market integration
• A Comparison of Optimization Algorithms for Deep Learning
• ROSEFusion: Random Optimization for Online Dense Reconstruction under Fast Camera Motion
• Federated Collaborative Filtering for Privacy-Preserving Personalized Recommendation System
• A feature agnostic approach for glaucoma detection in OCT volumes
• Backflow Transformations via Neural Networks for Quantum Many-Body Wave-Functions
• A comparison of LSTM and GRU networks for learning symbolic sequences
• Deep learning observables in computational fluid dynamics
• GHRS: Graph-based Hybrid Recommendation System with Application to Movie Recommendation
• Getting High: High Fidelity Simulation of High Granularity Calorimeters with High Speed
• Optimization for deep learning: theory and algorithms
• Deep Convolutions for In-Depth Automated Rock Typing
• Deep Learning in Mobile and Wireless Networking: A Survey
• Self-Adaptive Physics-Informed Neural Networks using a Soft Attention Mechanism
• An improvement of the convergence proof of the ADAM-Optimizer
• Quantum autoencoders to denoise quantum data
• Optimize TSK Fuzzy Systems for Classification Problems: Mini-Batch Gradient Descent with Uniform Regularization and Batch Normalization
• Neural Machine Translation and Sequence-to-sequence Models: A Tutorial
• Digitized-counterdiabatic quantum approximate optimization algorithm
• Supervised Contrastive Learning
• Illuminating Generalization in Deep Reinforcement Learning through Procedural Level Generation
• Pricing options and computing implied volatilities using neural networks
• DeePore: a deep learning workflow for rapid and comprehensive characterization of porous materials
• Variants of RMSProp and Adagrad with Logarithmic Regret Bounds
• Review: Deep Learning in Electron Microscopy
• A deep learning approach to cosmological dark energy models
• Self-Supervised Multisensor Change Detection
• What Do We Understand About Convolutional Networks?
• Channel Estimation for One-Bit Multiuser Massive MIMO Using Conditional GAN
• Knowledge-Preserving Incremental Social Event Detection via Heterogeneous GNNs
• Whetstone: A Method for Training Deep Artificial Neural Networks for Binary Communication
• Deep learning of thermodynamics-aware reduced-order models from data
• Artificial Neural Networks-Based Machine Learning for Wireless Networks: A Tutorial
• RCA-IUnet: A residual cross-spatial attention guided inception U-Net model for tumor segmentation in breast ultrasound imaging
• Biology and medicine in the landscape of quantum advantages
• Structure-preserving neural networks
• Ship Performance Monitoring using Machine-learning
• Learning to fail: Predicting fracture evolution in brittle material models using recurrent graph convolutional neural networks
• Quasi-hyperbolic momentum and Adam for deep learning
• A measurement-based variational quantum eigensolver
• Solving the Kolmogorov PDE by means of deep learning
• Understanding Data Storage and Ingestion for Large-Scale Deep Recommendation Model Training
• Boosting multiple sclerosis lesion segmentation through attention mechanism
• An Incremental Dimensionality Reduction Method for Visualizing Streaming Multidimensional Data
• A Tutorial on Deep Learning for Music Information Retrieval
• Constraint-Aware Neural Networks for Riemann Problems
• NSGA-Net: Neural Architecture Search using Multi-Objective Genetic Algorithm
• Common pulse retrieval algorithm: a fast and universal method to retrieve ultrashort pulses
• An Extensive Study on Cross-Dataset Bias and Evaluation Metrics Interpretation for Machine Learning applied to Gastrointestinal Tract Abnormality Classification
• Adaptive dynamic programming for nonaffine nonlinear optimal control problem with state constraints
• The Evolution of Distributed Systems for Graph Neural Networks and their Origin in Graph Processing and Deep Learning: A Survey
• Short-term forecasting of solar irradiance without local telemetry: a generalized model using satellite data
• SoftAdapt: Techniques for Adaptive Loss Weighting of Neural Networks with Multi-Part Loss Functions
• Supervised learning in a mechanical system
• Federated Multi-view Matrix Factorization for Personalized Recommendations
• New SAR target recognition based on YOLO and very deep multi-canonical correlation analysis
• Salient Object Detection with Lossless Feature Reflection and Weighted Structural Loss
• Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropout
• Physics-informed neural networks method in high-dimensional integrable systems
• Learning Large Neighborhood Search for Vehicle Routing in Airport Ground Handling
• Neural-network quantum state tomography
• Evolutionary algorithms for hyperparameter optimization in machine learning for application in high energy physics
• Real-time Plant Health Assessment Via Implementing Cloud-based Scalable Transfer Learning On AWS DeepLens
• Neural Network Models for the Anisotropic Reynolds Stress Tensor in Turbulent Channel Flow
• Fast simulation of muons produced at the SHiP experiment using Generative Adversarial Networks
• Analysis and Optimization of Convolutional Neural Network Architectures
• Making DeepFakes more spurious: evading deep face forgery detection via trace removal attack
• A Temporal Graph Neural Network for Cyber Attack Detection and Localization in Smart Grids
• Deep Learning as a Parton Shower
• A State Space Approach for Piecewise-Linear Recurrent Neural Networks for Reconstructing Nonlinear Dynamics from Neural Measurements
• SiTGRU: Single-Tunnelled Gated Recurrent Unit for Abnormality Detection
• Improving Galaxy Clustering Measurements with Deep Learning: analysis of the DECaLS DR7 data
• Cross-dimensional transfer learning in medical image segmentation with deep learning
• GLOBALEMU: A novel and robust approach for emulating the sky-averaged 21-cm signal from the cosmic dawn and epoch of reionisation
• Strong error analysis for stochastic gradient descent optimization algorithms
• Deep residual detection of radio frequency interference for FAST
• RobustART: Benchmarking Robustness on Architecture Design and Training Techniques
• Generative Modeling of Turbulence
• Radio Galaxy Zoo: Unsupervised Clustering of Convolutionally Auto-encoded Radio-astronomical Images
• INFERNO: Inference-Aware Neural Optimisation
• Deep Multi-Task Learning for Malware Image Classification
• Interpreting recurrent neural networks behaviour via excitable network attractors
• Primordial non-Gaussianity from the Completed SDSS-IV extended Baryon Oscillation Spectroscopic Survey I: Catalogue Preparation and Systematic Mitigation
• Deep multi-survey classification of variable stars
• AirLab: Autograd Image Registration Laboratory
• Clickbait Detection in Tweets Using Self-attentive Network
• Application of Machine Learning in Wireless Networks: Key Techniques and Open Issues
• Adorym: A multi-platform generic x-ray image reconstruction framework based on automatic differentiation
• Scalable Deep Learning on Distributed Infrastructures: Challenges, Techniques and Tools
• Short term prediction of demand for ride hailing services: A deep learning approach
• A Survey of Optimization Methods from a Machine Learning Perspective
• Robust and Subject-Independent Driving Manoeuvre Anticipation through Domain-Adversarial Recurrent Neural Networks
• Federated Learning with Nesterov Accelerated Gradient
• Seismic Facies Analysis: A Deep Domain Adaptation Approach
• CeyMo: See More on Roads -- A Novel Benchmark Dataset for Road Marking Detection
• Memristive Stochastic Computing for Deep Learning Parameter Optimization
• Deep Learning with Functional Inputs
• Transfer learning for radio galaxy classification
• Fine-Grained Trajectory-based Travel Time Estimation for Multi-city Scenarios Based on Deep Meta-Learning
• RedSync : Reducing Synchronization Traffic for Distributed Deep Learning
• A Convolutional Neural Network Approach to the Classification of Engineering Models
• Application of Decision Rules for Handling Class Imbalance in Semantic Segmentation
• Hierarchical Disentanglement-Alignment Network for Robust SAR Vehicle Recognition
• Automated Defect Recognition of Castings defects using Neural Networks
• Adversarial Generation of Training Examples: Applications to Moving Vehicle License Plate Recognition
• Quantum Architecture Search via Deep Reinforcement Learning
• Hybrid quantum physics-informed neural networks for simulating computational fluid dynamics in complex shapes
• Bayesian neural networks via MCMC: a Python-based tutorial
• A Review of 1D Convolutional Neural Networks toward Unknown Substance Identification in Portable Raman Spectrometer
• Transfer learning-based method for automated ewaste recycling in smart cities
• Mitigating Unfairness via Evolutionary Multi-objective Ensemble Learning
• MaxiMask and MaxiTrack: two new tools for identifying contaminants in astronomical images using convolutional neural networks
• Investigating and Mitigating Failure Modes in Physics-informed Neural Networks (PINNs)
• Deep Gamblers: Learning to Abstain with Portfolio Theory
• Reconfigurable Intelligent Surface Assisted Device-to-Device Communications
• Extracting gamma-ray information from images with convolutional neural network methods on simulated Cherenkov Telescope Array data
• Declarative Recursive Computation on an RDBMS, or, Why You Should Use a Database For Distributed Machine Learning
• On Physics-Informed Neural Networks for Quantum Computers
• Deep learning in radiology: an overview of the concepts and a survey of the state of the art
• Accelerating multimodal gravitational waveforms from precessing compact binaries with artificial neural networks
• Applications of Scientific Machine Learning for the Analysis of Functionally Graded Porous Beams
• The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Models
• CASI: A Convolutional Neural Network Approach for Shell Identification
• CHS-Net: A Deep learning approach for hierarchical segmentation of COVID-19 infected CT images
• Lower error bounds for the stochastic gradient descent optimization algorithm: Sharp convergence rates for slowly and fast decaying learning rates
• QGOpt: Riemannian optimization for quantum technologies
• FeederGAN: Synthetic Feeder Generation via Deep Graph Adversarial Nets
• Thermal experiments for fractured rock characterization: theoretical analysis and inverse modeling
• Generalizable control for multiparameter quantum metrology
• Supervised learning of time-independent Hamiltonians for gate design
• Physical deep learning based on optimal control of dynamical systems
• Artificial Intelligence-Based Analytics for Impacts of COVID-19 and Online Learning on College Students' Mental Health
• Automated Architecture Design for Deep Neural Networks
• Slow and Stale Gradients Can Win the Race: Error-Runtime Trade-offs in Distributed SGD
• A proof of convergence for gradient descent in the training of artificial neural networks for constant target functions
• Plug-and-Play Quantum Adaptive Denoiser for Deconvolving Poisson Noisy Images
• Privacy Preservation in Federated Learning: An insightful survey from the GDPR Perspective
• Deep Neural Networks for Marine Debris Detection in Sonar Images
• A Performance Comparison of Loss Functions for Deep Face Recognition
• The Echo Index and multistability in input-driven recurrent neural networks
• A Novel Hybrid Machine Learning Model for Rapid Assessment of Wave and Storm Surge Responses Over an Extended Coastal Region
• A comparative study of 2D image segmentation algorithms for traumatic brain lesions using CT data from the ProTECTIII multicenter clinical trial
• Application of Machine Learning in Seismic Fragility Assessment of Bridges with SMA-Restrained Rocking Columns
• Hierarchically modelling Kepler dwarfs and subgiants to improve inference of stellar properties with asteroseismology
• Hybrid Quantum-Classical Graph Convolutional Network
• Deep Learning Based Regression and Multi-class Models for Acute Oral Toxicity Prediction with Automatic Chemical Feature Extraction
• A Comprehensive Survey of Multilingual Neural Machine Translation
• Phaseless Microwave Imaging Of Dielectric Cylinders: An Artificial Neural Networks-Based Approach
• Stochastic learning control of inhomogeneous quantum ensembles
• Packing Sparse Convolutional Neural Networks for Efficient Systolic Array Implementations: Column Combining Under Joint Optimization
• Neural Parametric Fokker-Planck Equations
• RAIN: RegulArization on Input and Network for Black-Box Domain Adaptation
• Soft Actor-Critic Deep Reinforcement Learning for Fault Tolerant Flight Control
• Closing the Gap Between SGP4 and High-Precision Propagation via Differentiable Programming
• A Multi-modal and Multi-task Learning Method for Action Unit and Expression Recognition
• CLIP: Cheap Lipschitz Training of Neural Networks
• Mass Estimation of Galaxy Clusters with Deep Learning II: CMB Cluster Lensing
• Enabling Large-Scale and High-Precision Fluid Simulations on Near-Term Quantum Computers
• A Combined Data-driven and Physics-driven Method for Steady Heat Conduction Prediction using Deep Convolutional Neural Networks
• Fast Stochastic Variance Reduced Gradient Method with Momentum Acceleration for Machine Learning
• Gradient Descent: The Ultimate Optimizer
• Generalised gravitational burst generation with Generative Adversarial Networks
• MAG-Net: Multi-task attention guided network for brain tumor segmentation and classification
• Variational Quantum Optimization of Nonlocality in Noisy Quantum Networks
• A Deep Double Ritz Method (D${}^2$RM) for solving Partial Differential Equations using Neural Networks
• A machine learning framework for data driven acceleration of computations of differential equations
• EvoJAX: Hardware-Accelerated Neuroevolution
• Efficient Parallel Translating Embedding For Knowledge Graphs
• DPIS: An Enhanced Mechanism for Differentially Private SGD with Importance Sampling
• Deep Robust Kalman Filter
• AIive: Interactive Visualization and Sonification of Neural Networks in Virtual Reality
• MXNET-MPI: Embedding MPI parallelism in Parameter Server Task Model for scaling Deep Learning
• Machine learning for metal additive manufacturing: Predicting temperature and melt pool fluid dynamics using physics-informed neural networks
• Modelling of physical systems with a Hopf bifurcation using mechanistic models and machine learning
• Spacetime-Efficient Low-Depth Quantum State Preparation with Applications
• Deep-CEE I: Fishing for Galaxy Clusters with Deep Neural Nets
• Where Did My Optimum Go?: An Empirical Analysis of Gradient Descent Optimization in Policy Gradient Methods
• Avoiding local minima in Variational Quantum Algorithms with Neural Networks
• Deep learning for photoacoustic imaging: a survey
• Reverse Derivative Ascent: A Categorical Approach to Learning Boolean Circuits
• Variational Preparation of the Sachdev-Ye-Kitaev Thermofield Double
• Multi-limb Split Learning for Tumor Classification on Vertically Distributed Data
• Revisiting Tensor Basis Neural Networks for Reynolds stress modeling: application to plane channel and square duct flows
• Deep Learning Assessment of galaxy morphology in S-PLUS DataRelease 1
• Estimation of discrete choice models with hybrid stochastic adaptive batch size algorithms
• Influence Functions in Deep Learning Are Fragile
• Anatomically aware dual-hop learning for pulmonary embolism detection in CT pulmonary angiograms
• Double Neural Counterfactual Regret Minimization
• Convolutional Recurrent Neural Networks for Glucose Prediction
• Standardized Non-Intrusive Reduced Order Modeling Using Different Regression Models With Application to Complex Flow Problems
• Learning Rate Optimization for Federated Learning Exploiting Over-the-air Computation
• LaProp: Separating Momentum and Adaptivity in Adam
• On the existence of global minima and convergence analyses for gradient descent methods in the training of deep neural networks
• Demystifying Learning Rate Policies for High Accuracy Training of Deep Neural Networks
• Human Attention in Fine-grained Classification
• Privacy-Preserving Federated Learning for UAV-Enabled Networks: Learning-Based Joint Scheduling and Resource Management
• Distributed Learning of Deep Neural Networks using Independent Subnet Training
• Learning Fair Scoring Functions: Bipartite Ranking under ROC-based Fairness Constraints
• Sinkformers: Transformers with Doubly Stochastic Attention
• A proof of convergence for stochastic gradient descent in the training of artificial neural networks with ReLU activation for constant target functions
• Machine Vision for Natural Gas Methane Emissions Detection Using an Infrared Camera
• Spectrum Sharing in Vehicular Networks Based on Multi-Agent Reinforcement Learning
• Effects of the Nonlinearity in Activation Functions on the Performance of Deep Learning Models
• A Machine Learning Framework for Stock Selection
• Adaptive Gradient Descent for Convex and Non-Convex Stochastic Optimization
• Deep Learning for Effective and Efficient Reduction of Large Adaptation Spaces in Self-Adaptive Systems
• Understanding and Improvement of Adversarial Training for Network Embedding from an Optimization Perspective
• RDP-GAN: A Rényi-Differential Privacy based Generative Adversarial Network
• DeepSI: Interactive Deep Learning for Semantic Interaction
• Optimizing Quantum Convolutional Neural Network Architectures for Arbitrary Data Dimension
• Predicting the transverse emittance of space charge dominated beams using the phase advance scan technique and a fully connected neural network
• J-PLUS: Support Vector Regression to Measure Stellar Parameters
• HUMAP: Hierarchical Uniform Manifold Approximation and Projection
• Crack Semantic Segmentation using the U-Net with Full Attention Strategy
• Parameter and density estimation from real-world traffic data: A kinetic compartmental approach
• Analyzing the Galactic pulsar distribution with machine learning
• Adversarial Resilience Learning - Towards Systemic Vulnerability Analysis for Large and Complex Systems
• High-dimension Tensor Completion via Gradient-based Optimization Under Tensor-train Format
• Deep Learning in Wide-field Surveys: Fast Analysis of Strong Lenses in Ground-based Cosmic Experiments
• Small quantum computers and large classical data sets
• Metaball-Imaging Discrete Element Lattice Boltzmann Method for fluid-particle system of complex morphologies with settling case study
• A Continuous Variable Born Machine
• The Importance of Being Interpretable: Toward An Understandable Machine Learning Encoder for Galaxy Cluster Cosmology
• A Physical Model for Microstructural Characterization and Segmentation of 3D Tomography Data
• Language Independent Single Document Image Super-Resolution using CNN for improved recognition
• Quantitative phase retrieval for Zernike phase-contrast microscopy
• Quantum support vector data description for anomaly detection
• Momentum Residual Neural Networks
• Forensic Analysis and Localization of Multiply Compressed MP3 Audio Using Transformers
• Identification of high order closure terms from fully kinetic simulations using machine learning
• Mutually exciting point process graphs for modelling dynamic networks
• Improving Gradient Estimation in Evolutionary Strategies With Past Descent Directions
• Inverse design of functional photonic patches by adjoint optimization coupled to the generalized Mie theory
• Dual Policy Learning for Aggregation Optimization in Graph Neural Network-based Recommender Systems
• Empirical modeling and hybrid machine learning framework for nucleate pool boiling on microchannel structured surfaces
• Scalable Imaginary Time Evolution with Neural Network Quantum States
• Automatic differentiation and the optimization of differential equation models in biology
• Machine Learning practices and infrastructures
• Apple Leaf Disease Identification through Region-of-Interest-Aware Deep Convolutional Neural Network
• Model-Informed Generative Adversarial Network (MI-GAN) for Learning Optimal Power Flow
• The Deep Arbitrary Polynomial Chaos Neural Network or how Deep Artificial Neural Networks could benefit from Data-Driven Homogeneous Chaos Theory
• Large-Scale Deep Learning Optimizations: A Comprehensive Survey
• Kalman meets Bellman: Improving Policy Evaluation through Value Tracking
• An Introduction to Neural Architecture Search for Convolutional Networks
• AOL: Adaptive Online Learning for Human Trajectory Prediction in Dynamic Video Scenes
• OptTyper: Probabilistic Type Inference by Optimising Logical and Natural Constraints
• Variable Star Classification Using Multi-View Metric Learning
• Automatic Defect Detection in Sewer Network Using Deep Learning Based Object Detector
• Distributed Stochastic Algorithms for High-rate Streaming Principal Component Analysis
• Multi-variable integration with a variational quantum circuit
• Modularity in Deep Learning: A Survey
• A Continuous Convolutional Trainable Filter for Modelling Unstructured Data
• CHOPT : Automated Hyperparameter Optimization Framework for Cloud-Based Machine Learning Platforms
• PRETZEL: Opening the Black Box of Machine Learning Prediction Serving Systems
• Category Theory in Machine Learning
• Deep 3D Convolutional Neural Network for Automated Lung Cancer Diagnosis
• A fusion method for multi-valued data
• Symbolic Network: Generalized Neural Policies for Relational MDPs
• Explainable Rumor Detection using Inter and Intra-feature Attention Networks
• A Novel Framework for Neural Architecture Search in the Hill Climbing Domain
• Decision-based Universal Adversarial Attack
• nuts-flow/ml: data pre-processing for deep learning
• TorsionNet: A Reinforcement Learning Approach to Sequential Conformer Search
• A Survey of Latent Factor Models in Recommender Systems
• Asymptotic Analysis via Stochastic Differential Equations of Gradient Descent Algorithms in Statistical and Computational Paradigms
• Cross-Domain Adaptation for Animal Pose Estimation
• On The State of Data In Computer Vision: Human Annotations Remain Indispensable for Developing Deep Learning Models
• A Qualitative Study of the Dynamic Behavior for Adaptive Gradient Algorithms
• Neuron Campaign for Initialization Guided by Information Bottleneck Theory
• The Wang-Landau Algorithm as Stochastic Optimization and Its Acceleration
• Bringing Linearly Transformed Cosines to Anisotropic GGX
• On the Generalised Langevin Equation for Simulated Annealing
• An Unbiased Estimator of the Full-sky CMB Angular Power Spectrum at Large Scales using Neural Networks
• PECCO: A Profit and Cost-oriented Computation Offloading Scheme in Edge-Cloud Environment with Improved Moth-flame Optimisation
• Combination of Domain Knowledge and Deep Learning for Sentiment Analysis of Short and Informal Messages on Social Media
• Robust Learning Rate Selection for Stochastic Optimization via Splitting Diagnostic
• Privacy-preserving Federated Bayesian Learning of a Generative Model for Imbalanced Classification of Clinical Data
• Performance Comparison of Numerical Optimization Algorithms for RSS-TOA-Based Target Localization
• Fiedler Regularization: Learning Neural Networks with Graph Sparsity
• Finding the right scale of a network: Efficient identification of causal emergence through spectral clustering
• A Novel Stochastic Stratified Average Gradient Method: Convergence Rate and Its Complexity
• A Caputo fractional derivative-based algorithm for optimization
• Hidden Markov Chains, Entropic Forward-Backward, and Part-Of-Speech Tagging
• Evolutionary Multi-objective Optimisation in Neurotrajectory Prediction
• Generation of True Quantum Random Numbers with On-Demand Probability Distributions via Single-Photon Quantum Walks
• Sampling-based probabilistic inference emerges from learning in neural circuits with a cost on reliability
• Learning the Wireless V2I Channels Using Deep Neural Networks
• Accelerating Distributed ML Training via Selective Synchronization
• GraVAC: Adaptive Compression for Communication-Efficient Distributed DL Training
• Scavenger: A Cloud Service for Optimizing Cost and Performance of ML Training
• Sparse-View Spectral CT Reconstruction Using Deep Learning
• Real-time rapid leakage estimation for deep space habitats using exponentially-weighted adaptively-refined search
• A Local Block Coordinate Descent Algorithm for the Convolutional Sparse Coding Model
• Platoon trajectories generation: A unidirectional interconnected LSTM-based car following model
• EmbRace: Accelerating Sparse Communication for Distributed Training of NLP Neural Networks
• RTFN: A Robust Temporal Feature Network for Time Series Classification
• Efficient Sampling of Thermal Averages of Interacting Quantum Particle Systems with Random Batches
• The Role of Momentum Parameters in the Optimal Convergence of Adaptive Polyak's Heavy-ball Methods
• Predicting light curves of RR Lyrae variables using artificial neural network based interpolation of a grid of pulsation models
• AdaSGD: Bridging the gap between SGD and Adam
• Topology Optimization under Uncertainty using a Stochastic Gradient-based Approach
• Automated Copper Alloy Grain Size Evaluation Using a Deep-learning CNN
• LOSSGRAD: automatic learning rate in gradient descent
• Utilizing Explainable AI for Quantization and Pruning of Deep Neural Networks
• Certified Adversarial Defenses Meet Out-of-Distribution Corruptions: Benchmarking Robustness and Simple Baselines
• A multivariate water quality parameter prediction model using recurrent neural network
• Backdoor Attack and Defense for Deep Regression
• A Letter on Convergence of In-Parameter-Linear Nonlinear Neural Architectures with Gradient Learnings
• Sketch2code: Generating a website from a paper mockup
• Estimating initial conditions for dynamical systems with incomplete information
• Scalable Balanced Training of Conditional Generative Adversarial Neural Networks on Image Data
• Bifidelity data-assisted neural networks in nonintrusive reduced-order modeling
• Numerically Solving Parametric Families of High-Dimensional Kolmogorov Partial Differential Equations via Deep Learning
• Characterising lognormal fractional-Brownian-motion density fields with a Convolutional Neural Network
• Gradient descent with momentum --- to accelerate or to super-accelerate?
• Characterizing Deep Learning Training Workloads on Alibaba-PAI
• BigSurvSGD: Big Survival Data Analysis via Stochastic Gradient Descent
• Capacity Control of ReLU Neural Networks by Basis-path Norm
• PyTorchDIA: A flexible, GPU-accelerated numerical approach to Difference Image Analysis
• Resonant Scanning Design and Control for Fast Spatial Sampling
• Deep neural network for solving differential equations motivated by Legendre-Galerkin approximation
• Trust Region Value Optimization using Kalman Filtering
• Layerwise Optimization by Gradient Decomposition for Continual Learning
• Sequential Keystroke Behavioral Biometrics for Mobile User Identification via Multi-view Deep Learning
• A comparative study of source-finding techniques in HI emission line cubes using SoFiA, MTObjects, and supervised deep learning
• Optimal Potential Shaping on SE(3) via Neural ODEs on Lie Groups
• Multi-Loss Sub-Ensembles for Accurate Classification with Uncertainty Estimation
• Online Knowledge Distillation via Multi-branch Diversity Enhancement
• MixML: A Unified Analysis of Weakly Consistent Parallel Learning
• Neuro-Symbolic Execution: The Feasibility of an Inductive Approach to Symbolic Execution
• Learning the Universe: Learning to Optimize Cosmic Initial Conditions with Non-Differentiable Structure Formation Models
• Supervised Deep Neural Networks (DNNs) for Pricing/Calibration of Vanilla/Exotic Options Under Various Different Processes
• Deep learning for pedestrians: backpropagation in CNNs
• Accelerating Backward Aggregation in GCN Training with Execution Path Preparing on GPUs
• GrCAN: Gradient Boost Convolutional Autoencoder with Neural Decision Forest
• A multi-path 2.5 dimensional convolutional neural network system for segmenting stroke lesions in brain MRI images
• Evolution-based Fine-tuning of CNNs for Prostate Cancer Detection
• Improve Single-Point Zeroth-Order Optimization Using High-Pass and Low-Pass Filters
• End-To-End Audiovisual Feature Fusion for Active Speaker Detection
• Learning-based Prediction and Uplink Retransmission for Wireless Virtual Reality (VR) Network
• A High-Performance Adaptive Quantization Approach for Edge CNN Applications
• Temperature Estimation in Induction Motors using Machine Learning
• Efficient labeling of solar flux evolution videos by a deep learning model
• Tuned Inception V3 for Recognizing States of Cooking Ingredients
• Effective Multi-Stage Training Model For Edge Computing Devices In Intrusion Detection
• Energy reconstruction in a liquid argon calorimeter cell using convolutional neural networks
• GmFace: A Mathematical Model for Face Image Representation Using Multi-Gaussian
• Machine-learning Based Extraction of the Short-Range Part of the Interaction in Non-contact Atomic Force Microscopy
• EfficientHRNet: Efficient Scaling for Lightweight High-Resolution Multi-Person Pose Estimation
• Transfer Learning in Multi-Agent Reinforcement Learning with Double Q-Networks for Distributed Resource Sharing in V2X Communication
• Estimation of one-dimensional discrete-time quantum walk parameters by using machine learning algorithms
• A Comprehensive Study of Data Augmentation Strategies for Prostate Cancer Detection in Diffusion-weighted MRI using Convolutional Neural Networks
• Investigating performance of neural networks and gradient boosting models approximating microscopic traffic simulations in traffic optimization tasks
• Repaint: Improving the Generalization of Down-Stream Visual Tasks by Generating Multiple Instances of Training Examples
• Bayesian Optimization Algorithms for Accelerator Physics
• On Mini-Batch Training with Varying Length Time Series
• Influence-Augmented Online Planning for Complex Environments
• Boosting Naturalness of Language in Task-oriented Dialogues via Adversarial Training
• A Deep Learning Application for Psoriasis Detection
• MMCoVaR: Multimodal COVID-19 Vaccine Focused Data Repository for Fake News Detection and a Baseline Architecture for Classification
• Adaptive Physics-Informed Neural Networks for Markov-Chain Monte Carlo
• Consumer Behaviour in Retail: Next Logical Purchase using Deep Neural Network
• Deep Learning for Robust Motion Segmentation with Non-Static Cameras
• Convolutional Neural Network Models and Interpretability for the Anisotropic Reynolds Stress Tensor in Turbulent One-dimensional Flows
• On Coresets for Regularized Loss Minimization
• Effectiveness of Optimization Algorithms in Deep Image Classification
• Heuristic Rank Selection with Progressively Searching Tensor Ring Network
• Deep Neural Network Based Resource Allocation for V2X Communications
• PolyFold: an interactive visual simulator for distance-based protein folding
• Demystifying BERT: Implications for Accelerator Design
• Crowdsourced-based Deep Convolutional Networks for Urban Flood Depth Mapping
• Hydra: A Peer to Peer Distributed Training & Data Collection Framework
• Personalized Dynamic Treatment Regimes in Continuous Time: A Bayesian Approach for Optimizing Clinical Decisions with Timing
• Citadel: Protecting Data Privacy and Model Confidentiality for Collaborative Learning with SGX
• State-of-charge Estimation of a Li-ion Battery using Deep Learning and Stochastic Optimization
• The Implicit Bias for Adaptive Optimization Algorithms on Homogeneous Neural Networks
• Advances in the training, pruning and enforcement of shape constraints of Morphological Neural Networks using Tropical Algebra
• Parameter Prediction for Unseen Deep Architectures
• Evaluating Deep Learning in SystemML using Layer-wise Adaptive Rate Scaling(LARS) Optimizer
• Improving Online Forums Summarization via Hierarchical Unified Deep Neural Network
• Learning compact generalizable neural representations supporting perceptual grouping
• Block stochastic gradient descent for large-scale tomographic reconstruction in a parallel network
• Plateau Phenomenon in Gradient Descent Training of ReLU networks: Explanation, Quantification and Avoidance
• Convergence rates for gradient descent in the training of overparameterized artificial neural networks with piecewise affine activation
• Learning-based Prediction, Rendering and Transmission for Interactive Virtual Reality in RIS-Assisted Terahertz Networks
• Lattice models for protein organization throughout thylakoid membrane stacks
• Learning to Coordinate in Multi-Agent Systems: A Coordinated Actor-Critic Algorithm and Finite-Time Guarantees
• How Many Factors Influence Minima in SGD?
• Inverse Halftoning Through Structure-Aware Deep Convolutional Neural Networks
• Weighted Empirical Risk Minimization: Sample Selection Bias Correction based on Importance Sampling
• COMET: A Novel Memory-Efficient Deep Learning Training Framework by Using Error-Bounded Lossy Compression
• Bi-fidelity Stochastic Gradient Descent for Structural Optimization under Uncertainty
• Theoretical research without projects
• Activated Gradients for Deep Neural Networks
• Stochastic Gradient Langevin Dynamics Algorithms with Adaptive Drifts
• SMG: A Shuffling Gradient-Based Method with Momentum
• Compressing Heavy-Tailed Weight Matrices for Non-Vacuous Generalization Bounds
• OmniPrint: A Configurable Printed Character Synthesizer
• When and how epochwise double descent happens
• On Faster Convergence of Scaled Sign Gradient Descent
• Multi-agent Cooperative Games Using Belief Map Assisted Training
• Physical Parameters of Stars in NGC 6397 Using ANN-Based Interpolation and Full Spectrum Fitting
• Few-shot transfer of tool-use skills using human demonstrations with proximity and tactile sensing
• Simmering: Sufficient is better than optimal for training neural networks
• Adaptive Elastic Training for Sparse Deep Learning on Heterogeneous Multi-GPU Servers
• Fast gradient-free activation maximization for neurons in spiking neural networks
• Measure Transport with Kernel Stein Discrepancy
• MG-GCN: Fast and Effective Learning with Mix-grained Aggregators for Training Large Graph Convolutional Networks
• ZORB: A Derivative-Free Backpropagation Algorithm for Neural Networks
• Deep-Learning based Multiuser Detection for NOMA
• Temporal Autoencoder with U-Net Style Skip-Connections for Frame Prediction
• MUSCLE: Strengthening Semi-Supervised Learning Via Concurrent Unsupervised Learning Using Mutual Information Maximization
• Using the Naive Bayes as a discriminative classifier
• Predicting Poverty Level from Satellite Imagery using Deep Neural Networks
• Neuron with Steady Response Leads to Better Generalization
• Algorithmic Complexities in Backpropagation and Tropical Neural Networks
• A proof of convergence for the gradient descent optimization method with random initializations in the training of neural networks with ReLU activation for piecewise linear target functions
• Deep Neural Networks for Active Wave Breaking Classification
• IWA: Integrated Gradient based White-box Attacks for Fooling Deep Neural Networks
• Inverse design couplers for the excitation of odd plasmonic pairs in thin semiconducting films
• On the Importance of 3D Surface Information for Remote Sensing Classification Tasks
• Low Dimensional Landscape Hypothesis is True: DNNs can be Trained in Tiny Subspaces
• Asymmetric Heavy Tails and Implicit Bias in Gaussian Noise Injections
• Heterogeneous CPU+GPU Stochastic Gradient Descent Algorithms
• Reliability and Performance Assessment of Federated Learning on Clinical Benchmark Data
• Gravilon: Applications of a New Gradient Descent Method to Machine Learning
• Detecting Problem Statements in Peer Assessments
• A Novel RL-assisted Deep Learning Framework for Task-informative Signals Selection and Classification for Spontaneous BCIs
• Proximal bundle algorithms for nonsmooth convex optimization via fast gradient smooth methods
• Joint Parameter-and-Bandwidth Allocation for Improving the Efficiency of Partitioned Edge Learning
• Learn distributed GAN with Temporary Discriminators
• On the Trend-corrected Variant of Adaptive Stochastic Optimization Methods
• Learning scale-variant features for robust iris authentication with deep learning based ensemble framework
• A Survey on Large-scale Machine Learning
• Deep localization of protein structures in fluorescence microscopy images
• Topological Navigation Graph Framework
• A Gradient Free Neural Network Framework Based on Universal Approximation Theorem
• Population-based Gradient Descent Weight Learning for Graph Coloring Problems
• Knowledge-guided Unsupervised Rhetorical Parsing for Text Summarization
• Design of Capacity-Approaching Low-Density Parity-Check Codes using Recurrent Neural Networks
• Variational Optimization on Lie Groups, with Examples of Leading (Generalized) Eigenvalue Problems
• Optimizing Deep Neural Networks with Multiple Search Neuroevolution
• Flexible Operator Embeddings via Deep Learning
• Automated Search for Configurations of Deep Neural Network Architectures
• A Pragmatic AI Approach to Creating Artistic Visual Variations by Neural Style Transfer
• A Unified Framework of Deep Neural Networks by Capsules
• Quaternion Collaborative Filtering for Recommendation
• A Novel Deep Neural Network Based Approach for Sparse Code Multiple Access
• Incremental Concept Learning via Online Generative Memory Recall
• SA-GD: Improved Gradient Descent Learning Strategy with Simulated Annealing
• BCNet: A Deep Convolutional Neural Network for Breast Cancer Grading
• Neighbor-view Enhanced Model for Vision and Language Navigation
• HydaLearn: Highly Dynamic Task Weighting for Multi-task Learning with Auxiliary Tasks
• Automated Estimation of Construction Equipment Emission using Inertial Sensors and Machine Learning Models
• Graph Drawing by Stochastic Gradient Descent
• GRAFFL: Gradient-free Federated Learning of a Bayesian Generative Model
• Improving Offline Contextual Bandits with Distributional Robustness
• Generating a Machine-learned Equation of State for Fluid Properties
• One Backward from Ten Forward, Subsampling for Large-Scale Deep Learning
• An Artificial Intelligence System for Combined Fruit Detection and Georeferencing, Using RTK-Based Perspective Projection in Drone Imagery
• Efficient excitation-transfer across fully connected networks via local-energy optimization
• Trivial or impossible -- dichotomous data difficulty masks model differences (on ImageNet and beyond)
• RNN-based Online Learning: An Efficient First-Order Optimization Algorithm with a Convergence Guarantee
• DEAM: Adaptive Momentum with Discriminative Weight for Stochastic Optimization
• End-to-end analysis using image classification
• Error mitigation of entangled states using brainbox quantum autoencoders
• Topology Optimization under Microscale Uncertainty using Stochastic Gradients
• Landscape Correspondence of Empirical and Population Risks in the Eigendecomposition Problem
• Integration of TensorFlow based Acoustic Model with Kaldi WFST Decoder
• On Higher-order Moments in Adam
• Joint Design of Radar Waveform and Detector via End-to-end Learning with Waveform Constraints
• Evaluation of Neural Networks for Image Recognition Applications: Designing a 0-1 MILP Model of a CNN to create adversarials
• DeepLocalize: Fault Localization for Deep Neural Networks
• On Compression Principle and Bayesian Optimization for Neural Networks
• GraSSNet: Graph Soft Sensing Neural Networks
• Understanding the Disharmony between Weight Normalization Family and Weight Decay: $\epsilon -$shifted $L_2$ Regularizer
• Experimental Catalytic Amplification of Asymmetry
• Finite-Time Consensus Learning for Decentralized Optimization with Nonlinear Gossiping
• A Probabilistically Motivated Learning Rate Adaptation for Stochastic Optimization
• Position reconstruction and surface background model for the PandaX-4T detector
• Fast Approximation of Optimal Perturbed Long-Duration Impulsive Transfers via Deep Neural Networks
• Fast Large-Scale Discrete Optimization Based on Principal Coordinate Descent
• Pushing the boundaries of parallel Deep Learning -- A practical approach
• Photozilla: A Large-Scale Photography Dataset and Visual Embedding for 20 Photography Styles
• Sampling Kaczmarz Motzkin Method for Linear Feasibility Problems: Generalization & Acceleration
• Comparison of Neuronal Attention Models
• Melody Harmonization Using Orderless NADE, Chord Balancing, and Blocked Gibbs Sampling
• A Variant of Gradient Descent Algorithm Based on Gradient Averaging
• Progressive VAE Training on Highly Sparse and Imbalanced Data
• Second-order Information in First-order Optimization Methods
• Selectivity correction with online machine learning
• Adaptive Online Learning with Momentum for Contingency-based Voltage Stability Assessment
• Neural Networks as Functional Classifiers
• SSGD: A safe and efficient method of gradient descent
• DualNet: Locate Then Detect Effective Payload with Deep Attention Network
• Robust and Active Learning for Deep Neural Network Regression
• EMA: Auditing Data Removal from Trained Models
• Minorization-Maximization-based Steepest Ascent for Large-scale Survival Analysis with Time-Varying Effects: Application to the National Kidney Transplant Dataset
• Learn to Compress CSI and Allocate Resources in Vehicular Networks
• A Capsule-unified Framework of Deep Neural Networks for Graphical Programming
• Performance of artificial neural networks in an inverse problem of laser beam diagnostics
• Learning Compact Target-Oriented Feature Representations for Visual Tracking
• On the instability of embeddings for recommender systems: the case of Matrix Factorization
• On Imitation Learning of Linear Control Policies: Enforcing Stability and Robustness Constraints via LMI Conditions
• Learning DNN networks using un-rectifying ReLU with compressed sensing application
• A Convolutional Neural Network-based Approach to Field Reconstruction
• Spatio-Temporal Neural Network for Fitting and Forecasting COVID-19
• Neuromodulated Learning in Deep Neural Networks
• Carathéodory Sampling for Stochastic Gradient Descent
• Enhancing Biomedical Multi-modal Representation Learning with Multi-scale Pre-training and Perturbed Report Discrimination
• The problem of perfect predictors in statistical spike train models
• Heavy-Tail Phenomenon in Decentralized SGD
• Temporal mixture ensemble models for intraday volume forecasting in cryptocurrency exchange markets
• CSM-NN: Current Source Model Based Logic Circuit Simulation -- A Neural Network Approach
• Differentially Private ADMM Algorithms for Machine Learning
• Can we learn gradients by Hamiltonian Neural Networks?
• Learning formation energy of inorganic compounds using matrix variate deep Gaussian process
• GaussED: A Probabilistic Programming Language for Sequential Experimental Design
• A recurrent multi-scale approach to RBG-D Object Recognition
• Comparing Sample-wise Learnability Across Deep Neural Network Models
• Neural Architecture Search via Bregman Iterations
• Improving Adversarial Robustness for Free with Snapshot Ensemble
• Implementation of Parallel Simplified Swarm Optimization in CUDA
• Resource Allocation Based on Deep Neural Networks for Cognitive Radio Networks
• Towards a NISQ Algorithm to Simulate Hermitian Matrix Exponentiation
• AI-Powered Low-Order Focal Plane Wavefront Sensing in Infrared
• Fast Evaluation of Low-Thrust Transfers via Deep Neural Networks
• OLR-WA: Online Weighted Average Linear Regression in Multivariate Data Streams
• Deep Learning based Estimation of Weaving Target Maneuvers
• Neograd: Near-Ideal Gradient Descent
• On Deep Learning for Radio Resource Management in A Non-stationary Radio Environment
• A Skip-connected Multi-column Network for Isolated Handwritten Bangla Character and Digit recognition
• Iterative Domain Optimization
• Solving Zero-Sum Games through Alternating Projections
• Aion: Better Late than Never in Event-Time Streams
• Weighted Aggregating Stochastic Gradient Descent for Parallel Deep Learning
• Predictive Process Model Monitoring using Recurrent Neural Networks
• Deep Learning for Constrained Utility Maximisation
• Adaptive Hierarchical Hyper-gradient Descent
• Survey: Machine Learning in Production Rendering
• Training Neural Networks with an algorithm for piecewise linear functions
• Internal Calibration Process Using Chirp Pulses with Application of the Adam Learning Algorithm
• Link Prediction for Temporally Consistent Networks
• Explicit Gradient Learning
• Interaction Networks: Using a Reinforcement Learner to train other Machine Learning algorithms
• On the Equivalence of Neural and Production Networks
• Machine Biometrics -- Towards Identifying Machines in a Smart City Environment
• A Constructive, Type-Theoretic Approach to Regression via Global Optimisation
• Introducing the Hidden Neural Markov Chain framework
• Training Deep Neural Networks via Branch-and-Bound
• Absolute 3D Pose Estimation and Length Measurement of Severely Deformed Fish from Monocular Videos in Longline Fishing
• Multi-Tensor Network Representation for High-Order Tensor Completion
• Length Learning for Planar Euclidean Curves
• TAG: Task-based Accumulated Gradients for Lifelong learning
• Detection of Abnormal Behavior with Self-Supervised Gaze Estimation
• Network Learning with Local Propagation
• Modulating Regularization Frequency for Efficient Compression-Aware Model Training
• The impact of the additional features on the performance of regression analysis: a case study on regression analysis of music signal
• MGA: Momentum Gradient Attack on Network
• Tensor-Based Backpropagation in Neural Networks with Non-Sequential Input
• Optimizing Convergence for Iterative Learning of ARIMA for Stationary Time Series
• An Explainable Probabilistic Classifier for Categorical Data Inspired to Quantum Physics
• Expression Recognition Analysis in the Wild
• Why to "grow" and "harvest" deep learning models?
• Dynamical prediction of two meteorological factors using the deep neural network and the long short term memory $(1)$
• On segmentation of pectoralis muscle in digital mammograms by means of deep learning
• Scared into Action: How Partisanship and Fear are Associated with Reactions to Public Health Directives
• Advanced Astroinformatics for Variable Star Classification
• Memory-Efficient Factorization Machines via Binarizing both Data and Model Coefficients
• Recoding latent sentence representations -- Dynamic gradient-based activation modification in RNNs
• Sparse Network Inversion for Key Instance Detection in Multiple Instance Learning
• Learning Shape Features and Abstractions in 3D Convolutional Neural Networks for Detecting Alzheimer's Disease
• Adaptive Gradient Descent Methods for Computing Implied Volatility
• 2nd-order Updates with 1st-order Complexity
• A survey on deep learning approaches for breast cancer diagnosis
• Patch-based Medical Image Segmentation using Matrix Product State Tensor Networks
• BGADAM: Boosting based Genetic-Evolutionary ADAM for Neural Network Optimization
• Parameters for the best convergence of an optimization algorithm On-The-Fly
• Hyperlink Regression via Bregman Divergence
• Learn to Allocate Resources in Vehicular Networks
• Proposition d'un modèle pour l'optimisation automatique de boucles dans le compilateur Tiramisu : cas d'optimisation de déroulage
• LEBANONUPRISING: a thorough study of Lebanese tweets
• Better Approximate Inference for Partial Likelihood Models with a Latent Structure
• Generalized Categorisation of Digital Pathology Whole Image Slides using Unsupervised Learning
• Neural Network Based Qubit Environment Characterization
• A language processing algorithm for predicting tactical solutions to an operational planning problem under uncertainty
• Vehicle Re-identification Based on Dual Distance Center Loss
• Accounting for data heterogeneity in integrative analysis and prediction methods: An application to Chronic Obstructive Pulmonary Disease
• Feedback Control for Online Training of Neural Networks
• Buildings Classification using Very High Resolution Satellite Imagery
• Region segmentation via deep learning and convex optimization
• End-to-end Learning of Waveform Generation and Detection for Radar Systems
• Multiple Learning for Regression in big data
• Microwave Tomography with phaseless data on the calcaneus by means of artificial neural networks
• MYSTIKO : : Cloud-Mediated, Private, Federated Gradient Descent
• Layer Decomposition Learning Based on Gaussian Convolution Model and Residual Deblurring for Inverse Halftoning
• On Generalization of Adaptive Methods for Over-parameterized Linear Regression
• Trust Region Method for Coupled Systems of PDE Solvers and Deep Neural Networks
• Music Signal Processing Using Vector Product Neural Networks
• Convergence Analysis of Gradient Descent Algorithms with Proportional Updates
• Simulation-based inference methods for partially observed Markov model via the R package is2
• Lily-like twist distribution in toroidal nematics
• Stochastic Probabilistic Programs
• To each route its own ETA: A generative modeling framework for ETA prediction
• AIRSENSE-TO-ACT: A Concept Paper for COVID-19 Countermeasures based on Artificial Intelligence algorithms and multi-sources Data Processing
• Algebra of L-banded Matrices