SoftAdapt: Techniques for Adaptive Loss Weighting of Neural Networks with Multi-Part Loss Functions

Abstract

Adaptive loss function formulation is an active area of research and has gained a great deal of popularity in recent years, following the success of deep learning. However, existing frameworks of adaptive loss functions often suffer from slow convergence and poor choice of weights for the loss components. Traditionally, the elements of a multi-part loss function are weighted equally or their weights are determined through heuristic approaches that yield near-optimal (or sub-optimal) results. To address this problem, we propose a family of methods, called SoftAdapt, that dynamically change function weights for multi-part loss functions based on live performance statistics of the component losses. SoftAdapt is mathematically intuitive, computationally efficient and straightforward to implement. In this paper, we present the mathematical formulation and pseudocode for SoftAdapt, along with results from applying our methods to image reconstruction (Sparse Autoencoders) and synthetic data generation (Introspective Variational Autoencoders).

arXiv comment8 pages with 2 pages of references. 6 Figures and 3 Tables

References in corpus (11)

• Auto-Encoding Variational Bayes
• An overview of gradient descent optimization algorithms
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• A Literature Survey of Benchmark Functions For Global Optimization Problems
• GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks
• IntroVAE: Introspective Variational Autoencoders for Photographic Image Synthesis
• k-Sparse Autoencoders
• Preconditioned Stochastic Gradient Descent
• On the Convergence of Stochastic Gradient Descent with Adaptive Stepsizes
• Addressing the Loss-Metric Mismatch with Adaptive Loss Alignment
• Multi-Objective Optimization for Self-Adjusting Weighted Gradient in Machine Learning Tasks

Cited by in corpus (13)

• Multi-Objective Loss Balancing for Physics-Informed Deep Learning
• Self-Adaptive Physics-Informed Neural Networks using a Soft Attention Mechanism
• Preoperative brain tumor imaging: models and software for segmentation and standardized reporting
• Semi Conditional Variational Auto-Encoder for Flow Reconstruction and Uncertainty Quantification from Limited Observations
• Physics-Guided Neural Networks for Intraventricular Vector Flow Mapping
• Physics-informed neural networks modeling for systems with moving immersed boundaries: application to an unsteady flow past a plunging foil
• Revisiting Tensor Basis Neural Networks for Reynolds stress modeling: application to plane channel and square duct flows
• Variational PINNs with tree-based integration and boundary element data in the modeling of multi-phase architected materials
• Sequential learning based PINNs to overcome temporal domain complexities in unsteady flow past flapping wings
• Going Beyond Classification Accuracy Metrics in Model Compression
• A Variational Auto-Encoder for Reservoir Monitoring
• Transforming physics-informed machine learning to convex optimization
• DeepGD: A Deep Learning Framework for Graph Drawing Using GNN