SoftAdapt: Techniques for Adaptive Loss Weighting of Neural Networks with Multi-Part Loss Functions

Abstract

Adaptive loss function formulation is an active area of research and has gained a great deal of popularity in recent years, following the success of deep learning. However, existing frameworks of adaptive loss functions often suffer from slow convergence and poor choice of weights for the loss components. Traditionally, the elements of a multi-part loss function are weighted equally or their weights are determined through heuristic approaches that yield near-optimal (or sub-optimal) results. To address this problem, we propose a family of methods, called SoftAdapt, that dynamically change function weights for multi-part loss functions based on live performance statistics of the component losses. SoftAdapt is mathematically intuitive, computationally efficient and straightforward to implement. In this paper, we present the mathematical formulation and pseudocode for SoftAdapt, along with results from applying our methods to image reconstruction (Sparse Autoencoders) and synthetic data generation (Introspective Variational Autoencoders).

8 pages with 2 pages of references. 6 Figures and 3 Tables

References in corpus (2)

• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Addressing the Loss-Metric Mismatch with Adaptive Loss Alignment

Cited by in corpus (9)

• Preoperative brain tumor imaging: models and software for segmentation and standardized reporting
• Physics-Guided Neural Networks for Intraventricular Vector Flow Mapping
• Physics-informed neural networks modeling for systems with moving immersed boundaries: application to an unsteady flow past a plunging foil
• Revisiting Tensor Basis Neural Networks for Reynolds stress modeling: application to plane channel and square duct flows
• Variational PINNs with tree-based integration and boundary element data in the modeling of multi-phase architected materials
• Sequential learning based PINNs to overcome temporal domain complexities in unsteady flow past flapping wings
• A Variational Auto-Encoder for Reservoir Monitoring
• Transforming physics-informed machine learning to convex optimization
• DeepGD: A Deep Learning Framework for Graph Drawing Using GNN