AutoLoss-Zero: Searching Loss Functions from Scratch for Generic Tasks
arXiv:2103.14026
Abstract
Significant progress has been achieved in automating the design of various components in deep networks. However, the automatic design of loss functions for generic tasks with various evaluation metrics remains under-investigated. Previous works on handcrafting loss functions heavily rely on human expertise, which limits their extendibility. Meanwhile, existing efforts on searching loss functions mainly focus on specific tasks and particular metrics, with task-specific heuristics. Whether such works can be extended to generic tasks is not verified and questionable. In this paper, we propose AutoLoss-Zero, the first general framework for searching loss functions from scratch for generic tasks. Specifically, we design an elementary search space composed only of primitive mathematical operators to accommodate the heterogeneous tasks and evaluation metrics. A variant of the evolutionary algorithm is employed to discover loss functions in the elementary search space. A loss-rejection protocol and a gradient-equivalence-check strategy are developed so as to improve the search efficiency, which are applicable to generic tasks. Extensive experiments on various computer vision tasks demonstrate that our searched loss functions are on par with or superior to existing loss functions, which generalize well to different datasets and networks. Code shall be released.
References in corpus (19)
- Adam: A Method for Stochastic Optimization
- AutoML: A Survey of the State-of-the-Art
- DARTS: Differentiable Architecture Search
- Sequence Level Training with Recurrent Neural Networks
- MMDetection: Open MMLab Detection Toolbox and Benchmark
- DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs
- An Actor-Critic Algorithm for Sequence Prediction
- A Comprehensive Survey of Neural Architecture Search: Challenges and Solutions
- Bridging Category-level and Instance-level Semantic Image Segmentation
- Cascade R-CNN: High Quality Object Detection and Instance Segmentation
- AP-Loss for Accurate One-Stage Object Detection
- Enhancing Geometric Factors in Model Learning and Inference for Object Detection and Instance Segmentation
- Training Deep Neural Networks via Direct Loss Minimization
- Minimum Risk Training for Neural Machine Translation
- The Lovász-Softmax loss: A tractable surrogate for the optimization of the intersection-over-union measure in neural networks
- Loss Function Search for Face Recognition
- FBNetV3: Joint Architecture-Recipe Search using Predictor Pretraining
- A Study of Reinforcement Learning for Neural Machine Translation
- Loss Function Discovery for Object Detection via Convergence-Simulation Driven Search