Learning from Mistakes based on Class Weighting with Application to Neural Architecture Search

Abstract

Learning from mistakes is an effective learning approach widely used in human learning, where a learner pays greater focus on mistakes to circumvent them in the future to improve the overall learning outcomes. In this work, we aim to investigate how effectively we can leverage this exceptional learning ability to improve machine learning models. We propose a simple and effective multi-level optimization framework called learning from mistakes using class weighting (LFM-CW), inspired by mistake-driven learning to train better machine learning models. In this formulation, the primary objective is to train a model to perform effectively on target tasks by using a re-weighting technique. We learn the class weights by minimizing the validation loss of the model and re-train the model with the synthetic data from the image generator weighted by class-wise performance and real data. We apply our LFM-CW framework with differential architecture search methods on image classification datasets such as CIFAR and ImageNet, where the results show that our proposed strategy achieves lower error rate than the baselines.

References in corpus (17)

• Conditional Generative Adversarial Nets
• Neural Architecture Search with Reinforcement Learning
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Self-Attention Generative Adversarial Networks
• DARTS: Differentiable Architecture Search
• Learning to Reweight Examples for Robust Deep Learning
• Efficient Neural Architecture Search via Parameter Sharing
• Neural Architecture Optimization
• Designing Neural Network Architectures using Reinforcement Learning
• PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search
• Understanding and Robustifying Differentiable Architecture Search
• DARTS+: Improved Differentiable Architecture Search with Early Stopping
• Progressive Differentiable Architecture Search: Bridging the Depth Gap between Search and Evaluation
• BayesNAS: A Bayesian Approach for Neural Architecture Search
• Just Train Twice: Improving Group Robustness without Training Group Information
• Robust Conditional Generative Adversarial Networks
• DropNAS: Grouped Operation Dropout for Differentiable Architecture Search