What Objective Does Self-paced Learning Indeed Optimize?

Abstract

Self-paced learning (SPL) is a recently raised methodology designed through simulating the learning principle of humans/animals. A variety of SPL realization schemes have been designed for different computer vision and pattern recognition tasks, and empirically substantiated to be effective in these applications. However, the investigation on its theoretical insight is still a blank. To this issue, this study attempts to provide some new theoretical understanding under the SPL scheme. Specifically, we prove that the solving strategy on SPL accords with a majorization minimization algorithm implemented on a latent objective function. Furthermore, we find that the loss function contained in this latent objective has a similar configuration with non-convex regularized penalty (NSPR) known in statistics and machine learning. Such connection inspires us discovering more intrinsic relationship between SPL regimes and NSPR forms, like SCAD, LOG and EXP. The robustness insight under SPL can then be finely explained. We also analyze the capability of SPL on its easy loss prior embedding property, and provide an insightful interpretation to the effectiveness mechanism under previous SPL variations. Besides, we design a group-partial-order loss prior, which is especially useful to weakly labeled large-scale data processing tasks. Through applying SPL with this loss prior to the FCVID dataset, which is currently one of the biggest manually annotated video dataset, our method achieves state-of-the-art performance beyond previous methods, which further helps supports the proposed theoretical arguments.

25 pages, 1 figures

References in corpus (4)

• Nearly unbiased variable selection under minimax concave penalty
• Exploiting Feature and Class Relationships in Video Categorization with Regularized Deep Neural Networks
• Efficient Large Scale Video Classification
• On the Global Convergence of Majorization Minimization Algorithms for Nonconvex Optimization Problems

Cited by in corpus (17)

• MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels
• Active Bias: Training More Accurate Neural Networks by Emphasizing High Variance Samples
• Small Sample Learning in Big Data Era
• Derivative Manipulation for General Example Weighting
• A Self-paced Regularization Framework for Partial-Label Learning
• Complex Scene Classification of PolSAR Imagery based on a Self-paced Learning Approach
• Boosted Zero-Shot Learning with Semantic Correlation Regularization
• Bi-Skip: A Motion Deblurring Network Using Self-paced Learning
• From Recognition to Prediction: Analysis of Human Action and Trajectory Prediction in Video
• Robust Collaborative Learning with Noisy Labels
• Self-Paced Probabilistic Principal Component Analysis for Data with Outliers
• Distributed Self-Paced Learning in Alternating Direction Method of Multipliers
• Self-paced Principal Component Analysis
• Incompatibility Clustering as a Defense Against Backdoor Poisoning Attacks
• QSAR Classification Modeling for Bioactivity of Molecular Structure via SPL-Logsum
• Efficient online learning for large-scale peptide identification
• Carpe Diem, Seize the Samples Uncertain "At the Moment" for Adaptive Batch Selection