Deep Active Learning by Leveraging Training Dynamics

Abstract

Active learning theories and methods have been extensively studied in classical statistical learning settings. However, deep active learning, i.e., active learning with deep learning models, is usually based on empirical criteria without solid theoretical justification, thus suffering from heavy doubts when some of those fail to provide benefits in real applications. In this paper, by exploring the connection between the generalization performance and the training dynamics, we propose a theory-driven deep active learning method (dynamicAL) which selects samples to maximize training dynamics. In particular, we prove that the convergence speed of training and the generalization performance are positively correlated under the ultra-wide condition and show that maximizing the training dynamics leads to better generalization performance. Furthermore, to scale up to large deep neural networks and data sets, we introduce two relaxations for the subset selection problem and reduce the time complexity from polynomial to constant. Empirical results show that dynamicAL not only outperforms the other baselines consistently but also scales well on large deep learning models. We hope our work would inspire more attempts on bridging the theoretical findings of deep networks and practical impacts of deep active learning in real applications.

arXiv commentAccepted by NeurIPS 2022

References in corpus (13)

• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Understanding Black-box Predictions via Influence Functions
• On Exact Computation with an Infinitely Wide Neural Net
• BatchBALD: Efficient and Diverse Batch Acquisition for Deep Bayesian Active Learning
• Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networks
• Graph Neural Tangent Kernel: Fusing Graph Neural Networks with Graph Kernels
• Diverse mini-batch Active Learning
• Active Learning for Speech Recognition: the Power of Gradients
• Algorithmic stability and hypothesis complexity
• Optimization of Graph Neural Networks: Implicit Acceleration by Skip Connections and More Depth
• Gradients as Features for Deep Representation Learning
• A Bayesian Perspective on Training Speed and Model Selection
• Bayesian Uncertainty and Expected Gradient Length -- Regression: Two Sides Of The Same Coin?