Proximal Mapping for Deep Regularization
arXiv:2006.07822
Abstract
Underpinning the success of deep learning is effective regularizations that allow a variety of priors in data to be modeled. For example, robustness to adversarial perturbations, and correlations between multiple modalities. However, most regularizers are specified in terms of hidden layer outputs, which are not themselves optimization variables. In contrast to prevalent methods that optimize them indirectly through model weights, we propose inserting proximal mapping as a new layer to the deep network, which directly and explicitly produces well regularized hidden layer outputs. The resulting technique is shown well connected to kernel warping and dropout, and novel algorithms were developed for robust temporal learning and multiview modeling, both outperforming state-of-the-art methods.
24 pages, 7 figures
References in corpus (17)
- Prototypical Networks for Few-shot Learning
- Matching Networks for One Shot Learning
- Task-Driven Dictionary Learning
- Tune: A Research Platform for Distributed Model Selection and Training
- Unrolled Generative Adversarial Networks
- Meta-Learning with Implicit Gradients
- Differentiable MPC for End-to-end Planning and Control
- Meta-learning with differentiable closed-form solvers
- OptNet: Differentiable Optimization as a Layer in Neural Networks
- Differentiable Convex Optimization Layers
- On Differentiating Parameterized Argmin and Argmax Problems with Application to Bi-level Optimization
- Meta-Learning and Universality: Deep Representations and Gradient Descent can Approximate any Learning Algorithm
- SparseMAP: Differentiable Sparse Structured Inference
- Separated by an Un-common Language: Towards Judgment Language Informed Vector Space Modeling
- End-to-End Learning for Structured Prediction Energy Networks
- Adversarially Robust Training through Structured Gradient Regularization
- Max-Margin Invariant Features from Transformed Unlabeled Data