A case for new neural network smoothness constraints

Abstract

How sensitive should machine learning models be to input changes? We tackle the question of model smoothness and show that it is a useful inductive bias which aids generalization, adversarial robustness, generative modeling and reinforcement learning. We explore current methods of imposing smoothness constraints and observe they lack the flexibility to adapt to new tasks, they don't account for data modalities, they interact with losses, architectures and optimization in ways not yet fully understood. We conclude that new advances in the field are hinging on finding ways to incorporate data, tasks and learning into our definitions of smoothness.

References in corpus (26)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Auto-Encoding Variational Bayes
• Decoupled Weight Decay Regularization
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Spectral Normalization for Generative Adversarial Networks
• Reconciling modern machine learning practice and the bias-variance trade-off
• Variational Inference with Normalizing Flows
• Estimating divergence functionals and the likelihood ratio by convex risk minimization
• Adversarial Attacks and Defences: A Survey
• Sensitivity and Generalization in Neural Networks: an Empirical Study
• On Mutual Information Maximization for Representation Learning
• Robust Large Margin Deep Neural Networks
• Implicit Quantile Networks for Distributional Reinforcement Learning
• On Convergence and Stability of GANs
• Spectral Norm Regularization for Improving the Generalizability of Deep Learning
• Lipschitz-Margin Training: Scalable Certification of Perturbation Invariance for Deep Neural Networks
• Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness
• On gradient regularizers for MMD GANs
• Lipschitz Generative Adversarial Nets
• The Lipschitz Constant of Self-Attention
• Wasserstein Fair Classification
• Implicit competitive regularization in GANs
• Representing smooth functions as compositions of near-identity functions with implications for deep network optimization
• How to Learn a Useful Critic? Model-based Action-Gradient-Estimator Policy Optimization
• Laplacian Networks: Bounding Indicator Function Smoothness for Neural Network Robustness
• Generalized Energy Based Models

Cited by in corpus (6)

• On Feature Collapse and Deep Kernel Learning for Single Forward Pass Uncertainty
• Spectral Normalisation for Deep Reinforcement Learning: an Optimisation Perspective
• Learning to Assimilate in Chaotic Dynamical Systems
• Can convolutional ResNets approximately preserve input distances? A frequency analysis perspective
• RoMA: Robust Model Adaptation for Offline Model-based Optimization
• The Geometric Occam's Razor Implicit in Deep Learning