Towards Robust, Locally Linear Deep Networks

Abstract

Deep networks realize complex mappings that are often understood by their locally linear behavior at or around points of interest. For example, we use the derivative of the mapping with respect to its inputs for sensitivity analysis, or to explain (obtain coordinate relevance for) a prediction. One key challenge is that such derivatives are themselves inherently unstable. In this paper, we propose a new learning problem to encourage deep networks to have stable derivatives over larger regions. While the problem is challenging in general, we focus on networks with piecewise linear activation functions. Our algorithm consists of an inference step that identifies a region around a point where linear approximation is provably stable, and an optimization step to expand such regions. We propose a novel relaxation to scale the algorithm to realistic models. We illustrate our method with residual and recurrent networks on image and sequence datasets.

Published in International Conference on Learning Representations (ICLR), 2019

References in corpus (15)

• Axiomatic Attribution for Deep Networks
• Striving for Simplicity: The All Convolutional Net
• On the Convergence of Adam and Beyond
• Understanding Black-box Predictions via Influence Functions
• On the Number of Linear Regions of Deep Neural Networks
• SmoothGrad: removing noise by adding noise
• How to Explain Individual Classification Decisions
• Provable defenses against adversarial examples via the convex outer adversarial polytope
• The Cramer Distance as a Solution to Biased Wasserstein Gradients
• An approach to reachability analysis for feed-forward ReLU neural networks
• Deep Defense: Training DNNs with Improved Adversarial Robustness
• Learning to Draw Samples: With Application to Amortized MLE for Generative Adversarial Learning
• Provable Robustness of ReLU networks via Maximization of Linear Regions
• Deep Neural Networks as 0-1 Mixed Integer Linear Programs: A Feasibility Study
• A Generative Process for Sampling Contractive Auto-Encoders

Cited by in corpus (11)

• On the (In)fidelity and Sensitivity for Explanations
• Tight Certificates of Adversarial Robustness for Randomly Smoothed Classifiers
• Certifiably Robust Interpretation in Deep Learning
• Oblique Decision Trees from Derivatives of ReLU Networks
• Understanding Adversarial Robustness: The Trade-off between Minimum and Average Margin
• Sound and Complete Neural Network Repair with Minimality and Locality Guarantees
• Towards Ground Truth Explainability on Tabular Data
• How and When Adversarial Robustness Transfers in Knowledge Distillation?
• A Game Theoretic Analysis of Additive Adversarial Attacks and Defenses
• DWDN: Deep Wiener Deconvolution Network for Non-Blind Image Deblurring
• Recent Advances in Large Margin Learning