Learning Adversarially Robust Representations via Worst-Case Mutual Information Maximization

Abstract

Training machine learning models that are robust against adversarial inputs poses seemingly insurmountable challenges. To better understand adversarial robustness, we consider the underlying problem of learning robust representations. We develop a notion of representation vulnerability that captures the maximum change of mutual information between the input and output distributions, under the worst-case input perturbation. Then, we prove a theorem that establishes a lower bound on the minimum adversarial risk that can be achieved for any downstream classifier based on its representation vulnerability. We propose an unsupervised learning method for obtaining intrinsically robust representations by maximizing the worst-case mutual information between the input and output distributions. Experiments on downstream classification tasks support the robustness of the representations found using unsupervised learning with our training principle.

ICML 2020

References in corpus (12)

• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Learning deep representations by mutual information estimation and maximization
• Theoretically Principled Trade-off between Robustness and Accuracy
• Adversarially Robust Generalization Requires More Data
• MINE: Mutual Information Neural Estimation
• Adversarial vulnerability for any classifier
• Adversarial examples from computational constraints
• Adversarial Robustness as a Prior for Learned Representations
• Adversarial Spheres
• On the Connection Between Adversarial Robustness and Saliency Map Interpretability
• A Spectral View of Adversarially Robust Features
• Robust Classification using Robust Feature Augmentation

Cited by in corpus (3)

• InfoBERT: Improving Robustness of Language Models from An Information Theoretic Perspective
• Learning Certified Individually Fair Representations
• Adversarial Feature Desensitization