Robust Machine Learning via Privacy/Rate-Distortion Theory
arXiv:2007.11693
Abstract
Robust machine learning formulations have emerged to address the prevalent vulnerability of deep neural networks to adversarial examples. Our work draws the connection between optimal robust learning and the privacy-utility tradeoff problem, which is a generalization of the rate-distortion problem. The saddle point of the game between a robust classifier and an adversarial perturbation can be found via the solution of a maximum conditional entropy problem. This information-theoretic perspective sheds light on the fundamental tradeoff between robustness and clean data performance, which ultimately arises from the geometric structure of the underlying data distribution and perturbation constraints.
arXiv comment9 pages, 2 figures, accepted at 2021 IEEE International Symposium on Information Theory
References in corpus (16)
- Theoretically Principled Trade-off between Robustness and Accuracy
- Provable defenses against adversarial examples via the convex outer adversarial polytope
- Certified Defenses against Adversarial Examples
- Utility-Privacy Tradeoff in Databases: An Information-theoretic Approach
- Evaluating Robustness of Neural Networks with Mixed Integer Programming
- Certifying Some Distributional Robustness with Principled Adversarial Training
- Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methods
- Semidefinite relaxations for certifying robustness to adversarial examples
- Minimax Statistical Learning with Wasserstein Distances
- Managing your Private and Public Data: Bringing down Inference Attacks against your Privacy
- Adversarial camera stickers: A physical camera-based attack on deep learning systems
- A Minimax Approach to Supervised Learning
- Optimal Transport Based Distributionally Robust Optimization: Structural Properties and Iterative Schemes
- A Tunable Measure for Information Leakage
- Generalised Lipschitz Regularisation Equals Distributional Robustness
- Machine vs Machine: Minimax-Optimal Defense Against Adversarial Examples