Guess First to Enable Better Compression and Adversarial Robustness

Abstract

Machine learning models are generally vulnerable to adversarial examples, which is in contrast to the robustness of humans. In this paper, we try to leverage one of the mechanisms in human recognition and propose a bio-inspired classification framework in which model inference is conditioned on label hypothesis. We provide a class of training objectives for this framework and an information bottleneck regularizer which utilizes the advantage that label information can be discarded during inference. This framework enables better compression of the mutual information between inputs and latent representations without loss of learning capacity, at the cost of tractable inference complexity. Better compression and elimination of label information further bring better adversarial robustness without loss of natural accuracy, which is demonstrated in the experiment.

arXiv commentAccepted by NeurIPS 2019 workshop on Information Theory and Machine Learning

References in corpus (9)

• Explaining and Harnessing Adversarial Examples
• Learning deep representations by mutual information estimation and maximization
• Estimating divergence functionals and the likelihood ratio by convex risk minimization
• Deep Variational Information Bottleneck
• Defense-GAN: Protecting Classifiers Against Adversarial Attacks Using Generative Models
• Sparse DNNs with Improved Adversarial Robustness
• Excessive Invariance Causes Adversarial Vulnerability
• Learners that Use Little Information
• Complement Objective Training