Learning with Multiplicative Perturbations

Abstract

Adversarial Training (AT) and Virtual Adversarial Training (VAT) are the regularization techniques that train Deep Neural Networks (DNNs) with adversarial examples generated by adding small but worst-case perturbations to input examples. In this paper, we propose xAT and xVAT, new adversarial training algorithms, that generate \textbf{multiplicative} perturbations to input examples for robust training of DNNs. Such perturbations are much more perceptible and interpretable than their \textbf{additive} counterparts exploited by AT and VAT. Furthermore, the multiplicative perturbations can be generated transductively or inductively while the standard AT and VAT only support a transductive implementation. We conduct a series of experiments that analyze the behavior of the multiplicative perturbations and demonstrate that xAT and xVAT match or outperform state-of-the-art classification accuracies across multiple established benchmarks while being about 30\% faster than their additive counterparts. Furthermore, the resulting DNNs also demonstrate distinct weight distributions.

arXiv commentAccepted as a conference paper at ICPR 2020

References in corpus (8)

• Categorical Reparameterization with Gumbel-Softmax
• Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• Semi-Supervised Learning with Ladder Networks
• REBAR: Low-variance, unbiased gradient estimates for discrete latent variable models
• There Are Many Consistent Explanations of Unlabeled Data: Why You Should Average
• $L_0$-ARM: Network Sparsification via Stochastic Binary Optimization
• Stochastic Variational Optimization

Cited by in corpus (1)

• Improving Resistance to Adversarial Deformations by Regularizing Gradients