A Distributionally Robust Optimization Method for Adversarial Multiple Kernel Learning
arXiv:1902.10365
Abstract
We propose a novel data-driven method to learn a mixture of multiple kernels with random features that is certifiabaly robust against adverserial inputs. Specifically, we consider a distributionally robust optimization of the kernel-target alignment with respect to the distribution of training samples over a distributional ball defined by the Kullback-Leibler (KL) divergence. The distributionally robust optimization problem can be recast as a min-max optimization whose objective function includes a log-sum term. We develop a mini-batch biased stochastic primal-dual proximal method to solve the min-max optimization. To debias the minibatch algorithm, we use the Gumbel perturbation technique to estimate the log-sum term. We establish theoretical guarantees for the performance of the proposed multiple kernel learning method. In particular, we prove the consistency, asymptotic normality, stochastic equicontinuity, and the minimax rate of the empirical estimators. In addition, based on the notion of Rademacher and Gaussian complexities, we establish distributionally robust generalization bounds that are tighter than previous known bounds. More specifically, we leverage matrix concentration inequalities to establish distributionally robust generalization bounds. We validate our kernel learning approach for classification with the kernel SVMs on synthetic dataset generated by sampling multvariate Gaussian distributions with differernt variance structures. We also apply our kernel learning approach to the MNIST data-set and evaluate its robustness to perturbation of input images under different adversarial models. More specifically, we examine the robustness of the proposed kernel model selection technique against FGSM, PGM, C\&W, and DDN adversarial perturbations, and compare its performance with alternative state-of-the-art multiple kernel learning paradigms.
arXiv commentMajor revision. The title and abstract have been updated
References in corpus (26)
- Adam: A Method for Stochastic Optimization
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Towards Deep Learning Models Resistant to Adversarial Attacks
- Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples
- Estimating divergence functionals and the likelihood ratio by convex risk minimization
- Group Sparse Regularization for Deep Neural Networks
- Algorithms for Learning Kernels Based on Centered Alignment
- Adversarial Risk and the Dangers of Evaluating Against Weak Attacks
- Certifying Some Distributional Robustness with Principled Adversarial Training
- Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methods
- Kernel dimension reduction in regression
- Defense-GAN: Protecting Classifiers Against Adversarial Attacks Using Generative Models
- Robust Wasserstein Profile Inference and Applications to Machine Learning
- Certified Adversarial Robustness with Additive Noise
- Distributionally Robust Logistic Regression
- On the Robustness of the CVPR 2018 White-Box Adversarial Example Defenses
- Transfer Learning from Deep Features for Remote Sensing and Poverty Mapping
- Training verified learners with learned verifiers
- Provable Robustness of ReLU networks via Maximization of Linear Regions
- Kernel Feature Selection via Conditional Covariance Minimization
- MKLpy: a python-based framework for Multiple Kernel Learning
- Multiple Kernel Learning from Noisy Labels by Stochastic Programming
- Approximate Leave-One-Out for Fast Parameter Tuning in High Dimensions
- Concentration of quadratic forms under a Bernstein moment assumption
- New Generalization Bounds for Learning Kernels
- Learning Bounds for Greedy Approximation with Explicit Feature Maps from Multiple Kernels