Graph Interpolating Activation Improves Both Natural and Robust Accuracies in Data-Efficient Deep Learning

Abstract

Improving the accuracy and robustness of deep neural nets (DNNs) and adapting them to small training data are primary tasks in deep learning research. In this paper, we replace the output activation function of DNNs, typically the data-agnostic softmax function, with a graph Laplacian-based high dimensional interpolating function which, in the continuum limit, converges to the solution of a Laplace-Beltrami equation on a high dimensional manifold. Furthermore, we propose end-to-end training and testing algorithms for this new architecture. The proposed DNN with graph interpolating activation integrates the advantages of both deep learning and manifold learning. Compared to the conventional DNNs with the softmax function as output activation, the new framework demonstrates the following major advantages: First, it is better applicable to data-efficient learning in which we train high capacity DNNs without using a large number of training data. Second, it remarkably improves both natural accuracy on the clean images and robust accuracy on the adversarial images crafted by both white-box and black-box adversarial attacks. Third, it is a natural choice for semi-supervised learning. For reproducibility, the code is available at \url{https://github.com/BaoWangMath/DNN-DataDependentActivation}.

34 pages, 10 figures

References in corpus (17)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Improving neural networks by preventing co-adaptation of feature detectors
• Transferability in Machine Learning: from Phenomena to Black-Box Attacks using Adversarial Samples
• Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning
• Theoretically Principled Trade-off between Robustness and Accuracy
• Deep Learning using Linear Support Vector Machines
• Manifold Mixup: Better Representations by Interpolating Hidden States
• Learning Activation Functions to Improve Deep Neural Networks
• Improving the Adversarial Robustness and Interpretability of Deep Neural Networks by Regularizing their Input Gradients
• Deep Residual Learning and PDEs on Manifold
• Laplacian Smoothing Gradient Descent
• Deep Neural Nets with Interpolating Function as Output Activation
• Adversarial Defense via Data Dependent Activation Function and Total Variation Minimization
• ResNets Ensemble via the Feynman-Kac Formalism to Improve Natural and Robust Accuracies
• Mathematical Analysis of Adversarial Attacks
• Error estimation of weighted nonlocal Laplacian on random point cloud