Deep Neural Nets with Interpolating Function as Output Activation

Abstract

We replace the output layer of deep neural nets, typically the softmax function, by a novel interpolating function. And we propose end-to-end training and testing algorithms for this new architecture. Compared to classical neural nets with softmax function as output activation, the surrogate with interpolating function as output activation combines advantages of both deep and manifold learning. The new framework demonstrates the following major advantages: First, it is better applicable to the case with insufficient training data. Second, it significantly improves the generalization accuracy on a wide variety of networks. The algorithm is implemented in PyTorch, and code will be made publicly available.

11 pages, 4 figures

Cited by in corpus (12)

• Understanding Straight-Through Estimator in Training Activation Quantized Neural Nets
• On Robustness of Neural Ordinary Differential Equations
• Adversarial Defense via Data Dependent Activation Function and Total Variation Minimization
• ResNets Ensemble via the Feynman-Kac Formalism to Improve Natural and Robust Accuracies
• Blended Coarse Gradient Descent for Full Quantization of Deep Neural Networks
• Better Modelling Out-of-Distribution Regression on Distributed Acoustic Sensor Data Using Anchored Hidden State Mixup
• Error estimation of weighted nonlocal Laplacian on random point cloud
• Mathematical Analysis of Adversarial Attacks
• Robust Certification for Laplace Learning on Geometric Graphs
• Analytic Continuation of Noisy Data Using Adams Bashforth ResNet
• Graph Interpolating Activation Improves Both Natural and Robust Accuracies in Data-Efficient Deep Learning
• An Integrated Approach to Produce Robust Models with High Efficiency