Improving Compositionality of Neural Networks by Decoding Representations to Inputs

Abstract

In traditional software programs, it is easy to trace program logic from variables back to input, apply assertion statements to block erroneous behavior, and compose programs together. Although deep learning programs have demonstrated strong performance on novel applications, they sacrifice many of the functionalities of traditional software programs. With this as motivation, we take a modest first step towards improving deep learning programs by jointly training a generative model to constrain neural network activations to "decode" back to inputs. We call this design a Decodable Neural Network, or DecNN. Doing so enables a form of compositionality in neural networks, where one can recursively compose DecNN with itself to create an ensemble-like model with uncertainty. In our experiments, we demonstrate applications of this uncertainty to out-of-distribution detection, adversarial example detection, and calibration -- while matching standard neural networks in accuracy. We further explore this compositionality by combining DecNN with pretrained models, where we show promising results that neural networks can be regularized from using protected features.

arXiv comment9 pages content

References in corpus (22)

• Deep Learning in Neural Networks: An Overview
• Denoising Diffusion Probabilistic Models
• Diffusion Models Beat GANs on Image Synthesis
• Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networks
• Score-Based Generative Modeling through Stochastic Differential Equations
• Generative Modeling by Estimating Gradients of the Data Distribution
• A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks
• Density estimation using Real NVP
• Invertible Conditional GANs for image editing
• Deep Anomaly Detection with Outlier Exposure
• Detecting Adversarial Samples from Artifacts
• Improving the Adversarial Robustness and Interpretability of Deep Neural Networks by Regularizing their Input Gradients
• Masked Autoregressive Flow for Density Estimation
• Selective Classification for Deep Neural Networks
• Denoising Diffusion Implicit Models
• A Framework For Contrastive Self-Supervised Learning And Designing A New Approach
• SelectiveNet: A Deep Neural Network with an Integrated Reject Option
• Learning Controllable Fair Representations
• Compositional Explanations of Neurons
• Viewmaker Networks: Learning Views for Unsupervised Representation Learning
• Detecting Out-of-Distribution Examples with In-distribution Examples and Gram Matrices
• Network-to-Network Translation with Conditional Invertible Neural Networks