Understanding Deep Architectures with Reasoning Layer
arXiv:2006.13401
Abstract
Recently, there has been a surge of interest in combining deep learning models with reasoning in order to handle more sophisticated learning tasks. In many cases, a reasoning task can be solved by an iterative algorithm. This algorithm is often unrolled, and used as a specialized layer in the deep architecture, which can be trained end-to-end with other neural components. Although such hybrid deep architectures have led to many empirical successes, the theoretical foundation of such architectures, especially the interplay between algorithm layers and other neural layers, remains largely unexplored. In this paper, we take an initial step towards an understanding of such hybrid deep architectures by showing that properties of the algorithm layers, such as convergence, stability, and sensitivity, are intimately related to the approximation and generalization abilities of the end-to-end model. Furthermore, our analysis matches closely our experimental observations under various conditions, suggesting that our theory can provide useful guidelines for designing deep architectures with reasoning layers.
arXiv comment34th Conference on Neural Information Processing Systems (NeurIPS 2020)
References in corpus (21)
- Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising
- Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
- On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
- 2004 IMS Medallion Lecture: Local Rademacher complexities and oracle inequalities in risk minimization
- Train faster, generalize better: Stability of stochastic gradient descent
- A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks
- Deep Equilibrium Models
- Meta-Learning with Implicit Gradients
- Theoretical Linear Convergence of Unfolded ISTA and its Practical Weights and Thresholds
- Task-based End-to-end Model Learning in Stochastic Optimization
- ADMM-Net: A Deep Learning Approach for Compressive Sensing MRI
- Differentiable Dynamic Programming for Structured Prediction and Attention
- SparseMAP: Differentiable Sparse Structured Inference
- End to end learning and optimization on graphs
- Generalization and Representational Limits of Graph Neural Networks
- Learning with Differentiable Perturbed Optimizers
- Stability and Convergence Trade-off of Iterative Optimization Algorithms
- Structured Prediction Theory Based on Factor Graph Complexity
- On Generalization Bounds of a Family of Recurrent Neural Networks
- PAC-Bayes bounds for stable algorithms with instance-dependent priors
- Differentiable Ranks and Sorting using Optimal Transport