Learning Less-Overlapping Representations
arXiv:1711.09300
Abstract
In representation learning (RL), how to make the learned representations easy to interpret and less overfitted to training data are two important but challenging issues. To address these problems, we study a new type of regulariza- tion approach that encourages the supports of weight vectors in RL models to have small overlap, by simultaneously promoting near-orthogonality among vectors and sparsity of each vector. We apply the proposed regularizer to two models: neural networks (NNs) and sparse coding (SC), and develop an efficient ADMM-based algorithm for regu- larized SC. Experiments on various datasets demonstrate that weight vectors learned under our regularizer are more interpretable and have better generalization performance.
References in corpus (9)
- Neural Architecture Search with Reinforcement Learning
- Striving for Simplicity: The All Convolutional Net
- Recurrent Neural Network Regularization
- Pointer Sentinel Mixture Models
- Fractional Max-Pooling
- Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling
- Regularizing CNNs with Locally Constrained Decorrelations
- Deep Pyramidal Residual Networks with Separated Stochastic Depth
- Improving Interpretability of Deep Neural Networks with Semantic Information