Scattering Networks for Hybrid Representation Learning

Abstract

Scattering networks are a class of designed Convolutional Neural Networks (CNNs) with fixed weights. We argue they can serve as generic representations for modelling images. In particular, by working in scattering space, we achieve competitive results both for supervised and unsupervised learning tasks, while making progress towards constructing more interpretable CNNs. For supervised learning, we demonstrate that the early layers of CNNs do not necessarily need to be learned, and can be replaced with a scattering network instead. Indeed, using hybrid architectures, we achieve the best results with predefined representations to-date, while being competitive with end-to-end learned CNNs. Specifically, even applying a shallow cascade of small-windowed scattering coefficients followed by 1$\times$1-convolutions results in AlexNet accuracy on the ILSVRC2012 classification task. Moreover, by combining scattering networks with deep residual networks, we achieve a single-crop top-5 error of 11.4% on ILSVRC2012. Also, we show they can yield excellent performance in the small sample regime on CIFAR-10 and STL-10 datasets, exceeding their end-to-end counterparts, through their ability to incorporate geometrical priors. For unsupervised learning, scattering coefficients can be a competitive representation that permits image recovery. We use this fact to train hybrid GANs to generate images. Finally, we empirically analyze several properties related to stability and reconstruction of images from scattering coefficients.

arXiv commentarXiv admin note: substantial text overlap with arXiv:1703.08961

References in corpus (16)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Explaining and Harnessing Adversarial Examples
• Intriguing properties of neural networks
• How transferable are features in deep neural networks?
• Striving for Simplicity: The All Convolutional Net
• Wide Residual Networks
• Understanding Deep Convolutional Networks
• Adversarial Feature Learning
• Highway Networks
• What makes ImageNet good for transfer learning?
• Stacked What-Where Auto-encoders
• Convolutional Kernel Networks
• Audio Texture Synthesis with Scattering Moments
• Learning Stable Group Invariant Representations with Convolutional Networks
• Deep unsupervised learning through spatial contrasting
• Inverse Problems with Invariant Multiscale Statistics

Cited by in corpus (15)

• Differentially Private Learning Needs Better Features (or Much More Data)
• Image Classification with Small Datasets: Overview and Benchmark
• Genetic Programming-Based Evolutionary Deep Learning for Data-Efficient Image Classification
• Greedy Layerwise Learning Can Scale to ImageNet
• Group Equivariant Generative Adversarial Networks
• Deep Network Classification by Scattering and Homotopy Dictionary Learning
• Scattering Transform Based Image Clustering using Projection onto Orthogonal Complement
• MV-MR: multi-views and multi-representations for self-supervised learning and knowledge distillation
• Kernel-based Translations of Convolutional Networks
• ExplainFix: Explainable Spatially Fixed Deep Networks
• Biologically inspired architectures for sample-efficient deep reinforcement learning
• CXR-Net: An Artificial Intelligence Pipeline for Quick Covid-19 Screening of Chest X-Rays
• The Unreasonable Effectiveness of Patches in Deep Convolutional Kernels Methods
• Improving Differentially Private SGD via Randomly Sparsified Gradients
• Wavelet Design in a Learning Framework