Hierarchically Compositional Tasks and Deep Convolutional Networks

Abstract

The main success stories of deep learning, starting with ImageNet, depend on deep convolutional networks, which on certain tasks perform significantly better than traditional shallow classifiers, such as support vector machines, and also better than deep fully connected networks; but what is so special about deep convolutional networks? Recent results in approximation theory proved an exponential advantage of deep convolutional networks with or without shared weights in approximating functions with hierarchical locality in their compositional structure. More recently, the hierarchical structure was proved to be hard to learn from data, suggesting that it is a powerful prior embedded in the architecture of the network. These mathematical results, however, do not say which real-life tasks correspond to input-output functions with hierarchical locality. To evaluate this, we consider a set of visual tasks where we disrupt the local organization of images via "deterministic scrambling" to later perform a visual task on these images structurally-altered in the same way for training and testing. For object recognition we find, as expected, that scrambling does not affect the performance of shallow or deep fully connected networks contrary to the out-performance of convolutional networks. Not all tasks involving images are however affected. Texture perception and global color estimation are much less sensitive to deterministic scrambling showing that the underlying functions corresponding to these tasks are not hierarchically local; and also counter-intuitively showing that these tasks are better approximated by networks that are not deep (texture) nor convolutional (color). Altogether, these results shed light into the importance of matching a network architecture with its embedded prior of the task to be learned.

arXiv commentA pre-print. Currently Under Review

References in corpus (22)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Shortcut Learning in Deep Neural Networks
• On the Number of Linear Regions of Deep Neural Networks
• ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness
• Learning to Execute
• Recurrent computations for visual pattern completion
• Neural GPUs Learn Algorithms
• Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNet
• Perceiver: General Perception with Iterative Attention
• Representation Benefits of Deep Feedforward Networks
• Going in circles is the way forward: the role of recurrence in visual inference
• Nonlinear Approximation and (Deep) ReLU Networks
• Depth-Width Tradeoffs in Approximating Natural Functions with Neural Networks
• Failures of Gradient-Based Deep Learning
• Emergent Properties of Foveated Perceptual Systems
• Towards Metamerism via Foveated Style Transfer
• Towards Learning Convolutions from Scratch
• Error bounds for approximations with deep ReLU networks
• The local low-dimensionality of natural images
• Spatial and Colour Opponency in Anatomically Constrained Deep Networks
• Do Neural Networks for Segmentation Understand Insideness?
• The Effects of Image Distribution and Task on Adversarial Robustness

Cited by in corpus (2)

• Locality defeats the curse of dimensionality in convolutional teacher-student scenarios
• The Effects of Image Distribution and Task on Adversarial Robustness