Improving Anytime Prediction with Parallel Cascaded Networks and a Temporal-Difference Loss

Abstract

Although deep feedforward neural networks share some characteristics with the primate visual system, a key distinction is their dynamics. Deep nets typically operate in serial stages wherein each layer completes its computation before processing begins in subsequent layers. In contrast, biological systems have cascaded dynamics: information propagates from neurons at all layers in parallel but transmission occurs gradually over time, leading to speed-accuracy trade offs even in feedforward architectures. We explore the consequences of biologically inspired parallel hardware by constructing cascaded ResNets in which each residual block has propagation delays but all blocks update in parallel in a stateful manner. Because information transmitted through skip connections avoids delays, the functional depth of the architecture increases over time, yielding anytime predictions that improve with internal-processing time. We introduce a temporal-difference training loss that achieves a strictly superior speed-accuracy profile over standard losses and enables the cascaded architecture to outperform state-of-the-art anytime-prediction methods. The cascaded architecture has intriguing properties, including: it classifies typical instances more rapidly than atypical instances; it is more robust to both persistent and transient noise than is a conventional ResNet; and its time-varying output trace provides a signal that can be exploited to improve information processing and inference.

References in corpus (11)

• Wide Residual Networks
• LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
• Adaptive Computation Time for Recurrent Neural Networks
• Multi-Scale Dense Networks for Resource Efficient Image Classification
• IDK Cascades: Fast Deep Learning by Learning not to Overthink
• IamNN: Iterative and Adaptive Mobile Neural Network for Efficient Image Classification
• Learning to Stop While Learning to Predict
• Anytime Neural Prediction via Slicing Networks Vertically
• The streaming rollout of deep networks - towards fully model-parallel execution
• Convolutional Bipartite Attractor Networks
• Anytime Prediction as a Model of Human Reaction Time