Convolutional Bipartite Attractor Networks

Abstract

In human perception and cognition, a fundamental operation that brains perform is interpretation: constructing coherent neural states from noisy, incomplete, and intrinsically ambiguous evidence. The problem of interpretation is well matched to an early and often overlooked architecture, the attractor network---a recurrent neural net that performs constraint satisfaction, imputation of missing features, and clean up of noisy data via energy minimization dynamics. We revisit attractor nets in light of modern deep learning methods and propose a convolutional bipartite architecture with a novel training loss, activation function, and connectivity constraints. We tackle larger problems than have been previously explored with attractor nets and demonstrate their potential for image completion and super-resolution. We argue that this architecture is better motivated than ever-deeper feedforward models and is a viable alternative to more costly sampling-based generative methods on a range of supervised and unsupervised tasks.

References in corpus (8)

• Density estimation using Real NVP
• Recurrent computations for visual pattern completion
• A Theory of Generative ConvNet
• Task-Driven Convolutional Recurrent Models of the Visual System
• Implicit Generation and Generalization in Energy-Based Models
• Learning Attractor Dynamics for Generative Memory
• Associative content-addressable networks with exponentially many robust stable states
• State-Reification Networks: Improving Generalization by Modeling the Distribution of Hidden Representations

Cited by in corpus (3)

• On the Binding Problem in Artificial Neural Networks
• Hierarchical Associative Memory
• Improving Anytime Prediction with Parallel Cascaded Networks and a Temporal-Difference Loss