Emergent Symbols through Binding in External Memory

Abstract

A key aspect of human intelligence is the ability to infer abstract rules directly from high-dimensional sensory data, and to do so given only a limited amount of training experience. Deep neural network algorithms have proven to be a powerful tool for learning directly from high-dimensional data, but currently lack this capacity for data-efficient induction of abstract rules, leading some to argue that symbol-processing mechanisms will be necessary to account for this capacity. In this work, we take a step toward bridging this gap by introducing the Emergent Symbol Binding Network (ESBN), a recurrent network augmented with an external memory that enables a form of variable-binding and indirection. This binding mechanism allows symbol-like representations to emerge through the learning process without the need to explicitly incorporate symbol-processing machinery, enabling the ESBN to learn rules in a manner that is abstracted away from the particular entities to which those rules apply. Across a series of tasks, we show that this architecture displays nearly perfect generalization of learned rules to novel entities given only a limited number of training examples, and outperforms a number of other competitive neural network architectures.

References in corpus (7)

• DRAW: A Recurrent Neural Network For Image Generation
• Using Fast Weights to Attend to the Recent Past
• Analysing Mathematical Reasoning Abilities of Neural Models
• Compositional generalization in a deep seq2seq model by separating syntax and semantics
• Generalization of Reinforcement Learners with Working and Episodic Memory
• The Scattering Compositional Learner: Discovering Objects, Attributes, Relationships in Analogical Reasoning
• A Memory-Augmented Neural Network Model of Abstract Rule Learning