Discrete and continuous representations and processing in deep learning: Looking forward

Abstract

Discrete and continuous representations of content (e.g., of language or images) have interesting properties to be explored for the understanding of or reasoning with this content by machines. This position paper puts forward our opinion on the role of discrete and continuous representations and their processing in the deep learning field. Current neural network models compute continuous-valued data. Information is compressed into dense, distributed embeddings. By stark contrast, humans use discrete symbols in their communication with language. Such symbols represent a compressed version of the world that derives its meaning from shared contextual information. Additionally, human reasoning involves symbol manipulation at a cognitive level, which facilitates abstract reasoning, the composition of knowledge and understanding, generalization and efficient learning. Motivated by these insights, in this paper we argue that combining discrete and continuous representations and their processing will be essential to build systems that exhibit a general form of intelligence. We suggest and discuss several avenues that could improve current neural networks with the inclusion of discrete elements to combine the advantages of both types of representations.

References in corpus (15)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Sequence to Sequence Learning with Neural Networks
• Language Models are Few-Shot Learners
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• Unmasking Clever Hans Predictors and Assessing What Machines Really Learn
• Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challenge
• Zero-Shot Learning Through Cross-Modal Transfer
• A simple neural network module for relational reasoning
• The Next Decade in AI: Four Steps Towards Robust Artificial Intelligence
• Neural-Symbolic Learning and Reasoning: A Survey and Interpretation
• Measuring the tendency of CNNs to Learn Surface Statistical Regularities
• Deep Modular Co-Attention Networks for Visual Question Answering
• Contrastive Learning of Structured World Models
• An Unsupervised Autoregressive Model for Speech Representation Learning
• Visual Relationship Detection using Scene Graphs: A Survey