Discrete and continuous representations and processing in deep learning: Looking forward
arXiv:2201.01233 · doi:10.1016/j.aiopen.2021.07.002
Abstract
Discrete and continuous representations of content (e.g., of language or images) have interesting properties to be explored for the understanding of or reasoning with this content by machines. This position paper puts forward our opinion on the role of discrete and continuous representations and their processing in the deep learning field. Current neural network models compute continuous-valued data. Information is compressed into dense, distributed embeddings. By stark contrast, humans use discrete symbols in their communication with language. Such symbols represent a compressed version of the world that derives its meaning from shared contextual information. Additionally, human reasoning involves symbol manipulation at a cognitive level, which facilitates abstract reasoning, the composition of knowledge and understanding, generalization and efficient learning. Motivated by these insights, in this paper we argue that combining discrete and continuous representations and their processing will be essential to build systems that exhibit a general form of intelligence. We suggest and discuss several avenues that could improve current neural networks with the inclusion of discrete elements to combine the advantages of both types of representations.
References in corpus (62)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Efficient Estimation of Word Representations in Vector Space
- Auto-Encoding Variational Bayes
- Neural Machine Translation by Jointly Learning to Align and Translate
- Sequence to Sequence Learning with Neural Networks
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- Representation Learning with Contrastive Predictive Coding
- Categorical Reparameterization with Gumbel-Softmax
- Language Models are Few-Shot Learners
- Stochastic Backpropagation and Approximate Inference in Deep Generative Models
- SchNet - a deep learning architecture for molecules and materials
- Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
- Shortcut Learning in Deep Neural Networks
- ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
- Graph Neural Networks: A Review of Methods and Applications
- The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
- Unmasking Clever Hans Predictors and Assessing What Machines Really Learn
- Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challenge
- Zero-Shot Learning Through Cross-Modal Transfer
- VL-BERT: Pre-training of Generic Visual-Linguistic Representations
- Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations
- Disentangling by Factorising
- CNN Features off-the-shelf: an Astounding Baseline for Recognition
- A simple neural network module for relational reasoning
- More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touch
- Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding
- Visual Reinforcement Learning with Imagined Goals
- The Next Decade in AI: Four Steps Towards Robust Artificial Intelligence
- Discovering Symbolic Models from Deep Learning with Inductive Biases
- Deep Reinforcement Learning in Large Discrete Action Spaces
- Neural Relational Inference for Interacting Systems
- Attend, Infer, Repeat: Fast Scene Understanding with Generative Models
- Neural-Symbolic Learning and Reasoning: A Survey and Interpretation
- Object-Centric Learning with Slot Attention
- Transparency by Design: Closing the Gap Between Performance and Interpretability in Visual Reasoning
- Measuring the tendency of CNNs to Learn Surface Statistical Regularities
- VideoBERT: A Joint Model for Video and Language Representation Learning
- GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representations
- Compositional Attention Networks for Machine Reasoning
- Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training
- DeepProbLog: Neural Probabilistic Logic Programming
- Learning to Reason: End-to-End Module Networks for Visual Question Answering
- Deep Modular Co-Attention Networks for Visual Question Answering
- Pretrained Transformers as Universal Computation Engines
- Contrastive Learning of Structured World Models
- Interpretability and Explainability: A Machine Learning Zoo Mini-tour
- COBRA: Data-Efficient Model-Based RL through Unsupervised Object Discovery and Curiosity-Driven Exploration
- Image Generation from Scene Graphs
- Hyperbolic Attention Networks
- Generative Models of Visually Grounded Imagination
- An Unsupervised Autoregressive Model for Speech Representation Learning
- Unbiased Scene Graph Generation from Biased Training
- Neural Enhanced Belief Propagation on Factor Graphs
- Probing Neural Network Comprehension of Natural Language Arguments
- Convolutional Generation of Textured 3D Meshes
- Rearranging the Familiar: Testing Compositional Generalization in Recurrent Networks
- What they do when in doubt: a study of inductive biases in seq2seq learners
- Visual Relationship Detection using Scene Graphs: A Survey
- Connecting Look and Feel: Associating the visual and tactile properties of physical materials
- Vokenization: Improving Language Understanding with Contextualized, Visual-Grounded Supervision
- Algebraic Machine Learning
- Temporal Information Extraction by Predicting Relative Time-lines