Correlating neural and symbolic representations of language

Abstract

Analysis methods which enable us to better understand the representations and functioning of neural models of language are increasingly needed as deep learning becomes the dominant approach in NLP. Here we present two methods based on Representational Similarity Analysis (RSA) and Tree Kernels (TK) which allow us to directly quantify how strongly the information encoded in neural activation patterns corresponds to information represented by symbolic structures such as syntax trees. We first validate our methods on the case of a simple synthetic language for arithmetic expressions with clearly defined syntax and semantics, and show that they exhibit the expected pattern of results. We then apply our methods to correlate neural representations of English sentences with their constituency parse trees.

ACL 2019

References in corpus (1)

• Skip-Thought Vectors

Cited by in corpus (12)

• Discovering the Compositional Structure of Vector Representations with Role Learning Networks
• Analyzing analytical methods: The case of phonology in neural models of spoken language
• Blackbox meets blackbox: Representational Similarity and Stability Analysis of Neural Language Models and Brains
• On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptation
• Does Data Augmentation Improve Generalization in NLP?
• Wave to Syntax: Probing spoken language models for syntax
• Differential Privacy, Linguistic Fairness, and Training Data Influence: Impossibility and Possibility Theorems for Multilingual Language Models
• How Familiar Does That Sound? Cross-Lingual Representational Similarity Analysis of Acoustic Word Embeddings
• All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Quality
• We went to look for meaning and all we got were these lousy representations: aspects of meaning representation for computational semantics
• Distilling neural networks into skipgram-level decision lists
• Discrete representations in neural models of spoken language