Wave to Syntax: Probing spoken language models for syntax

Abstract

Understanding which information is encoded in deep models of spoken and written language has been the focus of much research in recent years, as it is crucial for debugging and improving these architectures. Most previous work has focused on probing for speaker characteristics, acoustic and phonological information in models of spoken language, and for syntactic information in models of written language. Here we focus on the encoding of syntax in several self-supervised and visually grounded models of spoken language. We employ two complementary probing methods, combined with baselines and reference representations to quantify the degree to which syntactic structure is encoded in the activations of the target models. We show that syntax is captured most prominently in the middle layers of the networks, and more explicitly within models with more parameters.

arXiv commentAccepted to Interspeech 2023

References in corpus (7)

• Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders
• Correlating neural and symbolic representations of language
• Visually grounded models of spoken language: A survey of datasets, architectures and evaluation techniques
• Analyzing Hidden Representations in End-to-End Automatic Speech Recognition Systems
• Analyzing analytical methods: The case of phonology in neural models of spoken language
• Self-Supervised Representation Learning for Speech Using Visual Grounding and Masked Language Modeling
• Probing phoneme, language and speaker information in unsupervised speech representations

Cited by in corpus (1)

• Human-like Linguistic Biases in Neural Speech Models: Phonetic Categorization and Phonotactic Constraints in Wav2Vec2.0