A Simple Recurrent Unit with Reduced Tensor Product Representations

Abstract

idely used recurrent units, including Long-short Term Memory (LSTM) and the Gated Recurrent Unit (GRU), perform well on natural language tasks, but their ability to learn structured representations is still questionable. Exploiting reduced Tensor Product Representations (TPRs) --- distributed representations of symbolic structure in which vector-embedded symbols are bound to vector-embedded structural positions --- we propose the TPRU, a simple recurrent unit that, at each time step, explicitly executes structural-role binding and unbinding operations to incorporate structural information into learning. A gradient analysis of our proposed TPRU is conducted to support our model design, and its performance on multiple datasets shows the effectiveness of our design choices. Furthermore, observations on a linguistically grounded study demonstrate the interpretability of our TPRU.

References in corpus (13)

• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• On the difficulty of training Recurrent Neural Networks
• Bidirectional Attention Flow for Machine Comprehension
• Visualizing and Understanding Recurrent Networks
• GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
• What you can cram into a single vector: Probing sentence embeddings for linguistic properties
• Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling
• FRAGE: Frequency-Agnostic Word Representation
• AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networks
• Correlating neural and symbolic representations of language
• Dynamic Evaluation of Transformer Language Models
• GLoMo: Unsupervisedly Learned Relational Graphs as Transferable Representations
• Learning to Reason with Third-Order Tensor Products