Semantic Representation and Inference for NLP

Abstract

Semantic representation and inference is essential for Natural Language Processing (NLP). The state of the art for semantic representation and inference is deep learning, and particularly Recurrent Neural Networks (RNNs), Convolutional Neural Networks (CNNs), and transformer Self-Attention models. This thesis investigates the use of deep learning for novel semantic representation and inference, and makes contributions in the following three areas: creating training data, improving semantic representations and extending inference learning. In terms of creating training data, we contribute the largest publicly available dataset of real-life factual claims for the purpose of automatic claim verification (MultiFC), and we present a novel inference model composed of multi-scale CNNs with different kernel sizes that learn from external sources to infer fact checking labels. In terms of improving semantic representations, we contribute a novel model that captures non-compositional semantic indicators. By definition, the meaning of a non-compositional phrase cannot be inferred from the individual meanings of its composing words (e.g., hot dog). Motivated by this, we operationalize the compositionality of a phrase contextually by enriching the phrase representation with external word embeddings and knowledge graphs. Finally, in terms of inference learning, we propose a series of novel deep learning architectures that improve inference by using syntactic dependencies, by ensembling role guided attention heads, incorporating gating layers, and concatenating multiple heads in novel and effective ways. This thesis consists of seven publications (five published and two under review).

arXiv commentPhD thesis, the University of Copenhagen

References in corpus (21)

• Distributed Representations of Words and Phrases and their Compositionality
• Machine Learning in Automated Text Categorization
• Natural Language Processing (almost) from Scratch
• An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling
• Analysing How People Orient to and Spread Rumours in Social Media by Looking at Conversational Threads
• Computational fact checking from knowledge networks
• Representation learning for very short texts using weighted word embedding aggregation
• Synthesizer: Rethinking Self-Attention in Transformer Models
• Discourse-Aware Rumour Stance Classification in Social Media Using Sequential Classifiers
• Evaluation of sentence embeddings in downstream and linguistic probing tasks
• A Tensorized Transformer for Language Modeling
• Encoding word order in complex embeddings
• From Word Embeddings to Item Recommendation
• exBERT: A Visual Analysis Tool to Explore Learned Representations in Transformers Models
• Neural Speed Reading with Structural-Jump-LSTM
• Information Aggregation for Multi-Head Attention with Routing-by-Agreement
• Multi-Granularity Self-Attention for Neural Machine Translation
• Adaptive Joint Learning of Compositional and Non-Compositional Phrase Embeddings
• Modelling Sequential Music Track Skips using a Multi-RNN Approach
• Gated Group Self-Attention for Answer Selection
• A Mixture of $h-1$ Heads is Better than $h$ Heads