Subjective Question Answering: Deciphering the inner workings of Transformers in the realm of subjectivity

Abstract

Understanding subjectivity demands reasoning skills beyond the realm of common knowledge. It requires a machine learning model to process sentiment and to perform opinion mining. In this work, I've exploited a recently released dataset for span-selection Question Answering, namely SubjQA. SubjQA is the first QA dataset that contains questions that ask for subjective opinions corresponding to review paragraphs from six different domains. Hence, to answer these subjective questions, a learner must extract opinions and process sentiment for various domains, and additionally, align the knowledge extracted from a paragraph with the natural language utterances in the corresponding question, which together enhance the difficulty of a QA task. The primary goal of this thesis was to investigate the inner workings (i.e., latent representations) of a Transformer-based architecture to contribute to a better understanding of these not yet well understood "black-box" models. Transformer's hidden representations, concerning the true answer span, are clustered more closely in vector space than those representations corresponding to erroneous predictions. This observation holds across the top three Transformer layers for both objective and subjective questions and generally increases as a function of layer dimensions. Moreover, the probability to achieve a high cosine similarity among hidden representations in latent space concerning the true answer span tokens is significantly higher for correct compared to incorrect answer span predictions. These results have decisive implications for down-stream applications, where it is crucial to know about why a neural network made mistakes, and in which point, in space and time the mistake has happened (e.g., to automatically predict correctness of an answer span prediction without the necessity of labeled data).

arXiv comment80 pages, Master's thesis in Computer Science (CS)

References in corpus (17)

• Scikit-learn: Machine Learning in Python
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Learning Transferable Features with Deep Adaptation Networks
• Teaching Machines to Read and Comprehend
• An Overview of Multi-Task Learning in Deep Neural Networks
• A Tutorial on Principal Component Analysis
• Bidirectional Attention Flow for Machine Comprehension
• Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning
• GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
• Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks
• Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping
• NewsQA: A Machine Comprehension Dataset
• Semantics-aware BERT for Language Understanding
• One Model to Rule them all: Multitask and Multilingual Modelling for Lexical Analysis
• Symmetric Regularization based BERT for Pair-wise Semantic Reasoning
• Whatcha lookin' at? DeepLIFTing BERT's Attention in Question Answering