Improving Natural Language Processing Tasks with Human Gaze-Guided Neural Attention

Abstract

A lack of corpora has so far limited advances in integrating human gaze data as a supervisory signal in neural attention mechanisms for natural language processing(NLP). We propose a novel hybrid text saliency model(TSM) that, for the first time, combines a cognitive model of reading with explicit human gaze supervision in a single machine learning framework. On four different corpora we demonstrate that our hybrid TSM duration predictions are highly correlated with human gaze ground truth. We further propose a novel joint modeling approach to integrate TSM predictions into the attention layer of a network designed for a specific upstream NLP task without the need for any task-specific human gaze data. We demonstrate that our joint model outperforms the state of the art in paraphrase generation on the Quora Question Pairs corpus by more than 10% in BLEU-4 and achieves state of the art performance for sentence compression on the challenging Google Sentence Compression corpus. As such, our work introduces a practical approach for bridging between data-driven and cognitive models and demonstrates a new way to integrate human gaze-guided neural attention into NLP tasks.

NeurIPS 2020

References in corpus (14)

• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Teaching Machines to Read and Comprehend
• Recurrent Models of Visual Attention
• Neural Paraphrase Generation with Stacked Residual LSTM Networks
• ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension
• A Deep Generative Framework for Paraphrase Generation
• R-Transformer: Recurrent Neural Network Enhanced Transformer
• Massive Exploration of Neural Machine Translation Architectures
• Seeing with Humans: Gaze-Assisted Neural Image Captioning
• A neural network walks into a lab: towards using deep nets as models for human behavior
• Exploring Human-like Attention Supervision in Visual Question Answering
• Advancing NLP with Cognitive Language Processing Signals
• Learning Semantic Sentence Embeddings using Sequential Pair-wise Discriminator
• GazeDPM: Early Integration of Gaze Information in Deformable Part Models

Cited by in corpus (7)

• Multimodal Integration of Human-Like Attention in Visual Question Answering
• EyeTrans: Merging Human and Machine Attention for Neural Code Summarization
• VQA-MHUG: A Gaze Dataset to Study Multimodal Neural Attention in Visual Question Answering
• Do Machines and Humans Focus on Similar Code? Exploring Explainability of Large Language Models in Code Summarization
• Perturbation-based Self-supervised Attention for Attention Bias in Text Classification
• Does Attention Mechanism Possess the Feature of Human Reading? A Perspective of Sentiment Classification Task
• CogAlign: Learning to Align Textual Neural Representations to Cognitive Language Processing Signals