NewsEmbed: Modeling News through Pre-trained Document Representations
arXiv:2106.00590 · doi:10.1145/3447548.3467392
Abstract
Effectively modeling text-rich fresh content such as news articles at document-level is a challenging problem. To ensure a content-based model generalize well to a broad range of applications, it is critical to have a training dataset that is large beyond the scale of human labels while achieving desired quality. In this work, we address those two challenges by proposing a novel approach to mine semantically-relevant fresh documents, and their topic labels, with little human supervision. Meanwhile, we design a multitask model called NewsEmbed that alternatively trains a contrastive learning with a multi-label classification to derive a universal document encoder. We show that the proposed approach can provide billions of high quality organic training examples and can be naturally extended to multilingual setting where texts in different languages are encoded in the same semantic space. We experimentally demonstrate NewsEmbed's competitive performance across multiple natural language understanding tasks, both supervised and unsupervised.
Accepted in SIGKDD 2021
References in corpus (21)
- Distributed Representations of Words and Phrases and their Compositionality
- A Simple Framework for Contrastive Learning of Visual Representations
- Natural Scales in Geographical Patterns
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
- Bootstrap your own latent: A new approach to self-supervised Learning
- Character-level Convolutional Networks for Text Classification
- Language Models are Few-Shot Learners
- Cross-lingual Language Model Pretraining
- Text Classification Algorithms: A Survey
- Contrastive Representation Learning: A Framework and Review
- A C-LSTM Neural Network for Text Classification
- mT5: A massively multilingual pre-trained text-to-text transformer
- SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation
- Text Understanding from Scratch
- SentEval: An Evaluation Toolkit for Universal Sentence Representations
- NPA: Neural News Recommendation with Personalized Attention
- XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization
- A Survey on Natural Language Processing for Fake News Detection
- Pre-training Tasks for Embedding-based Large-scale Retrieval
- News Session-Based Recommendations using Deep Neural Networks
- Multilingual Clustering of Streaming News