On the impressive performance of randomly weighted encoders in summarization tasks

Abstract

In this work, we investigate the performance of untrained randomly initialized encoders in a general class of sequence to sequence models and compare their performance with that of fully-trained encoders on the task of abstractive summarization. We hypothesize that random projections of an input text have enough representational power to encode the hierarchical structure of sentences and semantics of documents. Using a trained decoder to produce abstractive text summaries, we empirically demonstrate that architectures with untrained randomly initialized encoders perform competitively with respect to the equivalent architectures with fully-trained encoders. We further find that the capacity of the encoder not only improves overall model generalization but also closes the performance gap between untrained randomly initialized and full-trained encoders. To our knowledge, it is the first time that general sequence to sequence models with attention are assessed for trained and randomly projected representations on abstractive summarization.

arXiv commentAccepted to ACL 2019 SRW. First two authors contributed equally

References in corpus (6)

• Teaching Machines to Read and Comprehend
• Recent Trends in Deep Learning Based Natural Language Processing
• Skip-Thought Vectors
• Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond
• SentEval: An Evaluation Toolkit for Universal Sentence Representations
• No Training Required: Exploring Random Encoders for Sentence Classification