Sentence Bottleneck Autoencoders from Transformer Language Models

Ivan Montero, Nikolaos Pappas, Noah A. Smith

Representation learning for text via pretraining a language model on a large corpus has become a standard starting point for building NLP systems. This approach stands in contrast…