Discrete Autoencoders for Sequence Models

Abstract

Recurrent models for sequences have been recently successful at many tasks, especially for language modeling and machine translation. Nevertheless, it remains challenging to extract good representations from these models. For instance, even though language has a clear hierarchical structure going from characters through words to sentences, it is not apparent in current language models. We propose to improve the representation in sequence models by augmenting current approaches with an autoencoder that is forced to compress the sequence through an intermediate discrete latent space. In order to propagate gradients though this discrete representation we introduce an improved semantic hashing technique. We show that this technique performs well on a newly proposed quantitative efficiency measure. We also analyze latent codes produced by the model showing how they correspond to words and phrases. Finally, we present an application of the autoencoder-augmented model to generating diverse translations.

Cited by in corpus (12)

• Model-Based Reinforcement Learning for Atari
• Fast Decoding in Sequence Models using Discrete Latent Variables
• Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned
• Theory and Experiments on Vector Quantized Autoencoders
• Dynamic Neural Networks: A Survey
• Towards More Human-like AI Communication: A Review of Emergent Communication Research
• Encoding Musical Style with Transformer Autoencoders
• You Look Twice: GaterNet for Dynamic Filter Selection in CNNs
• Catalytic Role Of Noise And Necessity Of Inductive Biases In The Emergence Of Compositional Communication
• Extractive Summary as Discrete Latent Variables
• Self-Adaptive Network Pruning
• Entropy optimized semi-supervised decomposed vector-quantized variational autoencoder model based on transfer learning for multiclass text classification and generation