One-element Batch Training by Moving Window
arXiv:1905.12947
Abstract
Several deep models, esp. the generative, compare the samples from two distributions (e.g. WAE like AutoEncoder models, set-processing deep networks, etc) in their cost functions. Using all these methods one cannot train the model directly taking small size (in extreme -- one element) batches, due to the fact that samples are to be compared. We propose a generic approach to training such models using one-element mini-batches. The idea is based on splitting the batch in latent into parts: previous, i.e. historical, elements used for latent space distribution matching and the current ones, used both for latent distribution computation and the minimization process. Due to the smaller memory requirements, this allows to train networks on higher resolution images then in the classical approach.
References in corpus (12)
- Auto-Encoding Variational Bayes
- GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
- NICE: Non-linear Independent Components Estimation
- DRAW: A Recurrent Neural Network For Image Generation
- Autoencoding beyond pixels using a learned similarity metric
- Revisiting Small Batch Training for Deep Neural Networks
- Neural Photo Editing with Introspective Adversarial Networks
- Wasserstein Auto-Encoders
- Demystifying MMD GANs
- Sliced-Wasserstein Autoencoder: An Embarrassingly Simple Generative Model
- Cramer-Wold AutoEncoder
- Pioneer Networks: Progressively Growing Generative Autoencoder