Few-shot Autoregressive Density Estimation: Towards Learning to Learn Distributions

Abstract

Deep autoregressive models have shown state-of-the-art performance in density estimation for natural images on large-scale datasets such as ImageNet. However, such models require many thousands of gradient-based weight updates and unique image examples for training. Ideally, the models would rapidly learn visual concepts from only a handful of examples, similar to the manner in which humans learns across many vision tasks. In this paper, we show how 1) neural attention and 2) meta learning techniques can be used in combination with autoregressive models to enable effective few-shot density estimation. Our proposed modifications to PixelCNN result in state-of-the art few-shot density estimation on the Omniglot dataset. Furthermore, we visualize the learned attention policy and find that it learns intuitive algorithms for simple tasks such as image mirroring on ImageNet and handwriting on Omniglot without supervision. Finally, we extend the model to natural images and demonstrate few-shot image generation on the Stanford Online Products dataset.

References in corpus (6)

• Convolutional Sequence to Sequence Learning
• Theoretical Models of Learning to Learn
• One-Shot Visual Imitation Learning via Meta-Learning
• One-Shot Imitation Learning
• Apprenticeship Learning using Inverse Reinforcement Learning and Gradient Methods
• Attentive Recurrent Comparators

Cited by in corpus (19)

• Language Models are Few-Shot Learners
• Neural Voice Cloning with a Few Samples
• Conditional Neural Processes
• The Variational Homoencoder: Learning to learn high capacity generative models from few examples
• Attentive Neural Processes
• Meta-Learning Deep Energy-Based Memory Models
• Learning models for visual 3D localization with implicit mapping
• A Broader Study of Cross-Domain Few-Shot Learning
• Few-shot Learning for Time-series Forecasting
• Meta-Learning surrogate models for sequential decision making
• Encoding Musical Style with Transformer Autoencoders
• Meta-Amortized Variational Inference and Learning
• LaSO: Label-Set Operations networks for multi-label few-shot learning
• Compositional Visual Generation and Inference with Energy Based Models
• SketchEmbedNet: Learning Novel Concepts by Imitating Drawings
• Linguistically-Enriched and Context-Aware Zero-shot Slot Filling
• Few-shot Learning for Spatial Regression
• Meta-learning One-class Classifiers with Eigenvalue Solvers for Supervised Anomaly Detection
• Few-Shot Semantic Parsing for New Predicates