Bowtie Networks: Generative Modeling for Joint Few-Shot Recognition and Novel-View Synthesis

Abstract

We propose a novel task of joint few-shot recognition and novel-view synthesis: given only one or few images of a novel object from arbitrary views with only category annotation, we aim to simultaneously learn an object classifier and generate images of that type of object from new viewpoints. While existing work copes with two or more tasks mainly by multi-task learning of shareable feature representations, we take a different perspective. We focus on the interaction and cooperation between a generative model and a discriminative model, in a way that facilitates knowledge to flow across tasks in complementary directions. To this end, we propose bowtie networks that jointly learn 3D geometric and semantic representations with a feedback loop. Experimental evaluation on challenging fine-grained recognition datasets demonstrates that our synthesized images are realistic from multiple viewpoints and significantly improve recognition performance as ways of data augmentation, especially in the low-data regime. Code and pre-trained models are released at https://github.com/zpbao/bowtie_networks.

arXiv commentAccepted as a Poster paper at ICLR 2021

References in corpus (14)

• Distilling the Knowledge in a Neural Network
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Prototypical Networks for Few-shot Learning
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Instance Normalization: The Missing Ingredient for Fast Stylization
• An Overview of Multi-Task Learning in Deep Neural Networks
• Generative Moment Matching Networks
• Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
• Delta-encoder: an effective sample synthesis method for few-shot object recognition
• Which Tasks Should Be Learned Together in Multi-task Learning?
• BlockGAN: Learning 3D Object-aware Scene Representations from Unlabelled Images
• Image Synthesis with a Single (Robust) Classifier
• Learning Data Manipulation for Augmentation and Weighting
• Meta-Reinforced Synthetic Data for One-Shot Fine-Grained Visual Recognition