Generative Modeling for Multi-task Visual Learning

Abstract

Generative modeling has recently shown great promise in computer vision, but it has mostly focused on synthesizing visually realistic images. In this paper, motivated by multi-task learning of shareable feature representations, we consider a novel problem of learning a shared generative model that is useful across various visual perception tasks. Correspondingly, we propose a general multi-task oriented generative modeling (MGM) framework, by coupling a discriminative multi-task network with a generative network. While it is challenging to synthesize both RGB images and pixel-level annotations in multi-task scenarios, our framework enables us to use synthesized images paired with only weak annotations (i.e., image-level scene labels) to facilitate multiple visual tasks. Experimental evaluation on challenging multi-task benchmarks, including NYUv2 and Taskonomy, demonstrates that our MGM framework improves the performance of all the tasks by large margins, consistently outperforming state-of-the-art multi-task approaches.

References in corpus (12)

• A Simple Framework for Contrastive Learning of Visual Representations
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Self-Attention Generative Adversarial Networks
• An Overview of Multi-Task Learning in Deep Neural Networks
• GAN-based Synthetic Medical Image Augmentation for increased CNN Performance in Liver Lesion Classification
• Multi-Task Learning as Multi-Objective Optimization
• Triple Generative Adversarial Nets
• Which Tasks Should Be Learned Together in Multi-task Learning?
• AdaShare: Learning What To Share For Efficient Deep Multi-Task Learning
• Low-Shot Learning from Imaginary Data
• Verisimilar Image Synthesis for Accurate Detection and Recognition of Texts in Scenes