Disjoint Multi-task Learning between Heterogeneous Human-centric Tasks

Abstract

Human behavior understanding is arguably one of the most important mid-level components in artificial intelligence. In order to efficiently make use of data, multi-task learning has been studied in diverse computer vision tasks including human behavior understanding. However, multi-task learning relies on task specific datasets and constructing such datasets can be cumbersome. It requires huge amounts of data, labeling efforts, statistical consideration etc. In this paper, we leverage existing single-task datasets for human action classification and captioning data for efficient human behavior learning. Since the data in each dataset has respective heterogeneous annotations, traditional multi-task learning is not effective in this scenario. To this end, we propose a novel alternating directional optimization method to efficiently learn from the heterogeneous data. We demonstrate the effectiveness of our model and show performance improvements on both classification and sentence retrieval tasks in comparison to the models trained on each of the single-task datasets.

arXiv comment10 pages, 7 figures

References in corpus (15)

• Adam: A Method for Stochastic Optimization
• Overcoming catastrophic forgetting in neural networks
• Two-Stream Convolutional Networks for Action Recognition in Videos
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• Towards Good Practices for Very Deep Two-Stream ConvNets
• Convolutional Pose Machines
• Continual Learning with Deep Generative Replay
• Show and Tell: A Neural Image Caption Generator
• Learning Spatiotemporal Features with 3D Convolutional Networks
• Universal representations:The missing link between faces, text, planktons, and cat breeds
• Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields
• Order-Embeddings of Images and Language
• Visual Madlibs: Fill in the blank Image Generation and Question Answering
• What value do explicit high level concepts have in vision to language problems?
• Expert Gate: Lifelong Learning with a Network of Experts

Cited by in corpus (2)

• Image Captioning with Very Scarce Supervised Data: Adversarial Semi-Supervised Learning Approach
• Learning joint segmentation of tissues and brain lesions from task-specific hetero-modal domain-shifted datasets