Generating Videos of Zero-Shot Compositions of Actions and Objects
arXiv:1912.02401
Abstract
Human activity videos involve rich, varied interactions between people and objects. In this paper we develop methods for generating such videos -- making progress toward addressing the important, open problem of video generation in complex scenes. In particular, we introduce the task of generating human-object interaction videos in a zero-shot compositional setting, i.e., generating videos for action-object compositions that are unseen during training, having seen the target action and target object separately. This setting is particularly important for generalization in human activity video generation, obviating the need to observe every possible action-object combination in training and thus avoiding the combinatorial explosion involved in modeling complex scenes. To generate human-object interaction videos, we propose a novel adversarial framework HOI-GAN which includes multiple discriminators focusing on different aspects of a video. To demonstrate the effectiveness of our proposed framework, we perform extensive quantitative and qualitative evaluation on two challenging datasets: EPIC-Kitchens and 20BN-Something-Something v2.
arXiv commentAccepted at ECCV'20; Project Page: https://www.sfu.ca/~mnawhal/projects/zs_hoi_generation.html
References in corpus (12)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- Large Scale GAN Training for High Fidelity Natural Image Synthesis
- Spectral Normalization for Generative Adversarial Networks
- Generative Adversarial Text to Image Synthesis
- Conditional Image Synthesis With Auxiliary Classifier GANs
- Conditional Image Generation with PixelCNN Decoders
- Unsupervised Learning of Video Representations using LSTMs
- Energy-based Generative Adversarial Network
- Decomposing Motion and Content for Natural Video Sequence Prediction
- Video Inpainting of Complex Scenes
- Pixels to Graphs by Associative Embedding
- Detecting and Recognizing Human-Object Interactions