Compositional Video Synthesis with Action Graphs
arXiv:2006.15327
Abstract
Videos of actions are complex signals containing rich compositional structure in space and time. Current video generation methods lack the ability to condition the generation on multiple coordinated and potentially simultaneous timed actions. To address this challenge, we propose to represent the actions in a graph structure called Action Graph and present the new ``Action Graph To Video'' synthesis task. Our generative model for this task (AG2Vid) disentangles motion and appearance features, and by incorporating a scheduling mechanism for actions facilitates a timely and coordinated video generation. We train and evaluate AG2Vid on the CATER and Something-Something V2 datasets, and show that the resulting videos have better visual quality and semantic consistency compared to baselines. Finally, our model demonstrates zero-shot abilities by synthesizing novel compositions of the learned actions. For code and pretrained models, see the project page https://roeiherz.github.io/AG2Video
arXiv commentICML 2021 Camera Ready
References in corpus (15)
- Adam: A Method for Stochastic Optimization
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Semi-Supervised Classification with Graph Convolutional Networks
- Large Scale GAN Training for High Fidelity Natural Image Synthesis
- The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
- Interaction Networks for Learning about Objects, Relations and Physics
- Stochastic Video Generation with a Learned Prior
- Neural Relational Inference for Interacting Systems
- Stochastic Adversarial Video Prediction
- Pixels to Graphs by Associative Embedding
- Video-to-Video Synthesis
- Mapping Images to Scene Graphs with Permutation-Invariant Structured Prediction
- CLEVRER: CoLlision Events for Video REpresentation and Reasoning
- Talking Face Generation by Conditional Recurrent Adversarial Network
- Action Genome: Actions as Composition of Spatio-temporal Scene Graphs