Joint Discovery of Object States and Manipulation Actions
arXiv:1702.02738
Abstract
Many human activities involve object manipulations aiming to modify the object state. Examples of common state changes include full/empty bottle, open/closed door, and attached/detached car wheel. In this work, we seek to automatically discover the states of objects and the associated manipulation actions. Given a set of videos for a particular task, we propose a joint model that learns to identify object states and to localize state-modifying actions. Our model is formulated as a discriminative clustering cost with constraints. We assume a consistent temporal order for the changes in object states and manipulation actions, and introduce new optimization techniques to learn model parameters without additional supervision. We demonstrate successful discovery of seven manipulation actions and corresponding object states on a new dataset of videos depicting real-life object manipulations. We show that our joint formulation results in an improvement of object state discovery by action recognition and vice versa.
arXiv commentAppears in: International Conference on Computer Vision 2017 (ICCV 2017). 15 pages
References in corpus (12)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Two-Stream Convolutional Networks for Action Recognition in Videos
- Deep Residual Learning for Image Recognition
- Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding
- Learning Spatiotemporal Features with 3D Convolutional Networks
- On the Global Linear Convergence of Frank-Wolfe Optimization Variants
- Convergence Rate of Frank-Wolfe for Non-Convex Objectives
- Weakly Supervised Action Labeling in Videos Under Ordering Constraints
- Connectionist Temporal Modeling for Weakly Supervised Action Labeling
- Weakly-Supervised Alignment of Video With Text
- Unsupervised Semantic Parsing of Video Collections
- Actions ~ Transformations