Learning to Extract Motion from Videos in Convolutional Neural Networks

Abstract

This paper shows how to extract dense optical flow from videos with a convolutional neural network (CNN). The proposed model constitutes a potential building block for deeper architectures to allow using motion without resorting to an external algorithm, \eg for recognition in videos. We derive our network architecture from signal processing principles to provide desired invariances to image contrast, phase and texture. We constrain weights within the network to enforce strict rotation invariance and substantially reduce the number of parameters to learn. We demonstrate end-to-end training on only 8 sequences of the Middlebury dataset, orders of magnitude less than competing CNN-based motion estimation methods, and obtain comparable performance to classical methods on the Middlebury benchmark. Importantly, our method outputs a distributed representation of motion that allows representing multiple, transparent motions, and dynamic textures. Our contributions on network design and rotation invariance offer insights nonspecific to motion estimation.

References in corpus (2)

• Two-Stream Convolutional Networks for Action Recognition in Videos
• Unsupervised learning of depth and motion

Cited by in corpus (8)

• PixelNet: Representation of the pixels, by the pixels, and for the pixels
• Video Frame Interpolation via Adaptive Separable Convolution
• Video Frame Interpolation via Adaptive Convolution
• Hybrid Learning of Optical Flow and Next Frame Prediction to Boost Optical Flow in the Wild
• Deep Rotation Equivariant Network
• Non-Parametric Transformation Networks
• Learn to Model Motion from Blurry Footages
• Revisiting Optical Flow Estimation in 360 Videos