NeRV: Neural Representations for Videos

Abstract

We propose a novel neural representation for videos (NeRV) which encodes videos in neural networks. Unlike conventional representations that treat videos as frame sequences, we represent videos as neural networks taking frame index as input. Given a frame index, NeRV outputs the corresponding RGB image. Video encoding in NeRV is simply fitting a neural network to video frames and decoding process is a simple feedforward operation. As an image-wise implicit representation, NeRV output the whole image and shows great efficiency compared to pixel-wise implicit representation, improving the encoding speed by 25x to 70x, the decoding speed by 38x to 132x, while achieving better video quality. With such a representation, we can treat videos as neural networks, simplifying several video-related tasks. For example, conventional video compression methods are restricted by a long and complex pipeline, specifically designed for the task. In contrast, with NeRV, we can use any neural network compression method as a proxy for video compression, and achieve comparable performance to traditional frame-based video compression approaches (H.264, HEVC \etc). Besides compression, we demonstrate the generalization of NeRV for video denoising. The source code and pre-trained model can be found at https://github.com/haochen-rye/NeRV.git.

To appear at NeurIPS 2021

References in corpus (19)

• Adam: A Method for Stochastic Optimization
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Distilling the Knowledge in a Neural Network
• Gaussian Error Linear Units (GELUs)
• SGDR: Stochastic Gradient Descent with Warm Restarts
• A guide to convolution arithmetic for deep learning
• Quantizing deep convolutional networks for efficient inference: A whitepaper
• Deep Image Prior
• Group Equivariant Convolutional Networks
• Speeding up Convolutional Neural Networks with Low Rank Expansions
• Learning Structured Sparsity in Deep Neural Networks
• Model compression via distillation and quantization
• Scalable Methods for 8-bit Training of Neural Networks
• Exploiting Cyclic Symmetry in Convolutional Neural Networks
• On the Spectral Bias of Neural Networks
• Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations
• Adaptive Gradient Quantization for Data-Parallel SGD
• Convolutional Occupancy Networks
• Hierarchical Autoregressive Modeling for Neural Video Compression