HyperE2VID: Improving Event-Based Video Reconstruction via Hypernetworks
arXiv:2305.06382 · doi:10.1109/TIP.2024.3372460
Abstract
Event-based cameras are becoming increasingly popular for their ability to capture high-speed motion with low latency and high dynamic range. However, generating videos from events remains challenging due to the highly sparse and varying nature of event data. To address this, in this study, we propose HyperE2VID, a dynamic neural network architecture for event-based video reconstruction. Our approach uses hypernetworks to generate per-pixel adaptive filters guided by a context fusion module that combines information from event voxel grids and previously reconstructed intensity images. We also employ a curriculum learning strategy to train the network more robustly. Our comprehensive experimental evaluations across various benchmark datasets reveal that HyperE2VID not only surpasses current state-of-the-art methods in terms of reconstruction quality but also achieves this with fewer parameters, reduced computational requirements, and accelerated inference times.
arXiv comment20 pages, 11 figures. Accepted by IEEE Transactions on Image Processing. The project page can be found at https://ercanburak.github.io/HyperE2VID.html
References in corpus (9)
- Adam: A Method for Stochastic Optimization
- Event-based Vision: A Survey
- On the Convergence of Adam and Beyond
- The Event-Camera Dataset and Simulator: Event-based Data for Pose Estimation, Visual Odometry, and SLAM
- The Multi Vehicle Stereo Event Camera Dataset: An Event Camera Dataset for 3D Perception
- Event-based High Dynamic Range Image and Very High Frame Rate Video Generation using Conditional Generative Adversarial Networks
- Free as a Bird: Event-based Dynamic Sense-and-Avoid for Ornithopter Robot Flight
- Event-based Camera Pose Tracking using a Generative Event Model
- Formulating Event-based Image Reconstruction as a Linear Inverse Problem with Deep Regularization using Optical Flow