31 citations · 163 across the 28 of their papers we have counts for
28 papers
MSAVBench: Towards Comprehensive and Reliable Evaluation of Multi-Shot Audio-Video Generation
Yujie Wei, Yujin Han, Zhekai Chen +20
Video generation is rapidly evolving from single-shot synthesis to complex multi-shot audio-video (MSAV) narratives to meet real-world demands. However, evaluating such frontier mo…
Accelerating Pre-training of Multimodal LLMs via Chain-of-Sight
Ziyuan Huang, Kaixiang Ji, Biao Gong +6
This paper introduces Chain-of-Sight, a vision-language bridge module that accelerates the pre-training of Multimodal Large Language Models (MLLMs). Our approach employs a sequence…
A Recipe for Scaling up Text-to-Video Generation with Text-free Videos
Xiang Wang, Shiwei Zhang, Hangjie Yuan +6
Diffusion-based text-to-video generation has witnessed impressive progress in the past year yet still falls behind text-to-image generation. One of the key reasons is the limited s…
Hierarchical Spatio-temporal Decoupling for Text-to-Video Generation
Zhiwu Qing, Shiwei Zhang, Jiayu Wang +5
Despite diffusion models having shown powerful abilities to generate photorealistic images, generating videos that are realistic and diverse still remains in its infancy. One of th…
DreamVideo: Composing Your Dream Videos with Customized Subject and Motion
Yujie Wei, Shiwei Zhang, Zhiwu Qing +6
Customized generation using diffusion models has made impressive progress in image generation, but remains unsatisfactory in the challenging video generation task, as it requires t…
Disentangling Spatial and Temporal Learning for Efficient Image-to-Video Transfer Learning
Zhiwu Qing, Shiwei Zhang, Ziyuan Huang +4
Recently, large-scale pre-trained language-image models like CLIP have shown extraordinary capabilities for understanding spatial contents, but naively transferring such models to…
Temporally-Adaptive Models for Efficient Video Understanding
Ziyuan Huang, Shiwei Zhang, Liang Pan +4
Spatial convolutions are extensively used in numerous deep video models. It fundamentally assumes spatio-temporal invariance, i.e., using shared weights for every location in diffe…
HR-Pro: Point-supervised Temporal Action Localization via Hierarchical Reliability Propagation
Huaxin Zhang, Xiang Wang, Xiaohao Xu +3
Point-supervised Temporal Action Localization (PSTAL) is an emerging research direction for label-efficient learning. However, current methods mainly focus on optimizing the networ…
MoLo: Motion-augmented Long-short Contrastive Learning for Few-shot Action Recognition
Xiang Wang, Shiwei Zhang, Zhiwu Qing +4
Current state-of-the-art approaches for few-shot action recognition achieve promising performance by conducting frame-level matching on learned visual features. However, they gener…
Enlarging Instance-specific and Class-specific Information for Open-set Action Recognition
Jun Cen, Shiwei Zhang, Xiang Wang +4
Open-set action recognition is to reject unknown human action cases which are out of the distribution of the training set. Existing methods mainly focus on learning better uncertai…
HyRSM++: Hybrid Relation Guided Temporal Set Matching for Few-shot Action Recognition
Xiang Wang, Shiwei Zhang, Zhiwu Qing +4
Recent attempts mainly focus on learning deep representations for each video individually under the episodic meta-learning regime and then performing temporal alignment to match qu…
Learning a Condensed Frame for Memory-Efficient Video Class-Incremental Learning
Yixuan Pei, Zhiwu Qing, Jun Cen +6
Recent incremental learning for action recognition usually stores representative videos to mitigate catastrophic forgetting. However, only a few bulky videos can be stored due to t…
MAR: Masked Autoencoders for Efficient Action Recognition
Zhiwu Qing, Shiwei Zhang, Ziyuan Huang +5
Standard approaches for video recognition usually operate on the full input videos, which is inefficient due to the widely present spatio-temporal redundancy in videos. Recent prog…
Hybrid Relation Guided Set Matching for Few-shot Action Recognition
Xiang Wang, Shiwei Zhang, Zhiwu Qing +5
Current few-shot action recognition methods reach impressive performance by learning discriminative features for each video via episodic training and designing various temporal ali…
Learning from Untrimmed Videos: Self-Supervised Video Representation Learning with Hierarchical Consistency
Zhiwu Qing, Shiwei Zhang, Ziyuan Huang +6
Natural videos provide rich visual contents for self-supervised learning. Yet most existing approaches for learning spatio-temporal representations rely on manually trimmed videos,…
TAda! Temporally-Adaptive Convolutions for Video Understanding
Ziyuan Huang, Shiwei Zhang, Liang Pan +4
Spatial convolutions are widely used in numerous deep video models. It fundamentally assumes spatio-temporal invariance, i.e., using shared weights for every location in different…
ParamCrop: Parametric Cubic Cropping for Video Contrastive Learning
Zhiwu Qing, Ziyuan Huang, Shiwei Zhang +5
The central idea of contrastive learning is to discriminate between different instances and force different views from the same instance to share the same representation. To avoid…
Exploring Stronger Feature for Temporal Action Localization
Zhiwu Qing, Xiang Wang, Ziyuan Huang +6
Temporal action localization aims to localize starting and ending time with action category. Limited by GPU memory, mainstream methods pre-extract features for each video. Therefor…
OadTR: Online Action Detection with Transformers
Xiang Wang, Shiwei Zhang, Zhiwu Qing +4
Most recent approaches for online action detection tend to apply Recurrent Neural Network (RNN) to capture long-range temporal structure. However, RNN suffers from non-parallelism…
Proposal Relation Network for Temporal Action Detection
Xiang Wang, Zhiwu Qing, Ziyuan Huang +6
This technical report presents our solution for temporal action detection task in AcitivityNet Challenge 2021. The purpose of this task is to locate and identify actions of interes…
Weakly-Supervised Temporal Action Localization Through Local-Global Background Modeling
Xiang Wang, Zhiwu Qing, Ziyuan Huang +6
Weakly-Supervised Temporal Action Localization (WS-TAL) task aims to recognize and localize temporal starts and ends of action instances in an untrimmed video with only video-level…
Relation Modeling in Spatio-Temporal Action Localization
Yutong Feng, Jianwen Jiang, Ziyuan Huang +5
This paper presents our solution to the AVA-Kinetics Crossover Challenge of ActivityNet workshop at CVPR 2021. Our solution utilizes multiple types of relation modeling methods for…
A Stronger Baseline for Ego-Centric Action Detection
Zhiwu Qing, Ziyuan Huang, Xiang Wang +7
This technical report analyzes an egocentric video action detection method we used in the 2021 EPIC-KITCHENS-100 competition hosted in CVPR2021 Workshop. The goal of our task is to…
Towards Training Stronger Video Vision Transformers for EPIC-KITCHENS-100 Action Recognition
Ziyuan Huang, Zhiwu Qing, Xiang Wang +7
With the recent surge in the research of vision transformers, they have demonstrated remarkable potential for various challenging computer vision applications, such as image recogn…