315 citations · 481 across the 21 of their papers we have counts for
21 papers
Learning Referring Video Object Segmentation from Weak Annotation
Wangbo Zhao, Kepan Nan, Songyang Zhang +3
Referring video object segmentation (RVOS) is a task that aims to segment the target object in all video frames based on a sentence describing the object. Although existing RVOS me…
Unveiling Cross Modality Bias in Visual Question Answering: A Causal View with Possible Worlds VQA
Ali Vosoughi, Shijian Deng, Songyang Zhang +3
To increase the generalization capability of VQA systems, many recent studies have tried to de-bias spurious language or vision associations that shortcut the question or image to…
TG-VQA: Ternary Game of Video Question Answering
Hao Li, Peng Jin, Zesen Cheng +5
Video question answering aims at answering a question about the video content by reasoning the alignment semantics within them. However, since relying heavily on human instructions…
Latent-Shift: Latent Diffusion with Temporal Shift for Efficient Text-to-Video Generation
Jie An, Songyang Zhang, Harry Yang +4
We propose Latent-Shift -- an efficient text-to-video generation method based on a pretrained text-to-image generation model that consists of an autoencoder and a U-Net diffusion m…
Learning a Grammar Inducer from Massive Uncurated Instructional Videos
Songyang Zhang, Linfeng Song, Lifeng Jin +4
Video-aided grammar induction aims to leverage video information for finding more accurate syntactic grammars for accompanying text. While previous work focuses on building systems…
Make-A-Video: Text-to-Video Generation without Text-Video Data
Uriel Singer, Adam Polyak, Thomas Hayes +10
We propose Make-A-Video -- an approach for directly translating the tremendous recent progress in Text-to-Image (T2I) generation to Text-to-Video (T2V). Our intuition is simple: le…
Expanding Language-Image Pretrained Models for General Video Recognition
Bolin Ni, Houwen Peng, Minghao Chen +5
Contrastive language-image pretraining has shown great success in learning visual-textual joint representation from web-scale data, demonstrating remarkable "zero-shot" generalizat…
Rethinking the Evaluation of Unbiased Scene Graph Generation
Xingchen Li, Long Chen, Jian Shao +3
Current Scene Graph Generation (SGG) methods tend to predict frequent predicate categories and fail to recognize rare ones due to the severe imbalanced distribution of predicates.…
The Devil is in the Labels: Noisy Label Correction for Robust Scene Graph Generation
Lin Li, Long Chen, Yifeng Huang +3
Unbiased SGG has achieved significant progress over recent years. However, almost all existing SGG models have overlooked the ground-truth annotation qualities of prevailing SGG da…
MUGEN: A Playground for Video-Audio-Text Multimodal Understanding and GENeration
Thomas Hayes, Songyang Zhang, Xi Yin +6
Multimodal video-audio-text understanding and generation can benefit from datasets that are narrow but rich. The narrowness allows bite-sized challenges that the research community…
Instance-wise or Class-wise? A Tale of Neighbor Shapley for Concept-based Explanation
Jiahui Li, Kun Kuang, Lin Li +4
Deep neural networks have demonstrated remarkable performance in many data-driven and prediction-oriented applications, and sometimes even perform better than humans. However, thei…
Video-aided Unsupervised Grammar Induction
Songyang Zhang, Linfeng Song, Lifeng Jin +3
We investigate video-aided grammar induction, which learns a constituency parser from both unlabeled text and its corresponding video. Existing methods of multi-modal grammar induc…
SAT: 2D Semantics Assisted Training for 3D Visual Grounding
Zhengyuan Yang, Songyang Zhang, Liwei Wang +1
3D visual grounding aims at grounding a natural language description about a 3D scene, usually represented in the form of 3D point clouds, to the targeted object region. Point clou…
Boundary Proposal Network for Two-Stage Natural Language Video Localization
Shaoning Xiao, Long Chen, Songyang Zhang +4
We aim to address the problem of Natural Language Video Localization (NLVL)-localizing the video segment corresponding to a natural language description in a long and untrimmed vid…
Multi-Scale 2D Temporal Adjacent Networks for Moment Localization with Natural Language
Songyang Zhang, Houwen Peng, Jianlong Fu +2
We address the problem of retrieving a specific moment from an untrimmed video by natural language. It is a challenging problem because a target moment may take place in the contex…
Content-based Analysis of the Cultural Differences between TikTok and Douyin
Li Sun, Haoqi Zhang, Songyang Zhang +1
Short-form video social media shifts away from the traditional media paradigm by telling the audience a dynamic story to attract their attention. In particular, different combinati…
Global Image Sentiment Transfer
Jie An, Tianlang Chen, Songyang Zhang +1
Transferring the sentiment of an image is an unexplored research topic in the area of computer vision. This work proposes a novel framework consisting of a reference image retrieva…
Mi YouTube es Su YouTube? Analyzing the Cultures using YouTube Thumbnails of Popular Videos
Songyang Zhang, Tolga Aktas, Jiebo Luo
YouTube, a world-famous video sharing website, maintains a list of the top trending videos on the platform. Due to its huge amount of users, it enables researchers to understand pe…
Learning Sparse 2D Temporal Adjacent Networks for Temporal Action Localization
Songyang Zhang, Houwen Peng, Le Yang +2
In this report, we introduce the Winner method for HACS Temporal Action Localization Challenge 2019. Temporal action localization is challenging since a target proposal may be rela…
Learning 2D Temporal Adjacent Networks for Moment Localization with Natural Language
Songyang Zhang, Houwen Peng, Jianlong Fu +1
We address the problem of retrieving a specific moment from an untrimmed video by a query sentence. This is a challenging problem because a target moment may take place in relation…
Exploiting Temporal Relationships in Video Moment Localization with Natural Language
Songyang Zhang, Jinsong Su, Jiebo Luo
We address the problem of video moment localization with natural language, i.e. localizing a video segment described by a natural language sentence. While most prior work focuses o…