81 citations · 152 across the 10 of their papers we have counts for
9 papers · 1 filter
ChatBridge: Bridging Modalities with Large Language Model as a Language Catalyst
Zijia Zhao, Longteng Guo, Tongtian Yue +5
Building general-purpose models that can perceive diverse real-world modalities and solve various tasks is an appealing target in artificial intelligence. In this paper, we present…
MOSO: Decomposing MOtion, Scene and Object for Video Prediction
Mingzhen Sun, Weining Wang, Xinxin Zhu +1
Motion, scene and object are three primary visual components of a video. In particular, objects represent the foreground, scenes represent the background, and motion traces their d…
MAMO: Masked Multimodal Modeling for Fine-Grained Vision-Language Representation Learning
Zijia Zhao, Longteng Guo, Xingjian He +3
Multimodal representation learning has shown promising improvements on various vision-language tasks. Most existing methods excel at building global-level alignment between vision…
Exploiting Spatial-Temporal Semantic Consistency for Video Scene Parsing
Xingjian He, Weining Wang, Zhiyong Xu +3
Compared with image scene parsing, video scene parsing introduces temporal information, which can effectively improve the consistency and accuracy of prediction. In this paper, we…
OPT: Omni-Perception Pre-Trainer for Cross-Modal Understanding and Generation
Jing Liu, Xinxin Zhu, Fei Liu +8
In this paper, we propose an Omni-perception Pre-Trainer (OPT) for cross-modal understanding and generation, by jointly modeling visual, text and audio resources. OPT is constructe…
Temporal Memory Attention for Video Semantic Segmentation
Hao Wang, Weining Wang, Jing Liu
Video semantic segmentation requires to utilize the complex temporal relations between frames of the video sequence. Previous works usually exploit accurate optical flow to leverag…
Adaptive Context Network for Scene Parsing
Jun Fu, Jing Liu, Yuhang Wang +4
Recent works attempt to improve scene parsing performance by exploring different levels of contexts, and typically train a well-designed convolutional network to exploit useful con…
Vatex Video Captioning Challenge 2020: Multi-View Features and Hybrid Reward Strategies for Video Captioning
Xinxin Zhu, Longteng Guo, Peng Yao +3
This report describes our solution for the VATEX Captioning Challenge 2020, which requires generating descriptions for the videos in both English and Chinese languages. We identifi…
Aligning Linguistic Words and Visual Semantic Units for Image Captioning
Longteng Guo, Jing Liu, Jinhui Tang +3
Image captioning attempts to generate a sentence composed of several linguistic words, which are used to describe objects, attributes, and interactions in an image, denoted as visu…