109 citations · 265 across the 24 of their papers we have counts for
15 papers · 1 filter
Can MLLMs Reason Beyond Language? VisReason: A Comprehensive Benchmark for Vision-Centric Reasoning
Longteng Guo, Yifan Wang, Pengkang Huo +4
Recent multimodal large language models (MLLMs) achieve strong performance on visual reasoning benchmarks, yet it remains unclear to what extent such performance reflects reasoning…
AdaSpark: Adaptive Sparsity for Efficient Long-Video Understanding
Handong Li, Zikang Liu, Longteng Guo +10
Processing long-form videos with Video Large Language Models (Video-LLMs) is computationally prohibitive. Current efficiency methods often compromise fine-grained perception throug…
LaVi: Efficient Large Vision-Language Models via Internal Feature Modulation
Tongtian Yue, Longteng Guo, Yepeng Tang +4
Despite the impressive advancements of Large Vision-Language Models (LVLMs), existing approaches suffer from a fundamental bottleneck: inefficient visual-language integration. Curr…
COMUNI: Decomposing Common and Unique Video Signals for Diffusion-based Video Generation
Mingzhen Sun, Weining Wang, Xinxin Zhu +1
Since videos record objects moving coherently, adjacent video frames have commonness (similar object appearances) and uniqueness (slightly changed postures). To prevent redundant m…
MM-LDM: Multi-Modal Latent Diffusion Model for Sounding Video Generation
Mingzhen Sun, Weining Wang, Yanyuan Qiao +5
Sounding Video Generation (SVG) is an audio-video joint generation task challenged by high-dimensional signal spaces, distinct data formats, and different patterns of content infor…
ChatBridge: Bridging Modalities with Large Language Model as a Language Catalyst
Zijia Zhao, Longteng Guo, Tongtian Yue +5
Building general-purpose models that can perceive diverse real-world modalities and solve various tasks is an appealing target in artificial intelligence. In this paper, we present…
VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Dataset
Sihan Chen, Handong Li, Qunbo Wang +4
Vision and text have been fully explored in contemporary video-text foundational models, while other modalities such as audio and subtitles in videos have not received sufficient a…
Sounding Video Generator: A Unified Framework for Text-guided Sounding Video Generation
Jiawei Liu, Weining Wang, Sihan Chen +2
As a combination of visual and audio signals, video is inherently multi-modal. However, existing video generation methods are primarily intended for the synthesis of visual frames,…
MOSO: Decomposing MOtion, Scene and Object for Video Prediction
Mingzhen Sun, Weining Wang, Xinxin Zhu +1
Motion, scene and object are three primary visual components of a video. In particular, objects represent the foreground, scenes represent the background, and motion traces their d…
OPT: Omni-Perception Pre-Trainer for Cross-Modal Understanding and Generation
Jing Liu, Xinxin Zhu, Fei Liu +8
In this paper, we propose an Omni-perception Pre-Trainer (OPT) for cross-modal understanding and generation, by jointly modeling visual, text and audio resources. OPT is constructe…
CPTR: Full Transformer Network for Image Captioning
Wei Liu, Sihan Chen, Longteng Guo +2
In this paper, we consider the image captioning task from a new sequence-to-sequence prediction perspective and propose CaPtion TransformeR (CPTR) which takes the sequentialized ra…
Global-Local Propagation Network for RGB-D Semantic Segmentation
Sihan Chen, Xinxin Zhu, Wei Liu +2
Depth information matters in RGB-D semantic segmentation task for providing additional geometric information to color images. Most existing methods exploit a multi-stage fusion str…
AutoCaption: Image Captioning with Neural Architecture Search
Xinxin Zhu, Weining Wang, Longteng Guo +1
Image captioning transforms complex visual information into abstract natural language for representation, which can help computers understanding the world quickly. However, due to…
Normalized and Geometry-Aware Self-Attention Network for Image Captioning
Longteng Guo, Jing Liu, Xinxin Zhu +3
Self-attention (SA) network has shown profound value in image captioning. In this paper, we improve SA from two aspects to promote the performance of image captioning. First, we pr…
Vatex Video Captioning Challenge 2020: Multi-View Features and Hybrid Reward Strategies for Video Captioning
Xinxin Zhu, Longteng Guo, Peng Yao +3
This report describes our solution for the VATEX Captioning Challenge 2020, which requires generating descriptions for the videos in both English and Chinese languages. We identifi…