19 citations · 53 across the 23 of their papers we have counts for
23 papers
MM-Snowball: Evaluating and Mitigating Hallucination Snowballing in Multimodal Multi-Turn Dialogue
Yue Jiang, Xue Jiang, Lihua Zhang +6
Multimodal large language models (MLLMs) demonstrate remarkable visual understanding, yet their reliability in interactive settings is severely undermined by hallucination snowball…
LiveVLN: Breaking the Stop-and-Go Loop in Vision-Language Navigation
Xiangchen Wang, Weiye Zhu, Teng Wang +5
Recent navigation systems achieve strong benchmark results, yet real-world deployment often remains visibly stop-and-go. This bottleneck arises because the sense-inference-executio…
Learning Trajectory-Aware Multimodal Large Language Models for Video Reasoning Segmentation
Jingnan Luo, Mingqi Gao, Jun Liu +2
The prosperity of Multimodal Large Language Models (MLLMs) has stimulated the demand for video reasoning segmentation, which aims to segment video objects based on human instructio…
R-AVST: Empowering Video-LLMs with Fine-Grained Spatio-Temporal Reasoning in Complex Audio-Visual Scenarios
Lu Zhu, Tiantian Geng, Yangye Chen +3
Recently, rapid advancements have been made in multimodal large language models (MLLMs), especially in video understanding tasks. However, current research focuses on simple video…
Seeing More, Saying More: Lightweight Language Experts are Dynamic Video Token Compressors
Xiangchen Wang, Jinrui Zhang, Teng Wang +2
Recent advancements in large video-language models have revolutionized video understanding tasks. However, their efficiency is significantly constrained by processing high volumes…
LongVALE: Vision-Audio-Language-Event Benchmark Towards Time-Aware Omni-Modal Perception of Long Videos
Tiantian Geng, Jinrui Zhang, Qingni Wang +3
Despite impressive advancements in video understanding, most efforts remain limited to coarse-grained or visual-only video tasks. However, real-world videos encompass omni-modal in…
Sample then Identify: A General Framework for Risk Control and Assessment in Multimodal Large Language Models
Qingni Wang, Tiantian Geng, Zhiyuan Wang +3
Multimodal Large Language Models (MLLMs) exhibit promising advancements across various tasks, yet they still encounter significant trustworthiness issues. Prior studies apply Split…
Reflective Instruction Tuning: Mitigating Hallucinations in Large Vision-Language Models
Jinrui Zhang, Teng Wang, Haigang Zhang +2
Large vision-language models (LVLMs) have shown promising performance on a variety of vision-language tasks. However, they remain susceptible to hallucinations, generating outputs…
Two in One Go: Single-stage Emotion Recognition with Decoupled Subject-context Transformer
Xinpeng Li, Teng Wang, Jian Zhao +5
Emotion recognition aims to discern the emotional state of subjects within an image, relying on subject-centric and contextual visual cues. Current approaches typically follow a tw…
UniAV: Unified Audio-Visual Perception for Multi-Task Video Event Localization
Tiantian Geng, Teng Wang, Jinming Duan +4
Video event localization tasks include temporal action localization (TAL), sound event detection (SED) and audio-visual event localization (AVEL). Existing methods tend to over-spe…
Video Understanding with Large Language Models: A Survey
Yolo Y. Tang, Jing Bi, Siting Xu +17
With the burgeoning growth of online video platforms and the escalating volume of video content, the demand for proficient video understanding tools has intensified markedly. Given…
Knowledge-Aware Prompt Tuning for Generalizable Vision-Language Models
Baoshuo Kan, Teng Wang, Wenpeng Lu +3
Pre-trained vision-language models, e.g., CLIP, working with manually designed prompts have demonstrated great capacity of transfer learning. Recently, learnable prompts achieve st…
Transferable Decoding with Visual Entities for Zero-Shot Image Captioning
Junjie Fei, Teng Wang, Jinrui Zhang +3
Image-to-text generation aims to describe images using natural language. Recently, zero-shot image captioning based on pre-trained vision-language models (VLMs) and large language…
Set-level Guidance Attack: Boosting Adversarial Transferability of Vision-Language Pre-training Models
Dong Lu, Zhiqiang Wang, Teng Wang +3
Vision-language pre-training (VLP) models have shown vulnerability to adversarial examples in multimodal tasks. Furthermore, malicious adversaries can be deliberately transferred t…
LaunchpadGPT: Language Model as Music Visualization Designer on Launchpad
Siting Xu, Yolo Yunlong Tang, Feng Zheng
Launchpad is a musical instrument that allows users to create and perform music by pressing illuminated buttons. To assist and inspire the design of the Launchpad light effect, and…
LLMVA-GEBC: Large Language Model with Video Adapter for Generic Event Boundary Captioning
Yolo Yunlong Tang, Jinrui Zhang, Xiangchen Wang +2
Our winning entry for the CVPR 2023 Generic Event Boundary Captioning (GEBC) competition is detailed in this paper. Unlike conventional video captioning tasks, GEBC demands that th…
Caption Anything: Interactive Image Description with Diverse Multimodal Controls
Teng Wang, Jinrui Zhang, Junjie Fei +5
Controllable image captioning is an emerging multimodal topic that aims to describe the image with natural language following human purpose, , looking at the specifi…
Accelerating Vision-Language Pretraining with Free Language Modeling
Teng Wang, Yixiao Ge, Feng Zheng +4
The state of the arts in vision-language pretraining (VLP) achieves exemplary performance but suffers from high training costs resulting from slow convergence and long training tim…
Dense-Localizing Audio-Visual Events in Untrimmed Videos: A Large-Scale Benchmark and Baseline
Tiantian Geng, Teng Wang, Jinming Duan +2
Existing audio-visual event localization (AVE) handles manually trimmed videos with only a single instance in each of them. However, this setting is unrealistic as natural videos o…
Learning Grounded Vision-Language Representation for Versatile Understanding in Untrimmed Videos
Teng Wang, Jinrui Zhang, Feng Zheng +3
Joint video-language learning has received increasing attention in recent years. However, existing works mainly focus on single or multiple trimmed video clips (events), which make…
Multi-modal Segment Assemblage Network for Ad Video Editing with Importance-Coherence Reward
Yolo Yunlong Tang, Siting Xu, Teng Wang +3
Advertisement video editing aims to automatically edit advertising videos into shorter videos while retaining coherent content and crucial information conveyed by advertisers. It m…
Exploiting Context Information for Generic Event Boundary Captioning
Jinrui Zhang, Teng Wang, Feng Zheng +2
Generic Event Boundary Captioning (GEBC) aims to generate three sentences describing the status change for a given time boundary. Previous methods only process the information of a…
Semantic-Aware Pretraining for Dense Video Captioning
Teng Wang, Zhu Liu, Feng Zheng +3
This report describes the details of our approach for the event dense-captioning task in ActivityNet Challenge 2021. We present a semantic-aware pretraining method for dense video…