Publications (29)
VReST: Enhancing Reasoning in Large Vision-Language Models through Tree Search and Self-Reward Mechanism
Congzhi Zhang, Jiawei Peng, Zhenglin Wang +5
Large Vision-Language Models (LVLMs) have shown exceptional performance in multimodal tasks, but their effectiveness in complex visual reasoning is still constrained, especially wh…
Solving the Catastrophic Forgetting Problem in Generalized Category Discovery
Xinzi Cao, Xiawu Zheng, Guanhong Wang +5
Generalized Category Discovery (GCD) aims to identify a mix of known and novel categories within unlabeled data sets, providing a more realistic setting for image recognition. Esse…
OnlineHOI: Towards Online Human-Object Interaction Generation and Perception
Yihong Ji, Yunze Liu, Yiyao Zhuo +4
The perception and generation of Human-Object Interaction (HOI) are crucial for fields such as robotics, AR/VR, and human behavior understanding. However, current approaches model…
Intensive Vision-guided Network for Radiology Report Generation
Fudan Zheng, Mengfei Li, Ying Wang +5
Automatic radiology report generation is booming due to its huge application potential for the healthcare industry. However, existing computer vision and natural language processin…
Learning Fine-Grained Grounded Citations for Attributed Large Language Models
Lei Huang, Xiaocheng Feng, Weitao Ma +8
Despite the impressive performance on information-seeking tasks, large language models (LLMs) still struggle with hallucinations. Attributed LLMs, which augment generated text with…
Hybrid Reasoning Network for Video-based Commonsense Captioning
Weijiang Yu, Jian Liang, Lei Ji +4
The task of video-based commonsense captioning aims to generate event-wise captions and meanwhile provide multiple commonsense descriptions (e.g., attribute, effect and intention)…
AdaNAS: Adaptively Post-processing with Self-supervised Neural Architecture Search for Ensemble Rainfall Forecasts
Yingpeng Wen, Weijiang Yu, Fudan Zheng +2
Previous post-processing studies on rainfall forecasts using numerical weather prediction (NWP) mainly focus on statistics-based aspects, while learning-based aspects are rarely in…
Graph-to-Frame RAG: Visual-Space Knowledge Fusion for Training-Free and Auditable Video Reasoning
Songyuan Yang, Weijiang Yu, Ziyu Liu +4
When video reasoning requires external knowledge, many systems with large multimodal models (LMMs) adopt retrieval augmentation to supply the missing context. Appending textual or…
Deep Animation Video Interpolation in the Wild
Li Siyao, Shiyu Zhao, Weijiang Yu +4
In the animation industry, cartoon videos are usually produced at low frame rate since hand drawing of such frames is costly and time-consuming. Therefore, it is desirable to devel…
Layout-Graph Reasoning for Fashion Landmark Detection
Weijiang Yu, Xiaodan Liang, Ke Gong +3
Detecting dense landmarks for diverse clothes, as a fundamental technique for clothes analysis, has attracted increasing research attention due to its huge application potential. H…
Bailando: 3D Dance Generation by Actor-Critic GPT with Choreographic Memory
Li Siyao, Weijiang Yu, Tianpei Gu +5
Driving 3D characters to dance following a piece of music is highly challenging due to the spatial constraints applied to poses by choreography norms. In addition, the generated da…
TruthLens: Object Hallucination Detection via Self-Evaluating Truthfulness Scores in LVLMs
Yanqi Wu, Runhe Lai, Xinhua Lu +5
Despite the remarkable progress of large vision language models (LVLMs), object hallucination remains a fundamental challenge that hinders their trustworthy deployment. A key findi…
PointTalk: Audio-Driven Dynamic Lip Point Cloud for 3D Gaussian-based Talking Head Synthesis
Yifan Xie, Tao Feng, Xin Zhang +6
Talking head synthesis with arbitrary speech audio is a crucial challenge in the field of digital humans. Recently, methods based on radiance fields have received increasing attent…
Trends in Integration of Knowledge and Large Language Models: A Survey and Taxonomy of Methods, Benchmarks, and Applications
Zhangyin Feng, Weitao Ma, Weijiang Yu +7
Large language models (LLMs) exhibit superior performance on various natural language tasks, but they are susceptible to issues stemming from outdated data and domain-specific limi…
BeamAggR: Beam Aggregation Reasoning over Multi-source Knowledge for Multi-hop Question Answering
Zheng Chu, Jingchang Chen, Qianglong Chen +6
Large language models (LLMs) have demonstrated strong reasoning capabilities. Nevertheless, they still suffer from factual errors when tackling knowledge-intensive tasks. Retrieval…
TimeBench: A Comprehensive Evaluation of Temporal Reasoning Abilities in Large Language Models
Zheng Chu, Jingchang Chen, Qianglong Chen +4
Grasping the concept of time is a fundamental facet of human cognition, indispensable for truly comprehending the intricacies of the world. Previous studies typically focus on spec…
Learning to Break: Knowledge-Enhanced Reasoning in Multi-Agent Debate System
Haotian Wang, Xiyuan Du, Weijiang Yu +5
Multi-agent debate system (MAD) imitating the process of human discussion in pursuit of truth, aims to align the correct cognition of different agents for the optimal solution. It…
Stable Diffusion Segmentation for Biomedical Images with Single-step Reverse Process
Tianyu Lin, Zhiguang Chen, Zhonghao Yan +2
Diffusion models have demonstrated their effectiveness across various generative tasks. However, when applied to medical image segmentation, these models encounter several challeng…
Instruction Lens Score: Your Instruction Contributes a Powerful Object Hallucination Detector for Multimodal Large Language Models
Runhe Lai, Xinhua Lu, Yanqi Wu +3
Multimodal large language models (MLLMs) have achieved remarkable progress, yet the object hallucination remains a critical challenge for reliable deployment. In this paper, we pre…
Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Future
Zheng Chu, Jingchang Chen, Qianglong Chen +7
Reasoning, a fundamental cognitive process integral to human intelligence, has garnered substantial interest within artificial intelligence. Notably, recent studies have revealed t…
Heterogeneous Graph Learning for Visual Commonsense Reasoning
Weijiang Yu, Jingwen Zhou, Weihao Yu +2
Visual commonsense reasoning task aims at leading the research field into solving cognition-level reasoning with the ability of predicting correct answers and meanwhile providing c…
From Specific-MLLMs to Omni-MLLMs: A Survey on MLLMs Aligned with Multi-modalities
Shixin Jiang, Jiafeng Liang, Jiyuan Wang +6
To tackle complex tasks in real-world scenarios, more researchers are focusing on Omni-MLLMs, which aim to achieve omni-modal understanding and generation. Beyond the constraints o…
A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions
Lei Huang, Weijiang Yu, Weitao Ma +8
The emergence of large language models (LLMs) has marked a significant breakthrough in natural language processing (NLP), fueling a paradigm shift in information acquisition. Never…
An Information Bottleneck Perspective for Effective Noise Filtering on Retrieval-Augmented Generation
Kun Zhu, Xiaocheng Feng, Xiyuan Du +7
Retrieval-augmented generation integrates the capabilities of large language models with relevant information retrieved from an extensive corpus, yet encounters challenges when con…
MuseFace: Text-driven Face Editing via Diffusion-based Mask Generation Approach
Xin Zhang, Siting Huang, Xiangyang Luo +5
Face editing modifies the appearance of face, which plays a key role in customization and enhancement of personal images. Although much work have achieved remarkable success in tex…
Exploring Low-Resource Medical Image Classification with Weakly Supervised Prompt Learning
Fudan Zheng, Jindong Cao, Weijiang Yu +3
Most advances in medical image recognition supporting clinical auxiliary diagnosis meet challenges due to the low-resource situation in the medical field, where annotations are hig…
Domain-Adaptive Text Classification with Structured Knowledge from Unlabeled Data
Tian Li, Xiang Chen, Zhen Dong +4
Domain adaptive text classification is a challenging problem for the large-scale pretrained language models because they often require expensive additional labeled data to adapt to…
Gradual Network for Single Image De-raining
Zhe Huang, Weijiang Yu, Wayne Zhang +2
Most advances in single image de-raining meet a key challenge, which is removing rain streaks with different scales and shapes while preserving image details. Existing single image…
Reinforce to Learn, Elect to Reason: A Dual Paradigm for Video Reasoning
Songyuan Yang, Weijiang Yu, Jilin Ma +5
Video reasoning has advanced with large multimodal models (LMMs), yet their inference is often a single pass that returns an answer without verifying whether the reasoning is evide…