76 citations · 117 across the 25 of their papers we have counts for
28 papers
FrameScout: Scouting Query-Relevant Frames for Long Video Understanding
Haonan Hu, Shuhao Chen, Weisen Jiang +3
Long video understanding requires VideoLLMs to answer queries about videos spanning tens of minutes to hours, but such long videos produce far more visual tokens than current Video…
VizRAG: Enhancing Retrieval-Augmented Generation with Hypergraph Visualization
Yanbin Wei, Yang Chen, Renling Gan +7
Hypergraph-based RAG systems surpass traditional graph-based approaches by organizing complex n-ary atomic facts among entities, rather than relying solely on binary relationships.…
SPARD: Defending Harmful Fine-Tuning Attack via Safety Projection with Relevance-Diversity Data Selection
Shuhao Chen, Weisen Jiang, Yeqi Gong +5
Fine-tuning large language models often undermines their safety alignment, a problem further amplified by harmful fine-tuning attacks in which adversarial data removes safeguards a…
RxEval: A Prescription-Level Benchmark for Evaluating LLM Medication Recommendation
Shuhao Chen, Weisen Jiang, Changmiao Wang +4
Inpatient medication recommendation requires clinicians to repeatedly select specific medications, doses, and routes as a patient's condition evolves. Existing benchmarks formulate…
Efficient Video Diffusion Models: Advancements and Challenges
Shitong Shao, Lichen Bai, Pengfei Wan +2
Video diffusion models have rapidly become the dominant paradigm for high-fidelity generative video synthesis, but their practical deployment remains constrained by severe inferenc…
HyperGVL: Benchmarking and Improving Large Vision-Language Models in Hypergraph Understanding and Reasoning
Yanbin Wei, Chun Kang, Siwei Li +11
Large Vision-Language Models (LVLMs) consistently require new arenas to guide their expanding boundaries, yet their capabilities with hypergraphs remain unexplored. In the real wor…
DynamicGTR: Leveraging Graph Topology Representation Preferences to Boost VLM Capabilities on Graph QAs
Yanbin Wei, Jiangyue Yan, Chun Kang +4
Vision-Language Models (VLMs) have emerged as versatile solutions for zero-shot question answering (QA) across various domains. However, enabling VLMs to effectively comprehend str…
Alternating Training-based Label Smoothing Enhances Prompt Generalization
Yang Chen, Yanbin Wei, Ke Jin +3
Recent advances in pre-trained vision-language models have demonstrated remarkable zero-shot generalization capabilities. To further enhance these models' adaptability to various d…
DocPruner: A Storage-Efficient Framework for Multi-Vector Visual Document Retrieval via Adaptive Patch-Level Embedding Pruning
Yibo Yan, Guangwei Xu, Xin Zou +3
Visual Document Retrieval (VDR), the task of retrieving visually-rich document pages using queries that combine visual and textual cues, is crucial for numerous real-world applicat…
Harnessing Adaptive Topology Representations for Zero-Shot Graph Question Answering
Yanbin Wei, Jiangyue Yan, Chun Kang +4
Large Multimodal Models (LMMs) have shown generalized zero-shot capabilities in diverse domain question-answering (QA) tasks, including graph QA that involves complex graph topolog…
DynamicMind: A Tri-Mode Thinking System for Large Language Models
Wei Li, Yanbin Wei, Qiushi Huang +4
Modern large language models (LLMs) often struggle to dynamically adapt their reasoning depth to varying task complexities, leading to suboptimal performance or inefficient resourc…
Open Your Eyes: Vision Enhances Message Passing Neural Networks in Link Prediction
Yanbin Wei, Xuehao Wang, Zhan Zhuang +5
Message-passing graph neural networks (MPNNs) and structural features (SFs) are cornerstones for the link prediction task. However, as a common and intuitive mode of understanding,…
Reasoning-Aligned Perception Decoupling for Scalable Multi-modal Reasoning
Yunhao Gou, Kai Chen, Zhili Liu +5
Recent breakthroughs in reasoning language models have significantly advanced text-based reasoning. On the other hand, Multi-modal Large Language Models (MLLMs) still lag behind, h…
Corrupted but Not Broken: Understanding and Mitigating the Negative Impacts of Corrupted Data in Visual Instruction Tuning
Yunhao Gou, Hansi Yang, Zhili Liu +8
Visual Instruction Tuning (VIT) aims to enhance Multimodal Large Language Models (MLLMs), yet its effectiveness is often compromised by corrupted datasets with issues such as hallu…
RouterDC: Query-Based Router by Dual Contrastive Learning for Assembling Large Language Models
Shuhao Chen, Weisen Jiang, Baijiong Lin +2
Recent works show that assembling multiple off-the-shelf large language models (LLMs) can harness their complementary abilities. To achieve this, routing is a promising method, whi…
EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotions
Kai Chen, Yunhao Gou, Runhui Huang +28
GPT-4o, an omni-modal model that enables vocal conversations with diverse emotions and tones, marks a milestone for omni-modal foundation models. However, empowering Large Language…
CodeGraph: Enhancing Graph Reasoning of LLMs with Code
Qiaolong Cai, Zhaowei Wang, Shizhe Diao +2
With the increasing popularity of large language models (LLMs), reasoning on basic graph algorithm problems is an essential intermediate step in assessing their abilities to proces…
Learning Scalable Model Soup on a Single GPU: An Efficient Subspace Training Strategy
Tao Li, Weisen Jiang, Fanghui Liu +2
Pre-training followed by fine-tuning is widely adopted among practitioners. The performance can be improved by "model soups"~\cite{wortsman2022model} via exploring various hyperpar…
Mixture of insighTful Experts (MoTE): The Synergy of Thought Chains and Expert Mixtures in Self-Alignment
Zhili Liu, Yunhao Gou, Kai Chen +8
As the capabilities of large language models (LLMs) continue to expand, aligning these models with human values remains a significant challenge. Recent studies show that reasoning…
Eyes Closed, Safety On: Protecting Multimodal LLMs via Image-to-Text Transformation
Yunhao Gou, Kai Chen, Zhili Liu +6
Multimodal large language models (MLLMs) have shown impressive reasoning abilities. However, they are also more vulnerable to jailbreak attacks than their LLM predecessors. Althoug…
KICGPT: Large Language Model with Knowledge in Context for Knowledge Graph Completion
Yanbin Wei, Qiushi Huang, James T. Kwok +1
Knowledge Graph Completion (KGC) is crucial for addressing knowledge graph incompleteness and supporting downstream applications. Many models have been proposed for KGC. They can b…
GITA: Graph to Visual and Textual Integration for Vision-Language Graph Reasoning
Yanbin Wei, Shuai Fu, Weisen Jiang +5
Large Language Models (LLMs) are increasingly used for various tasks with graph structures. Though LLMs can process graph information in a textual format, they overlook the rich vi…
Mixture of Cluster-conditional LoRA Experts for Vision-language Instruction Tuning
Yunhao Gou, Zhili Liu, Kai Chen +6
Instruction tuning of Large Vision-language Models (LVLMs) has revolutionized the development of versatile models with zero-shot generalization across a wide range of downstream vi…
Implicit Concept Removal of Diffusion Models
Zhili Liu, Kai Chen, Yifan Zhang +6
Text-to-image (T2I) diffusion models often inadvertently generate unwanted concepts such as watermarks and unsafe images. These concepts, termed as the "implicit concepts", could b…