6 papers
ViCA: Efficient Multimodal LLMs with Vision-Only Cross-Attention
Wenjie Liu, Hao Wu, Xin Qiu +6
Modern multimodal large language models (MLLMs) adopt a unified self-attention design that processes visual and textual tokens at every Transformer layer, incurring substantial com…
Modality as Heterogeneity: Node Splitting and Graph Rewiring for Multimodal Graph Learning
Yihan Zhang, Ercan E. Kuruoglu
Multimodal graphs are gaining increasing attention due to their rich representational power and wide applicability, yet they introduce substantial challenges arising from severe mo…
Reverse Thinking Enhances Missing Information Detection in Large Language Models
Yuxin Liu, Chaojie Gu, Yihang Zhang +2
Large Language Models (LLMs) have demonstrated remarkable capabilities in various reasoning tasks, yet they often struggle with problems involving missing information, exhibiting i…
DFAMS: Dynamic-flow guided Federated Alignment based Multi-prototype Search
Zhibang Yang, Xinke Jiang, Rihong Qiu +8
Federated Retrieval (FR) routes queries across multiple external knowledge sources, to mitigate hallucinations of LLMs, when necessary external knowledge is distributed. However, e…
Sentence-level Reward Model can Generalize Better for Aligning LLM from Human Preference
Wenjie Qiu, Yi-Chen Li, Xuqin Zhang +4
Learning reward models from human preference datasets and subsequently optimizing language models via reinforcement learning has emerged as a fundamental paradigm for aligning LLMs…
SSMLoRA: Enhancing Low-Rank Adaptation with State Space Model
Jiayang Yu, Yihang Zhang, Bin Wang +3
Fine-tuning is a key approach for adapting language models to specific downstream tasks, but updating all model parameters becomes impractical as model sizes increase. Parameter-Ef…