6 papers · 1 filter
How Do Large Language Models Learn Concepts During Continual Pre-Training?
Barry Menglong Yao, Sha Li, Yunzhi Yao +4
Human beings primarily understand the world through concepts (e.g., dog), abstract mental representations that structure perception, reasoning, and learning. However, how large lan…
AMELI: Enhancing Multimodal Entity Linking with Fine-Grained Attributes
Barry Menglong Yao, Sijia Wang, Yu Chen +5
We propose attribute-aware multimodal entity linking, where the input consists of a mention described with a text paragraph and images, and the goal is to predict the corresponding…
Modality-Specialized Synergizers for Interleaved Vision-Language Generalists
Zhiyang Xu, Minqian Liu, Ying Shen +5
Recent advancements in Vision-Language Models (VLMs) have led to the emergence of Vision-Language Generalists (VLGs) capable of understanding and generating both text and images. H…
Error-driven Data-efficient Large Multimodal Model Tuning
Barry Menglong Yao, Qifan Wang, Lifu Huang
Large Multimodal Models (LMMs) have demonstrated impressive performance across numerous academic benchmarks. However, fine-tuning still remains essential to achieve satisfactory pe…
RoRA-VLM: Robust Retrieval-Augmented Vision Language Models
Jingyuan Qi, Zhiyang Xu, Rulin Shao +5
Current vision-language models (VLMs) still exhibit inferior performance on knowledge-intensive tasks, primarily due to the challenge of accurately encoding all the associations be…
InternalInspector : Robust Confidence Estimation in LLMs through Internal States
Mohammad Beigi, Ying Shen, Runing Yang +7
Despite their vast capabilities, Large Language Models (LLMs) often struggle with generating reliable outputs, frequently producing high-confidence inaccuracies known as hallucinat…