13 papers
Looking in the Mirror: Introspecting Side-Effect Misalignments Induced by Fine-Tuning
Kotaro Yoshida, Laura Gomezjurado Gonzalez, Yukinori Yamamoto +3
Fine-tuning enables a source model to acquire desired capabilities and behaviors in a target domain while retaining much of its general-purpose competence. However, this adaptation…
Learning More from Less: Exploiting Counterfactuals for Data-Efficient Chart Understanding
Jianzhu Bao, Haozhen Zhang, Kuicai Dong +5
Vision-Language Models (VLMs) have demonstrated remarkable progress in chart understanding, largely driven by supervised fine-tuning (SFT) on increasingly large synthetic datasets.…
Vocabulary Hijacking in LVLMs: Unveiling Critical Attention Heads by Excluding Inert Tokens to Mitigate Hallucination
Yangneng Chen, Junlin Li, Weijun Yao +4
Large Vision-Language Models (LVLMs) have achieved remarkable progress in multimodal tasks, yet their reliability is persistently undermined by hallucinations-generating text that…
Personalizing LLMs with Binary Feedback: A Preference-Corrected Optimization Framework
Xilai Ma, Liye Zhao, Weijun Yao +3
Large Language Model (LLM) personalization aims to align model behaviors with individual user preferences. Existing methods often focus on isolated user histories, neglecting the e…
Task-Aware LLM Routing with Multi-Level Task-Profile-Guided Data Synthesis for Cold-Start Scenarios
Hui Liu, Bin Zou, Kecheng Chen +3
Large language models (LLMs) exhibit substantial variability in performance and computational cost across tasks and queries, motivating routing systems that select models to meet u…
Knowledge Fusion of Large Language Models Via Modular SkillPacks
Guodong Du, Zhuo Li, Xuanning Zhou +9
Cross-capability transfer is a key challenge in large language model (LLM) research, with applications in multi-task integration, model compression, and continual learning. Recent…