11 papers
SHALA-LLM: Smartly Handling Ambiguous Labels in Aligning LLMs
Jingyao Wu, Ashley Wang, Keane Ong +2
Many human-centered tasks, including natural language inference (NLI) and emotion recognition (ER), have multiple plausible interpretations, leading to label ambiguity and challeng…
Self-Captioning Multimodal Interaction Tuning: Amplifying Exploitable Redundancies for Robust Vision Language Models
Yuriel Ryan, Hei Man Ip, Adriel Kuek +2
Current vision language models face hallucination and robustness issues against ambiguous or corrupted modalities. We hypothesize that these issues can be addressed by exploiting t…
SynIB: Informational Bottleneck for Maximizing Synergy in Multimodal Learning
Konstantinos Kontras, Teodora Gagaleska, Thomas Strypsteen +4
A central objective in multimodal learning is to capture synergy: task-relevant information that arises only from the joint use of multiple modalities, and is not available from an…
SCATR: Simple Calibrated Test-Time Ranking
Divya Shyamal, Marta Knežević, Lan Tran +3
Test-time scaling (TTS) improves large language models (LLMs) by allocating additional compute at inference time. In practice, TTS is often achieved through parallel scaling: gener…
Interleaved Head Attention
Sai Surya Duvvuri, Chanakya Ekbote, Rachit Bansal +6
Multi-Head Attention (MHA) is the core computational primitive underlying modern Large Language Models (LLMs). However, MHA suffers from a fundamental linear scaling limitation: $H…
TAMP: Token-Adaptive Layerwise Pruning in Multimodal Large Language Models
Jaewoo Lee, Keyang Xuan, Chanakya Ekbote +3
Multimodal Large Language Models (MLLMs) have shown remarkable versatility in understanding diverse multimodal data and tasks. However, these capabilities come with an increased mo…
Alice: Proactive Learning with Teacher's Demonstrations for Weak-to-Strong Generalization
Shujin Wu, Cheng Qian, Yi R. Fung +2
The growing capabilities of large language models (LLMs) present a key challenge of maintaining effective human oversight. Weak-to-strong generalization (W2SG) offers a promising f…
MimeQA: Towards Socially-Intelligent Nonverbal Foundation Models
Hengzhi Li, Megan Tjandrasuwita, Yi R. Fung +2
As AI becomes more closely integrated with peoples' daily activities, socially intelligent AI that can understand and interact seamlessly with humans in daily lives is increasingly…
Understanding the Emergence of Multimodal Representation Alignment
Megan Tjandrasuwita, Chanakya Ekbote, Liu Ziyin +1
Multimodal representation learning is fundamentally about transforming incomparable modalities into comparable representations. While prior research primarily focused on explicitly…
VLM2-Bench: A Closer Look at How Well VLMs Implicitly Link Explicit Matching Visual Cues
Jianshu Zhang, Dongyu Yao, Renjie Pi +2
Visually linking matching cues is a crucial ability in daily life, such as identifying the same person in multiple photos based on their cues, even without knowing who they are. De…
Group-Adaptive Threshold Optimization for Robust AI-Generated Text Detection
Minseok Jung, Cynthia Fuertes Panizo, Liam Dugan +4
The advancement of large language models (LLMs) has made it difficult to differentiate human-written text from AI-generated text. Several AI-text detectors have been developed in r…