10 papers
Scaling Laws for Multimodal Data Mixtures
Aditi Khandelwal, Ayush K. Tarun, Yixuan Xu +5
Frontier AI systems are increasingly natively multimodal, jointly pretrained on multiple modalities, such as text, vision, and audio. However, the scientific understanding of the m…
LACUNA: A Testbed for Evaluating Localization Precision for LLM Unlearning
Matteo Boglioni, Thibault Rousset, Siva Reddy +2
LLMs memorize sensitive training data, including personally identifiable information (PII), creating a pressing need for reliable post hoc removal methods. Unlearning has emerged a…
Toward Open Weight Models Without Risks: Separating Public and Private Capabilities in LLMs
Charbel El Feghali, Arkil Patel, Nicholas Meade +3
Open-weight Large Language Models (LLMs) enable scientific progress and broad deployment. However, they make it difficult to control access to sensitive capabilities. Current pract…
Leveraging Routing Dynamics in Mixture-of-Experts Models for Efficient Language Adaptation
Aditi Khandelwal, Marius Mosbach, Verna Dankers +2
Mixture-of-Experts (MoE) models are widely used to scale language models, yet their expert routing behavior and adaptation in a multilingual setting remain underexplored. In this w…
Weasel: Out-of-Domain Generalization for Web Agents via Importance-Diversity Data Selection
Fatemeh Pesaran Zadeh, Seyeon Choi, Xing Han Lù +2
Large language models (LLMs) have enabled web agents that follow natural language goals through multi-step browser interactions. However, agents fine-tuned on specific trajectories…
Forecasting Downstream Performance of LLMs With Proxy Metrics
Arkil Patel, Siva Reddy, Marius Mosbach +1
Progress in language model development is often driven by comparative decisions: which architecture to adopt, which pretraining corpus to use, or which training recipe to apply. Ma…
LLM2Vec-Gen: Generative Embeddings from Large Language Models
Parishad BehnamGhader, Vaibhav Adlakha, Fabian David Schmidt +3
Fine-tuning LLM-based text embedders via contrastive learning maps inputs and outputs into a new representational space, discarding the LLM's output semantics. We propose LLM2Vec-G…
LatentLens: Revealing Highly Interpretable Visual Tokens in LLMs
Benno Krojer, Shravan Nayak, Oscar Mañas +4
Transforming a large language model (LLM) into a vision-language model (VLM) can be achieved by mapping the visual tokens from a vision encoder into the embedding space of an LLM.…
Value Drifts: Tracing Value Alignment During LLM Post-Training
Mehar Bhatia, Shravan Nayak, Gaurav Kamath +4
As LLMs occupy an increasingly important role in society, they are more and more confronted with questions that require them not only to draw on their general knowledge but also to…
Understanding the Influence of Synthetic Data for Text Embedders
Jacob Mitchell Springer, Vaibhav Adlakha, Siva Reddy +2
Recent progress in developing general purpose text embedders has been driven by training on ever-growing corpora of synthetic LLM-generated data. Nonetheless, no publicly available…