4 papers · 1 filter
Scaling Laws for Multimodal Data Mixtures
Aditi Khandelwal, Ayush K. Tarun, Yixuan Xu +5
Frontier AI systems are increasingly natively multimodal, jointly pretrained on multiple modalities, such as text, vision, and audio. However, the scientific understanding of the m…
Compressed Sensing for Capability Localization in Large Language Models
Anna Bair, Yixuan Even Xu, Mingjie Sun +1
Large language models (LLMs) exhibit a wide range of capabilities, including mathematical reasoning, code generation, and linguistic behaviors. We show that Transformer architectur…
Apertus: Democratizing Open and Compliant LLMs for Global Language Environments
Project Apertus, Alejandro Hernández-Cano, Alexander Hägele +100
We present Apertus, a fully open suite of large language models (LLMs) designed to address two systemic shortcomings in today's open model ecosystem: data compliance and multilingu…
Positional Fragility in LLMs: How Offset Effects Reshape Our Understanding of Memorization Risks
Yixuan Xu, Antoni-Joan Solergibert i Llaquet, Antoine Bosselut +1
Large language models are known to memorize parts of their training data, posing risk of copyright violations. To systematically examine this risk, we pretrain language models (1B/…