250 citations · 366 across the 47 of their papers we have counts for
53 papers
Scaling Inference-Time Compute with Teams of Coding Agents
Keenan Samway, Daniel D'souza, Omar Darwiche Domingues +3
Multi-agent systems promise a new axis for scaling inference: rather than having one agent think longer or sample more attempts, a task is split across many agents that work in par…
Building Multilingual Bridges: Data Mixing as the Pillar of Generalization for In-Language Reasoning
Mehrnaz Mofakhami, Ananya Sahu, Alejandro R. Salamanca +5
Reasoning language models have made substantial advances on a variety of complex tasks, yet their capabilities remain overwhelmingly English-centric: models primarily reason in Eng…
The IOL-AI Challenge: An Open Challenge towards Advancing Linguistic Reasoning
Eduardo Sánchez, Rita Berrada, Dan-Mircea Mirea +8
Reasoning in LLMs is overwhelmingly studied in domains that provide a model with rules: mathematics and code. Linguistic puzzles invert this: the solver must first discover the sys…
CALIBER: Calibrating Confidence Before and After Reasoning in Language Models
Conor Finlay, Joshua Kurien, Saurabh Dash +2
Reasoning language models are increasingly asked not only to answer difficult questions, but also to estimate their likelihood of success. Existing methods typically elicit confide…
AI Exposure Scores: what they measure, what they miss, and what comes next
Campbell Lund, Thomas Euyang, Zanele Munyikwa +1
A set of exposure scores calculated in 2023 has become a central empirical input to the future of work debate. Produced by Eloundou et al. (2023) and referred to here as the GPTs a…
The Culture Funnel: You Can't Align What isn't in the Data
Ananya Sahu, Mehrnaz Mofakhami, Daniel D'Souza +3
Current cultural alignment approaches focus on inference-time interventions, assuming models already contain sufficient cultural knowledge. We argue modern LLM pipelines suffer fro…
Soft-SVeRL: Self-Verified Reinforcement Learning with Soft Rewards
Saurabh Dash, Pierre Clavier, John Dang +4
Reinforcement Learning from Verifiable Rewards (RLVR) has improved language models in domains such as mathematics and code, where correctness can be checked automatically. However,…
CIRCLE: A Framework for Evaluating AI from a Real-World Lens
Reva Schwartz, Carina Westling, Morgan Briggs +12
This study proposes CIRCLE, a six-stage, lifecycle-based framework to bridge the reality gap between model-centric performance metrics and AI system outcomes in deployment. Current…
Tiny Aya: Bridging Scale and Multilingual Depth
Alejandro R. Salamanca, Diana Abagyan, Daniel D'souza +23
Tiny Aya redefines what a small multilingual language model can achieve. Trained on 70 languages and refined through region-aware posttraining, it delivers state-of-the-art in tran…
Unlocking Reasoning Capability on Machine Translation in Large Language Models
Sara Rajaee, Sebastian Vincent, Alexandre Berard +3
Reasoning-oriented large language models (RLMs) achieve strong gains on tasks such as mathematics and coding by generating explicit intermediate reasoning. However, their impact on…
SimMerge: Learning to Select Merge Operators from Similarity Signals
Oliver Bolton, Aakanksha, Arash Ahmadian +3
Model merging combines multiple models into a single model with aggregated capabilities, making it a powerful tool for large language model (LLM) development. However, scaling mode…
The Art of Asking: Multilingual Prompt Optimization for Synthetic Data
David Mora, Viraat Aryabumi, Wei-Yin Ko +3
Synthetic data has become a cornerstone for scaling large language models, yet its multilingual use remains bottlenecked by translation-based prompts. This strategy inherits Englis…
Making, not Taking, the Best of N
Ammar Khairi, Daniel D'souza, Marzieh Fadaee +1
Obtaining high-quality generations in modern LLMs has largely been framed as a selection problem: identifying a single winning generation from a diverse pool of N samples, the Best…
Verification Limits Code LLM Training
Srishti Gureja, Elena Tommasone, Jingyi He +3
Large language models for code generation increasingly rely on synthetic data, where both problem solutions and verification tests are generated by models. While this enables scala…
NeoBabel: A Multilingual Open Tower for Visual Generation
Mohammad Mahdi Derakhshani, Dheeraj Varghese, Marzieh Fadaee +1
Text-to-image generation advancements have been predominantly English-centric, creating barriers for non-English speakers and perpetuating digital inequities. While existing system…
One Tokenizer To Rule Them All: Emergent Language Plasticity via Multilingual Tokenizers
Diana Abagyan, Alejandro R. Salamanca, Andres Felipe Cruz-Salinas +6
Pretraining massively multilingual Large Language Models (LLMs) for many languages at once is challenging due to limited model capacity, scarce high-quality data, and compute const…
The State of Multilingual LLM Safety Research: From Measuring the Language Gap to Mitigating It
Zheng-Xin Yong, Beyza Ermis, Marzieh Fadaee +2
This paper presents a comprehensive analysis of the linguistic diversity of LLM safety research, highlighting the English-centric nature of the field. Through a systematic review o…
Reality Check: A New Evaluation Ecosystem Is Necessary to Understand AI's Real World Effects
Reva Schwartz, Rumman Chowdhury, Akash Kundu +17
Conventional AI evaluation approaches concentrated within the AI stack exhibit systemic limitations for exploring, navigating and resolving the human and societal factors that play…
The Multilingual Divide and Its Impact on Global AI Safety
Aidan Peppin, Julia Kreutzer, Alice Schoenauer Sebag +13
Despite advances in large language model capabilities in recent years, a large gap remains in their capabilities and safety performance for many languages beyond a relatively small…
Aya Vision: Advancing the Frontier of Multilingual Multimodality
Saurabh Dash, Yiyang Nan, John Dang +22
Building multimodal language models is fundamentally challenging: it requires aligning vision and language modalities, curating high-quality instruction data, and avoiding the degr…
The Leaderboard Illusion
Shivalika Singh, Yiyang Nan, Alex Wang +10
Measuring progress is fundamental to the advancement of any scientific field. As benchmarks play an increasingly central role, they also grow more susceptible to distortion. Chatbo…
Kaleidoscope: In-language Exams for Massively Multilingual Vision Evaluation
Israfel Salazar, Manuel Fernández Burda, Shayekh Bin Islam +42
The evaluation of vision-language models (VLMs) has mainly relied on English-language benchmarks, leaving significant gaps in both multilingual and multicultural coverage. While mu…
A Post-trainer's Guide to Multilingual Training Data: Uncovering Cross-lingual Transfer Dynamics
Luisa Shimabucoro, Ahmet Ustun, Marzieh Fadaee +1
In order for large language models to be useful across the globe, they are fine-tuned to follow instructions on multilingual data. Despite the ubiquity of such post-training, a cle…
Command A: An Enterprise-Ready Large Language Model
Team Cohere, :, Aakanksha +227
In this report we describe the development of Command A, a powerful large language model purpose-built to excel at real-world enterprise use cases. Command A is an agent-optimised…