11 citations · 55 across the 26 of their papers we have counts for
34 papers
Agents Explore but Agents Ignore: LLMs Lack Environmental Curiosity
Leon Engländer, Sophia Althammer, Ahmet Üstün +2
LLM-based agents are assumed to integrate environmental observations into their reasoning: discovering highly relevant but unexpected information should naturally lead to a model e…
Tiny Aya: Bridging Scale and Multilingual Depth
Alejandro R. Salamanca, Diana Abagyan, Daniel D'souza +23
Tiny Aya redefines what a small multilingual language model can achieve. Trained on 70 languages and refined through region-aware posttraining, it delivers state-of-the-art in tran…
The Disparate Impacts of Speculative Decoding
Jameson Sandler, Ahmet Üstün, Marco Romanelli +2
The practice of speculative decoding, whereby inference is probabilistically supported by a smaller, cheaper, ``drafter'' model, has become a standard technique for systematically…
Treasure Hunt: Real-time Targeting of the Long Tail using Training-Time Markers
Daniel D'souza, Julia Kreutzer, Adrien Morisot +2
One of the most profound challenges of modern machine learning is performing well on the long-tail of rare and underrepresented features. Large general-purpose models are trained f…
One Tokenizer To Rule Them All: Emergent Language Plasticity via Multilingual Tokenizers
Diana Abagyan, Alejandro R. Salamanca, Andres Felipe Cruz-Salinas +6
Pretraining massively multilingual Large Language Models (LLMs) for many languages at once is challenging due to limited model capacity, scarce high-quality data, and compute const…
The Multilingual Divide and Its Impact on Global AI Safety
Aidan Peppin, Julia Kreutzer, Alice Schoenauer Sebag +13
Despite advances in large language model capabilities in recent years, a large gap remains in their capabilities and safety performance for many languages beyond a relatively small…
Aya Vision: Advancing the Frontier of Multilingual Multimodality
Saurabh Dash, Yiyang Nan, John Dang +22
Building multimodal language models is fundamentally challenging: it requires aligning vision and language modalities, curating high-quality instruction data, and avoiding the degr…
The Leaderboard Illusion
Shivalika Singh, Yiyang Nan, Alex Wang +10
Measuring progress is fundamental to the advancement of any scientific field. As benchmarks play an increasingly central role, they also grow more susceptible to distortion. Chatbo…
A Post-trainer's Guide to Multilingual Training Data: Uncovering Cross-lingual Transfer Dynamics
Luisa Shimabucoro, Ahmet Ustun, Marzieh Fadaee +1
In order for large language models to be useful across the globe, they are fine-tuned to follow instructions on multilingual data. Despite the ubiquity of such post-training, a cle…
Command A: An Enterprise-Ready Large Language Model
Team Cohere, :, Aakanksha +227
In this report we describe the development of Command A, a powerful large language model purpose-built to excel at real-world enterprise use cases. Command A is an agent-optimised…
Aya Expanse: Combining Research Breakthroughs for a New Multilingual Frontier
John Dang, Shivalika Singh, Daniel D'souza +42
We introduce the Aya Expanse model family, a new generation of 8B and 32B parameter multilingual language models, aiming to address the critical challenge of developing highly perf…
If You Can't Use Them, Recycle Them: Optimizing Merging at Scale Mitigates Performance Tradeoffs
Muhammad Khalifa, Yi-Chern Tan, Arash Ahmadian +6
Model merging has shown great promise at combining expert models, but the benefit of merging is unclear when merging "generalist" models trained on many tasks. We explore merging i…
Nexus: Specialization meets Adaptability for Efficiently Training Mixture of Experts
Nikolas Gritsch, Qizhen Zhang, Acyr Locatelli +2
Efficiency, specialization, and adaptability to new data distributions are qualities that are hard to combine in current Large Language Models. The Mixture of Experts (MoE) archite…
To Code, or Not To Code? Exploring Impact of Code in Pre-training
Viraat Aryabumi, Yixuan Su, Raymond Ma +6
Including code in the pre-training data mixture, even for models not specifically designed for code, has become a common practice in LLMs pre-training. While there has been anecdot…
BAM! Just Like That: Simple and Efficient Parameter Upcycling for Mixture of Experts
Qizhen Zhang, Nikolas Gritsch, Dwaraknath Gnaneshwar +8
The Mixture of Experts (MoE) framework has become a popular architecture for large language models due to its superior performance over dense models. However, training MoEs from sc…
RLHF Can Speak Many Languages: Unlocking Multilingual Preference Optimization for LLMs
John Dang, Arash Ahmadian, Kelly Marchisio +3
Preference optimization techniques have become a standard final stage for training state-of-art large language models (LLMs). However, despite widespread adoption, the vast majorit…
How Does Quantization Affect Multilingual LLMs?
Kelly Marchisio, Saurabh Dash, Hongyu Chen +4
Quantization techniques are widely used to improve inference speed and deployment of large language models. While a wide body of work examines the impact of quantization on LLMs in…
Aya 23: Open Weight Releases to Further Multilingual Progress
Viraat Aryabumi, John Dang, Dwarak Talupuru +18
This technical report introduces Aya 23, a family of multilingual language models. Aya 23 builds on the recent release of the Aya model (Üstün et al., 2024), focusing on pairing a…
Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model
Ahmet Üstün, Viraat Aryabumi, Zheng-Xin Yong +14
Recent breakthroughs in large language models (LLMs) have centered around a handful of data-rich languages. What does it take to broaden access to breakthroughs beyond first-class…
Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuning
Shivalika Singh, Freddie Vargus, Daniel Dsouza +30
Datasets are foundational to many breakthroughs in modern artificial intelligence. Many recent achievements in the space of natural language processing (NLP) can be attributed to t…
Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuning
Ted Zadouri, Ahmet Üstün, Arash Ahmadian +3
The Mixture of Experts (MoE) is a widely known neural architecture where an ensemble of specialized sub-models optimizes overall performance with a constant computational cost. How…
Intriguing Properties of Quantization at Scale
Arash Ahmadian, Saurabh Dash, Hongyu Chen +5
Emergent properties have been widely adopted as a term to describe behavior not present in smaller models but observed in larger models. Recent work suggests that the trade-off inc…
Hyper-X: A Unified Hypernetwork for Multi-Task Multilingual Transfer
Ahmet Üstün, Arianna Bisazza, Gosse Bouma +2
Massively multilingual models are promising for transfer learning across tasks and languages. However, existing methods are unable to fully leverage training data when it is availa…
When does Parameter-Efficient Transfer Learning Work for Machine Translation?
Ahmet Üstün, Asa Cooper Stickland
Parameter-efficient fine-tuning methods (PEFTs) offer the promise of adapting large pre-trained models while only tuning a small number of parameters. They have been shown to be co…