414 citations · 723 across the 50 of their papers we have counts for
10 papers · 1 filter
Adapting Language Models via Token Translation
Zhili Feng, Tanya Marwah, Nicolo Fusi +2
Modern large language models use a fixed tokenizer to effectively compress text drawn from a source domain. However, applying the same tokenizer to a new target domain often leads…
Mixture of Parrots: Experts improve memorization more than reasoning
Samy Jelassi, Clara Mohri, David Brandfonbrener +7
The Mixture-of-Experts (MoE) architecture enables a significant increase in the total number of model parameters with minimal computational overhead. However, it is not clear what…
Understanding the Role of Functional Diversity in Weight-Ensembling with Ingredient Selection and Multidimensional Scaling
Alex Rojas, David Alvarez-Melis
Weight-ensembles are formed when the parameters of multiple neural networks are directly averaged into a single model. They have demonstrated generalization capability in-distribut…
What is the Right Notion of Distance between Predict-then-Optimize Tasks?
Paula Rodriguez-Diaz, Lingkai Kong, Kai Wang +2
Comparing datasets is a fundamental task in machine learning, essential for various learning paradigms-from evaluating train and test datasets for model generalization to using dat…
Strongly Isomorphic Neural Optimal Transport Across Incomparable Spaces
Athina Sotiropoulou, David Alvarez-Melis
Optimal Transport (OT) has recently emerged as a powerful framework for learning minimal-displacement maps between distributions. The predominant approach involves a neural paramet…
CharED: Character-wise Ensemble Decoding for Large Language Models
Kevin Gu, Eva Tuecke, Dmitriy Katz +3
Large language models (LLMs) have shown remarkable potential for problem solving, with open source models achieving increasingly impressive performance on benchmarks measuring area…