36 citations · 51 across the 27 of their papers we have counts for
8 papers · 1 filter
Benchmarking Optimizers for Large Language Model Pretraining
Andrei Semenov, Matteo Pagliardini, Martin Jaggi
The recent development of Large Language Models (LLMs) has been accompanied by an effervescence of novel ideas and methods to better optimize the loss of deep learning models. Clai…
Semantic uncertainty in advanced decoding methods for LLM generation
Darius Foodeei, Simin Fan, Martin Jaggi
This study investigates semantic uncertainty in large language model (LLM) outputs across different decoding methods, focusing on emerging techniques like speculative sampling and…
Gradient-Normalized Smoothness for Optimization with Approximate Hessians
Andrei Semenov, Martin Jaggi, Nikita Doikov
In this work, we develop new optimization algorithms that use approximate second-order information combined with the gradient regularization technique to achieve fast global conver…
Balancing Knowledge Delivery and Emotional Comfort in Healthcare Conversational Systems
Shang-Chi Tsai, Yun-Nung Chen
With the advancement of large language models, many dialogue systems are now capable of providing reasonable and informative responses to patients' medical conditions. However, whe…
GRAPE: Optimize Data Mixture for Group Robust Multi-target Adaptive Pretraining
Simin Fan, Maria Ios Glarou, Martin Jaggi
The performance of large language models (LLMs) across diverse downstream applications is fundamentally governed by the quality and composition of their pretraining corpora. Existi…
NeuralGrok: Accelerate Grokking by Neural Gradient Transformation
Xinyu Zhou, Simin Fan, Martin Jaggi +1
Grokking is proposed and widely studied as an intricate phenomenon in which generalization is achieved after a long-lasting period of overfitting. In this work, we propose NeuralGr…
Mitigating Unintended Memorization with LoRA in Federated Learning for LLMs
Thierry Bossy, Julien Vignoud, Tahseen Rabbani +2
Federated learning (FL) is a popular paradigm for collaborative training which avoids direct data exposure between clients. However, data privacy issues still remain: FL-trained la…
Leveraging the true depth of LLMs
Ramón Calvo González, Daniele Paliotta, Matteo Pagliardini +2
The remarkable capabilities of Large Language Models (LLMs) are overshadowed by their immense computational cost. While recent work has shown that many LLM layers can be reordered…