8 papers
ABBA-Adapters: Efficient and Expressive Fine-Tuning of Foundation Models
Raghav Singhal, Kaustubh Ponkshe, Rohit Vartak +1
Large Language Models have demonstrated strong performance across a wide range of tasks, but adapting them efficiently to new domains remains a key challenge. Parameter-Efficient F…
Safety Subspaces are Not Linearly Distinct: A Fine-Tuning Case Study
Kaustubh Ponkshe, Shaan Shah, Raghav Singhal +1
Large Language Models (LLMs) rely on safety alignment to produce socially acceptable responses. However, this behavior is known to be brittle: further fine-tuning, even on benign o…
Apertus: Democratizing Open and Compliant LLMs for Global Language Environments
Project Apertus, Alejandro Hernández-Cano, Alexander Hägele +100
We present Apertus, a fully open suite of large language models (LLMs) designed to address two systemic shortcomings in today's open model ecosystem: data compliance and multilingu…
Fed-SB: A Silver Bullet for Extreme Communication Efficiency and Performance in (Private) Federated LoRA Fine-Tuning
Raghav Singhal, Kaustubh Ponkshe, Rohit Vartak +2
Low-Rank Adaptation (LoRA) has become ubiquitous for efficiently fine-tuning foundation models. However, federated fine-tuning using LoRA is challenging due to suboptimal updates a…
Initialization using Update Approximation is a Silver Bullet for Extremely Efficient Low-Rank Fine-Tuning
Kaustubh Ponkshe, Raghav Singhal, Eduard Gorbunov +3
Low-rank adapters have become standard for efficiently fine-tuning large language models, but they often fall short of achieving the performance of full fine-tuning. We propose a m…
Jacobian-Free Newton-Krylov with a globalization method for solving groundwater flow models of multi-layer aquifer systems
Raghav Singhal, Emin Can Dogrul, Zhaojun Bai
A Jacobian free Newton Krylov (JFNK) method with a globalization scheme is introduced to solve large and complex nonlinear systems of equations that arise in groundwater flow model…