7 papers
Initialization Improves LLM-Driven Discovery
Mansi Sakarvadia, Marco Ciccone, Colin Raffel
Large Language Models (LLMs) have been used for novel discovery of algorithms, theorems, drugs, and other tasks through the use of harnesses that prompt an LLM to iteratively optim…
LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning via Shortcut Modulation
Ahmadreza Jeddi, Marco Ciccone, Babak Taati
Looped Transformers have emerged as an efficient and powerful class of models for reasoning in the language domain. Recent studies show that these models achieve strong performance…
The Appeal and Reality of Recycling LoRAs with Adaptive Merging
Haokun Liu, Gyung Hyun Je, Marco Ciccone +3
The widespread availability of fine-tuned LoRA modules for open pre-trained models has led to an interest in methods that can adaptively merge LoRAs to improve performance. These m…
FlexRank: Nested Low-Rank Knowledge Decomposition for Adaptive Model Deployment
Riccardo Zaccone, Stefanos Laskaridis, Marco Ciccone +1
The growing scale of deep neural networks, encompassing large language models (LLMs) and vision transformers (ViTs), has made training from scratch prohibitively expensive and depl…
TokSuite: Measuring the Impact of Tokenizer Choice on Language Model Behavior
Gül Sena Altıntaş, Malikeh Ehghaghi, Brian Lester +4
Tokenizers provide the fundamental basis through which text is represented and processed by language models (LMs). Despite the importance of tokenization, its role in LM performanc…
Gradient-Sign Masking for Task Vector Transport Across Pre-Trained Models
Filippo Rinaldi, Aniello Panariello, Giacomo Salici +4
When a new release of a foundation model is published, practitioners typically need to repeat fine-tuning, even if the same task was already tackled in the previous version. A prom…
From Charts to Atlas: Merging Latent Spaces into One
Donato Crisostomi, Irene Cannistraci, Luca Moschella +4
Models trained on semantically related datasets and tasks exhibit comparable inter-sample relations within their latent spaces. We investigate in this study the aggregation of such…