19 citations · 78 across the 27 of their papers we have counts for
7 papers · 1 filter
Initialization Improves LLM-Driven Discovery
Mansi Sakarvadia, Marco Ciccone, Colin Raffel
Large Language Models (LLMs) have been used for novel discovery of algorithms, theorems, drugs, and other tasks through the use of harnesses that prompt an LLM to iteratively optim…
Risk Under Pressure: Compute-Aware Evaluation of Adversarial Robustness in Language Models
Malikeh Ehghaghi, Boglárka Ecsedi, Marsha Chechik +1
Adversarial robustness evaluations of large language models (LLMs) typically report attack success rate (ASR) under fixed query budgets, implicitly treating all attacks as equally…
Better and Worse with Scale: How Contextual Entrainment Diverges with Model Size
Dikshant Kukreja, Kshitij Sah, Gautam Gupta +5
Larger language models become simultaneously better and worse at handling contextual information -- better at ignoring false claims, worse at ignoring irrelevant tokens. We formali…
How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Data
Joel Niklaus, Atsuki Yamaguchi, Michal Štefánik +9
Synthetic data is a standard component in training large language models, yet systematic comparisons across design dimensions, including rephrasing strategy, generator model, and s…
Model Merging via Data-Free Covariance Estimation
Marawan Gamal Abdel Hameed, Derek Tam, Pascal Jr Tikeng Notsawo +2
Model merging provides a way of cheaply combining individual models to produce a model that inherits each individual's capabilities. While some merging methods can approach the per…
The Appeal and Reality of Recycling LoRAs with Adaptive Merging
Haokun Liu, Gyung Hyun Je, Marco Ciccone +3
The widespread availability of fine-tuned LoRA modules for open pre-trained models has led to an interest in methods that can adaptively merge LoRAs to improve performance. These m…
FineInstructions: Scaling Synthetic Instructions to Pre-Training Scale
Ajay Patel, Colin Raffel, Chris Callison-Burch
Due to limited supervised training data, large language models (LLMs) are typically pre-trained via a self-supervised "predict the next word" objective on a vast amount of unstruct…