8 papers
DIET-CP: Lightweight and Data Efficient Self Supervised Continued Pretraining
Bryan Rodas, Natalie Montesino, Jakob Ambsdorf +2
Continued pretraining offers a promising solution for adapting foundation models to a new target domain. However, in specialized domains, available datasets are often very small, l…
Your Attention Matters: to Improve Model Robustness to Noise and Spurious Correlations
Camilo Tamayo-Rousseau, Yunjia Zhao, Yiqun Zhang +1
Self-attention mechanisms are foundational to Transformer architectures, supporting their impressive success in a wide range of tasks. While there are many self-attention variants,…
Task Priors: Enhancing Model Evaluation by Considering the Entire Space of Downstream Tasks
Niket Patel, Randall Balestriero
The grand goal of AI research, and particularly Self Supervised Learning (SSL), is to produce systems that can successfully solve any possible task. In contrast, current evaluation…
FastDINOv2: Frequency Based Curriculum Learning Improves Robustness and Training Speed
Jiaqi Zhang, Juntuo Wang, Zhixin Sun +2
Large-scale vision foundation models such as DINOv2 boast impressive performances by leveraging massive architectures and training datasets. But numerous scenarios require practiti…
General Methods Make Great Domain-specific Foundation Models: A Case-study on Fetal Ultrasound
Jakob Ambsdorf, Asbjørn Munk, Sebastian Llambias +6
With access to large-scale, unlabeled medical datasets, researchers are confronted with two questions: Should they attempt to pretrain a custom foundation model on this medical dat…
LoRA Users Beware: A Few Spurious Tokens Can Manipulate Your Finetuned Model
Marcel Mateos Salles, Praney Goyal, Pradyut Sekhsaria +2
Large Language Models (LLMs) are commonly finetuned for a variety of use cases and domains. A common approach is to leverage Low-Rank Adaptation (LoRA) -- known to provide strong p…