7 papers
Mind the Spike: Mechanisms and Brittleness of Visual Massive Activations in Large Vision-Language Models
Jonas Ngnawé, Yann Pequignot, Sabyasachi Sahoo +3
Large vision-language models (LVLMs) inherit massive activations from their text-only bases: spikes where a few fixed hidden channels receive values thousands of times above the ty…
A Stochastic--Geometric Theory of Scaling Laws in Grokking
Róisín Luo, Christian Gagné, Jonas Ngnawé +2
Delayed generalization (\ie~grokking) refers to the phenomenon in which a neural network fits its training data early in training but only begins to generalize after a prolonged de…
Robust Fine-Tuning from Non-Robust Pretrained Models: Mitigating Suboptimal Transfer With Epsilon-Scheduling
Jonas Ngnawé, Maxime Heuillet, Sabyasachi Sahoo +5
Fine-tuning pretrained models is a standard and effective workflow in modern machine learning. However, robust fine-tuning (RFT), which aims to simultaneously achieve adaptation to…
A Guide to Robust Generalization: The Impact of Architecture, Pre-training, and Optimization Strategy
Maxime Heuillet, Rishika Bhagwatkar, Jonas Ngnawé +6
Deep learning models operating in the image domain are vulnerable to small input perturbations. For years, robustness to such perturbations was pursued by training models from scra…
Detecting Brittle Decisions for Free: Leveraging Margin Consistency in Deep Robust Classifiers
Jonas Ngnawé, Sabyasachi Sahoo, Yann Pequignot +2
Despite extensive research on adversarial training strategies to improve robustness, the decisions of even the most robust deep learning models can still be quite sensitive to impe…
A Layer Selection Approach to Test Time Adaptation
Sabyasachi Sahoo, Mostafa ElAraby, Jonas Ngnawe +3
Test Time Adaptation (TTA) addresses the problem of distribution shift by adapting a pretrained model to a new domain during inference. When faced with challenging shifts, most met…
Robustmix: Improving Robustness by Regularizing the Frequency Bias of Deep Nets
Jonas Ngnawe, Marianne Abemgnigni Njifon, Jonathan Heek +1
Deep networks have achieved impressive results on a range of well-curated benchmark datasets. Surprisingly, their performance remains sensitive to perturbations that have little ef…