1 citations · 1 across the 7 of their papers we have counts for
5 papers · 1 filter
When do data mixtures improve scaling laws? Insights from high-dimensional regression
Diyuan Wu, Lehan Chen, Theodor Misiakiewicz +1
Modern machine learning systems are trained on mixtures of data from different domains, and choosing the right mixture can substantially improve downstream performance. Despite an…
A Sharp Transition in Data Reconstruction under Differential Privacy
Max Cairney-Leeming, Simone Bombari, Marco Mondelli
Data reconstruction attacks have empirically been successful in recovering training samples from learned models, raising privacy concerns and motivating defenses with guarantees th…
Optimal Representation Size: High-Dimensional Analysis of Pretraining and Linear Probing
Valentina Njaradi, Clémentine Dominé, Rachel Swanson +2
Learning to generalise from limited data is a fundamental challenge for both artificial and biological systems. A common strategy is to extract reusable structure from abundant unl…
Sink vs. diagonal patterns as mechanisms for attention switch and oversmoothing prevention
Peter Súkeník, Cristina López Amado, Christoph H. Lampert +1
This paper studies the role of sinks and diagonal patterns as attention switch and anti-oversmoothing mechanisms. We analyze geometric conditions under which sinks can be represent…
Improved Scaling Laws via Weak-to-Strong Generalization in Random Feature Ridge Regression
Diyuan Wu, Lehan Chen, Theodor Misiakiewicz +1
It is increasingly common in machine learning to use learned models to label data and then employ such data to train more capable models. The phenomenon of weak-to-strong generaliz…