1 citations · 1 across the 7 of their papers we have counts for
9 papers
When do data mixtures improve scaling laws? Insights from high-dimensional regression
Diyuan Wu, Lehan Chen, Theodor Misiakiewicz +1
Modern machine learning systems are trained on mixtures of data from different domains, and choosing the right mixture can substantially improve downstream performance. Despite an…
A Sharp Transition in Data Reconstruction under Differential Privacy
Max Cairney-Leeming, Simone Bombari, Marco Mondelli
Data reconstruction attacks have empirically been successful in recovering training samples from learned models, raising privacy concerns and motivating defenses with guarantees th…
Optimal Representation Size: High-Dimensional Analysis of Pretraining and Linear Probing
Valentina Njaradi, Clémentine Dominé, Rachel Swanson +2
Learning to generalise from limited data is a fundamental challenge for both artificial and biological systems. A common strategy is to extract reusable structure from abundant unl…
Sink vs. diagonal patterns as mechanisms for attention switch and oversmoothing prevention
Peter Súkeník, Cristina López Amado, Christoph H. Lampert +1
This paper studies the role of sinks and diagonal patterns as attention switch and anti-oversmoothing mechanisms. We analyze geometric conditions under which sinks can be represent…
Improved Scaling Laws via Weak-to-Strong Generalization in Random Feature Ridge Regression
Diyuan Wu, Lehan Chen, Theodor Misiakiewicz +1
It is increasingly common in machine learning to use learned models to label data and then employ such data to train more capable models. The phenomenon of weak-to-strong generaliz…
A Law of Data Reconstruction for Random Features (and Beyond)
Leonardo Iurada, Simone Bombari, Tatiana Tommasi +1
Large-scale deep learning models are known to memorize parts of the training set. In machine learning theory, memorization is often framed as interpolation or label fitting, and cl…
High-Dimensional Private Linear Regression with Optimal Rates
Simone Bombari, Jialei Luo, Inbar Seroussi +1
Differentially private (DP) linear regression has received significant attention in the recent theoretical literature, with several approaches proposed to improve error rates. Our…
Compression of Structured Data with Autoencoders: Provable Benefit of Nonlinearities and Depth
Kevin Kögler, Alexander Shevchenko, Hamed Hassani +1
Autoencoders are a prominent model in many empirical branches of machine learning and lossy data compression. However, basic theoretical questions remain unanswered even in a shall…
Tight Bounds for Linear and Non-Linear Contraction of Divergences via Duality
Amedeo Roberto Esposito, Marco Mondelli
We develop a novel framework for bounding the contraction of information divergences, using duality and associated norms in Orlicz spaces. By working in the dual space, we obtain a…