9 papers
From Privacy to Generalization: Linear Max-Information Bounds for DP-SGD
Christoph H. Lampert, Hossein Zakerinia
Understanding the relationship between generalization and privacy remains a central challenge in modern machine learning theory, particularly for deep networks trained by variants…
Beyond Square Roots: Explicit Memory-Efficient Factorization for Multi-Epoch Private Learning
Nikita P. Kalinin, Aki Rehn, Joel Daniel Andersson +2
Correlated-noise mechanisms are among the most promising approaches for improving the utility of differentially private model training, but rigorous guarantees require explicit, an…
Population Risk Bounds for Kolmogorov-Arnold Networks Trained by DP-SGD with Correlated Noise
Puyu Wang, Jan Schuchardt, Nikita Kalinin +4
We establish the first population risk bounds for Kolmogorov-Arnold Networks (KANs) trained by mini-batch SGD with gradient clipping, covering non-private SGD as well as differenti…
DP-λCGD: Efficient Noise Correlation for Differentially Private Model Training
Nikita P. Kalinin, Ryan McKenna, Rasmus Pagh +1
Differentially private stochastic gradient descent (DP-SGD) is the gold standard for training machine learning models with formal differential privacy guarantees. Several recent ex…
Matrix Factorization for Practical Continual Mean Estimation Under User-Level Differential Privacy
Nikita P. Kalinin, Ali Najar, Valentin Roth +1
We study continual mean estimation, where data vectors arrive sequentially and the goal is to maintain accurate estimates of the running mean. We address this problem under user-le…
Sink vs. diagonal patterns as mechanisms for attention switch and oversmoothing prevention
Peter SúkenÃk, Cristina López Amado, Christoph H. Lampert +1
This paper studies the role of sinks and diagonal patterns as attention switch and anti-oversmoothing mechanisms. We analyze geometric conditions under which sinks can be represent…