33 citations · 101 across the 19 of their papers we have counts for
21 papers
High-Dimensional Asymptotics and Dataset Selection for Private Transfer Learning
Filip Kovačević, Edwige Cyffers, Stefano Sarao Mannelli +1
To commit to buying external data or participate in collaborative learning, one must decide whether the additional data will improve prediction enough to justify the cost. This com…
Position: the Stochastic Parrot in the Coal Mine. Model Collapse is a Threat to Low-Resource Communities
Devon Jarvis, Richard Klein, Benjamin Rosman +2
Model collapse, the degradation in performance that arises when generative models are trained on the outputs of prior models, is an increasing concern as artificially generated con…
Sharp description of local minima in the loss landscape of high-dimensional two-layer ReLU neural networks
Jie Huang, Bruno Loureiro, Stefano Sarao Mannelli
We study the population loss landscape of two-layer ReLU networks of the form in a realisable teacher-student setting with Gaussian covaria…
The Interplay of Data Structure and Imbalance in the Learning Dynamics of Diffusion Models
Flavio Nicoletti, Chenxiao Ma, Enrico Ventura +2
Real-world datasets are inherently heterogeneous, yet how per-class structural differences and sampling imbalance shape the training dynamics of diffusion models-and potentially ex…
A Theory of Initialisation's Impact on Specialisation
Devon Jarvis, Sebastian Lee, Clémentine Carla Juliette Dominé +2
Prior work has demonstrated a consistent tendency in neural networks engaged in continual learning tasks, wherein intermediate task similarity results in the highest levels of cata…
Optimal Protocols for Continual Learning via Statistical Physics and Control Theory
Francesco Mori, Stefano Sarao Mannelli, Francesca Mignacco
Artificial neural networks often struggle with catastrophic forgetting when learning multiple tasks sequentially, as training on new tasks degrades the performance on previously le…
Bias in Motion: Theoretical Insights into the Dynamics of Bias in SGD Training
Anchit Jain, Rozhin Nobahari, Aristide Baratin +1
Machine learning systems often acquire biases by leveraging undesired features in the data, impacting accuracy variably across different sub-populations. Current understanding of b…
Why Do Animals Need Shaping? A Theory of Task Composition and Curriculum Learning
Jin Hwa Lee, Stefano Sarao Mannelli, Andrew Saxe
Diverse studies in systems neuroscience begin with extended periods of curriculum training known as `shaping' procedures. These involve progressively studying component parts of mo…
The RL Perceptron: Generalisation Dynamics of Policy Learning in High Dimensions
Nishil Patel, Sebastian Lee, Stefano Sarao Mannelli +2
Reinforcement learning (RL) algorithms have proven transformative in a range of domains. To tackle real-world domains, these systems often use neural networks to learn policies dir…
Optimal transfer protocol by incremental layer defrosting
Federica Gerace, Diego Doimo, Stefano Sarao Mannelli +2
Transfer learning is a powerful tool enabling model training with limited amounts of data. This technique is particularly useful in real-world problems where data availability is o…
Maslow's Hammer for Catastrophic Forgetting: Node Re-Use vs Node Activation
Sebastian Lee, Stefano Sarao Mannelli, Claudia Clopath +2
Continual learning - learning new tasks in sequence while maintaining performance on old tasks - remains particularly challenging for artificial neural networks. Surprisingly, the…
An Analytical Theory of Curriculum Learning in Teacher-Student Networks
Luca Saglietti, Stefano Sarao Mannelli, Andrew Saxe
In humans and animals, curriculum learning -- presenting data in a curated order - is critical to rapid learning and effective pedagogy. Yet in machine learning, curricula are not…
Analytical Study of Momentum-Based Acceleration Methods in Paradigmatic High-Dimensional Non-Convex Problems
Stefano Sarao Mannelli, Pierfrancesco Urbani
The optimization step in many machine learning problems rarely relies on vanilla gradient descent but it is common practice to use momentum-based accelerated methods. Despite these…
Post-Workshop Report on Science meets Engineering in Deep Learning, NeurIPS 2019, Vancouver
Levent Sagun, Caglar Gulcehre, Adriana Romero +2
Science meets Engineering in Deep Learning took place in Vancouver as part of the Workshop section of NeurIPS 2019. As organizers of the workshop, we created the following report i…
Complex Dynamics in Simple Neural Networks: Understanding Gradient Flow in Phase Retrieval
Stefano Sarao Mannelli, Giulio Biroli, Chiara Cammarota +3
Despite the widespread use of gradient-based algorithms for optimizing high-dimensional non-convex functions, understanding their ability of finding good minima instead of being tr…
Optimization and Generalization of Shallow Neural Networks with Quadratic Activation Functions
Stefano Sarao Mannelli, Eric Vanden-Eijnden, Lenka Zdeborová
We study the dynamics of optimization and the generalization properties of one-hidden layer neural networks with quadratic activation function in the over-parametrized regime where…
Reducing Recurrent Competitive Epidemics via Dynamic Resource Allocation
Argyris Kalogeratos, Gaspard Abel, Stefano Sarao Mannelli
Motivated by scenarios of epidemic competition, as well as how social contagions spread at the level of individuals, this work considers the competition between two conflicting nod…
Thresholds of descending algorithms in inference problems
Stefano Sarao Mannelli, Lenka Zdeborova
We review recent works on analyzing the dynamics of gradient-based algorithms in a prototypical statistical inference problem. Using methods and insights from the physics of glassy…
Who is Afraid of Big Bad Minima? Analysis of Gradient-Flow in a Spiked Matrix-Tensor Model
Stefano Sarao Mannelli, Giulio Biroli, Chiara Cammarota +2
Gradient-based algorithms are effective for many machine learning tasks, but despite ample recent effort and some progress, it often remains unclear why they work in practice in op…
Passed & Spurious: Descent Algorithms and Local Minima in Spiked Matrix-Tensor Models
Stefano Sarao Mannelli, Florent Krzakala, Pierfrancesco Urbani +1
In this work we analyse quantitatively the interplay between the loss landscape and performance of descent algorithms in a prototypical inference problem, the spiked matrix-tensor…
Marvels and Pitfalls of the Langevin Algorithm in Noisy High-dimensional Inference
Stefano Sarao Mannelli, Giulio Biroli, Chiara Cammarota +3
Gradient-descent-based algorithms and their stochastic versions have widespread applications in machine learning and statistical inference. In this work we perform an analytic stud…