743 citations · 2.1k across the 27 of their papers we have counts for
16 papers · 1 filter
An Effective Theory of Bias Amplification
Arjun Subramonian, Samuel J. Bell, Levent Sagun +1
Machine learning models can capture and amplify biases present in data, leading to disparate test performance across social groups. To better understand, evaluate, and mitigate the…
Reassessing the Validity of Spurious Correlations Benchmarks
Samuel J. Bell, Diane Bouchacourt, Levent Sagun
Neural networks can fail when the data contains spurious correlations. To understand this phenomenon, researchers have proposed numerous spurious correlations benchmarks upon which…
Networked Inequality: Preferential Attachment Bias in Graph Neural Network Link Prediction
Arjun Subramonian, Levent Sagun, Yizhou Sun
Graph neural network (GNN) link prediction is increasingly deployed in citation, collaboration, and online social networks to recommend academic literature, collaborators, and frie…
Simplicity Bias Leads to Amplified Performance Disparities
Samuel J. Bell, Levent Sagun
Which parts of a dataset will a given model find difficult? Recent work has shown that SGD-trained models have a bias towards simplicity, leading them to prioritize learning a majo…
Understanding out-of-distribution accuracies through quantifying difficulty of test samples
Berfin Simsek, Melissa Hall, Levent Sagun
Existing works show that although modern neural networks achieve remarkable generalization performance on the in-distribution (ID) dataset, the accuracy drops significantly on the…
Transformed CNNs: recasting pre-trained convolutional layers with self-attention
Stéphane d'Ascoli, Levent Sagun, Giulio Biroli +1
Vision Transformers (ViT) have recently emerged as a powerful alternative to convolutional networks (CNNs). Although hybrid models attempt to bridge the gap between these two archi…
On the interplay between data structure and loss function in classification problems
Stéphane d'Ascoli, Marylou Gabrié, Levent Sagun +1
One of the central puzzles in modern machine learning is the ability of heavily overparametrized models to generalize well. Although the low-dimensional structure of typical datase…
Post-Workshop Report on Science meets Engineering in Deep Learning, NeurIPS 2019, Vancouver
Levent Sagun, Caglar Gulcehre, Adriana Romero +2
Science meets Engineering in Deep Learning took place in Vancouver as part of the Workshop section of NeurIPS 2019. As organizers of the workshop, we created the following report i…
Triple descent and the two kinds of overfitting: Where & why do they appear?
Stéphane d'Ascoli, Levent Sagun, Giulio Biroli
A recent line of research has highlighted the existence of a "double descent" phenomenon in deep learning, whereby increasing the number of training examples causes the general…
Finding the Needle in the Haystack with Convolutions: on the benefits of architectural bias
Stéphane d'Ascoli, Levent Sagun, Joan Bruna +1
Despite the phenomenal success of deep neural networks in a broad range of learning tasks, there is a lack of theory to understand the way they work. In particular, Convolutional N…
A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networks
Umut Simsekli, Levent Sagun, Mert Gurbuzbalaban
The gradient noise (GN) in the stochastic gradient descent (SGD) algorithm is often considered to be Gaussian in the large data regime by assuming that the classical central limit…
A jamming transition from under- to over-parametrization affects loss landscape and generalization
Stefano Spigler, Mario Geiger, Stéphane d'Ascoli +3
We argue that in fully-connected networks a phase transition delimits the over- and under-parametrized regimes where fitting can or cannot be achieved. Under some general condition…
Empirical Analysis of the Hessian of Over-Parametrized Neural Networks
Levent Sagun, Utku Evci, V. Ugur Guney +2
We study the properties of common loss surfaces through their Hessian matrix. In particular, in the context of deep learning, we empirically show that the spectrum of the Hessian i…
Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond
Levent Sagun, Leon Bottou, Yann LeCun
We look at the eigenvalues of the Hessian of a loss function before and after training. The eigenvalue distribution is seen to be composed of two parts, the bulk which is concentra…
Entropy-SGD: Biasing Gradient Descent Into Wide Valleys
Pratik Chaudhari, Anna Choromanska, Stefano Soatto +6
This paper proposes a new optimization algorithm called Entropy-SGD for training deep neural networks that is motivated by the local geometry of the energy landscape. Local extrema…
Universal halting times in optimization and machine learning
Levent Sagun, Thomas Trogdon, Yann LeCun
The authors present empirical distributions for the halting time (measured by the number of iterations to reach a given accuracy) of optimization algorithms applied to two random s…