1.9k citations · 5.6k across the 24 of their papers we have counts for
14 papers · 1 filter
Ticketed Learning-Unlearning Schemes
Badih Ghazi, Pritish Kamath, Ravi Kumar +3
We consider the learning--unlearning paradigm defined as follows. First given a dataset, the goal is to learn a good predictor, such as one minimizing a certain loss. Subsequently,…
Private Ad Modeling with DP-SGD
Carson Denison, Badih Ghazi, Pritish Kamath +6
A well-known algorithm in privacy-preserving ML is differentially private stochastic gradient descent (DP-SGD). While this algorithm has been evaluated on text and image data, it h…
Learning to Reason with Neural Networks: Generalization, Unseen Data and Boolean Measures
Emmanuel Abbe, Samy Bengio, Elisabetta Cornacchia +4
This paper considers the Pointer Value Retrieval (PVR) benchmark introduced in [ZRKB21], where a 'reasoning' function acts on a string of digits to produce the label. More generall…
Quantifying Memorization Across Neural Language Models
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski +3
Large language models (LMs) have been shown to memorize parts of their training data, and when prompted appropriately, they will emit the memorized training data verbatim. This is…
Understanding Invariance via Feedforward Inversion of Discriminatively Trained Classifiers
Piotr Teterwak, Chiyuan Zhang, Dilip Krishnan +1
A discriminatively trained neural net classifier can fit the training data perfectly if all information about its input other than class membership has been discarded prior to the…
Deep Learning with Label Differential Privacy
Badih Ghazi, Noah Golowich, Ravi Kumar +2
The Randomized Response (RR) algorithm is a classical technique to improve robustness in survey aggregation, and has been widely adopted in applications with differential privacy g…
What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation
Vitaly Feldman, Chiyuan Zhang
Deep learning algorithms are well-known to have a propensity for fitting the training data very well and often fit even outliers and mislabeled data points. Such fitting requires m…
What is being transferred in transfer learning?
Behnam Neyshabur, Hanie Sedghi, Chiyuan Zhang
One desired capability for machines is the ability to transfer their knowledge of one domain to another where data is (usually) scarce. Despite ample adaptation of transfer learnin…
Characterizing Structural Regularities of Labeled Data in Overparameterized Models
Ziheng Jiang, Chiyuan Zhang, Kunal Talwar +1
Humans are accustomed to environments that contain both regularities and exceptions. For example, at most gas stations, one pays prior to pumping, but the occasional rural station…
A Study on Overfitting in Deep Reinforcement Learning
Chiyuan Zhang, Oriol Vinyals, Remi Munos +1
Recent years have witnessed significant progresses in deep Reinforcement Learning (RL). Empowered with large scale neural networks, carefully designed architectures, novel training…
Theory of Deep Learning IIb: Optimization Properties of SGD
Chiyuan Zhang, Qianli Liao, Alexander Rakhlin +3
In Theory IIb we characterize with a mix of theory and experiments the optimization of deep convolutional networks by Stochastic Gradient Descent. The main new result in this paper…
Understanding deep learning requires rethinking generalization
Chiyuan Zhang, Samy Bengio, Moritz Hardt +2
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small difference between training and test performance. Conventional wisdom attribut…
Training Deep Nets with Sublinear Memory Cost
Tianqi Chen, Bing Xu, Chiyuan Zhang +1
We propose a systematic approach to reduce the memory consumption of deep neural network training. Specifically, we design an algorithm that costs O(sqrt(n)) memory to train a n la…
Learning with a Wasserstein Loss
Charlie Frogner, Chiyuan Zhang, Hossein Mobahi +2
Learning to predict multi-label outputs is challenging, but in many problems there is a natural metric on the outputs that can be used to improve predictions. In this paper we deve…