Efficient Per-Example Gradient Computations
arXiv:1510.01799
Abstract
This technical report describes an efficient technique for computing the norm of the gradient of the loss function for a neural network with respect to its parameters. This gradient norm can be computed efficiently for every example.
This revision fixed some typos. Many thanks to Hugo Larochelle for reporting them!
References in corpus (1)
Cited by in corpus (23)
- Deep Learning with Differential Privacy
- How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacy
- Practical Deep Learning with Bayesian Principles
- Efficient Deep Learning on Multi-Source Private Data
- Large Language Models Can Be Strong Differentially Private Learners
- Opacus: User-Friendly Differential Privacy Library in PyTorch
- Enabling Fast Differentially Private SGD via Just-in-Time Compilation and Vectorization
- A Study of Gradient Variance in Deep Learning
- An Empirical Study of Large-Batch Stochastic Gradient Descent with Structured Covariance Noise
- Fast and Memory Efficient Differentially Private-SGD via JL Projections
- Gradient Perturbation is Underrated for Differentially Private Convex Optimization
- Efficient Computation of Hessian Matrices in TensorFlow
- Efficient Per-Example Gradient Computations in Convolutional Neural Networks
- Deep Learning with Label Differential Privacy
- Differentially Private Bayesian Neural Networks on Accuracy, Privacy and Reliability
- BackPACK: Packing more into backprop
- Model-agnostic out-of-distribution detection using combined statistical tests
- Learning Deep Neural Networks under Agnostic Corrupted Supervision
- Introspective Learning by Distilling Knowledge from Online Self-explanation
- Bayesian Uncertainty and Expected Gradient Length -- Regression: Two Sides Of The Same Coin?
- An Adaptive and Fast Convergent Approach to Differentially Private Deep Learning
- Adaptive Optimization with Examplewise Gradients
- Training Efficiency and Robustness in Deep Learning