Model Repair: Robust Recovery of Over-Parameterized Statistical Models
arXiv:2005.09912
Abstract
A new type of robust estimation problem is introduced where the goal is to recover a statistical model that has been corrupted after it has been estimated from data. Methods are proposed for "repairing" the model using only the design and not the response values used to fit the model in a supervised learning setting. Theory is developed which reveals that two important ingredients are necessary for model repair---the statistical model must be over-parameterized, and the estimator must incorporate redundancy. In particular, estimators based on stochastic gradient descent are seen to be well suited to model repair, but sparse estimators are not in general repairable. After formulating the problem and establishing a key technical lemma related to robust estimation, a series of results are presented for repair of over-parameterized linear models, random feature models, and artificial neural networks. Simulation studies are presented that corroborate and illustrate the theoretical findings.
References in corpus (10)
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
- Density estimation using Real NVP
- A Convergence Theory for Deep Learning via Over-Parameterization
- Gradient Descent Provably Optimizes Over-parameterized Neural Networks
- The generalization error of random features regression: Precise asymptotics and double descent curve
- On risk bounds in isotonic and other shape restricted regression problems
- Capacity-achieving Sparse Superposition Codes via Approximate Message Passing Decoding
- A new perspective on least squares under convex constraint
- Recent Advances in Algorithmic High-Dimensional Robust Statistics
- Adaptive Hard Thresholding for Near-optimal Consistent Robust Regression