Deep Neural Networks for Estimation and Inference

Abstract

We study deep neural networks and their use in semiparametric inference. We establish novel rates of convergence for deep feedforward neural nets. Our new rates are sufficiently fast (in some cases minimax optimal) to allow us to establish valid second-step inference after first-step estimation with deep learning, a result also new to the literature. Our estimation rates and semiparametric inference results handle the current standard architecture: fully connected feedforward neural networks (multi-layer perceptrons), with the now-common rectified linear unit activation function and a depth explicitly diverging with the sample size. We discuss other architectures as well, including fixed-width, very deep networks. We establish nonasymptotic bounds for these deep nets for a general class of nonparametric regression-type loss functions, which includes as special cases least squares, logistic regression, and other generalized linear models. We then apply our theory to develop semiparametric inference, focusing on causal parameters for concreteness, such as treatment effects, expected welfare, and decomposition effects. Inference in many other semiparametric contexts can be readily obtained. We demonstrate the effectiveness of deep learning with a Monte Carlo analysis and an empirical application to direct mail marketing.

References in corpus (10)

• Simultaneous analysis of Lasso and Dantzig selector
• Understanding deep learning requires rethinking generalization
• On asymptotically optimal confidence regions and tests for high-dimensional models
• 2004 IMS Medallion Lecture: Local Rademacher complexities and oracle inequalities in risk minimization
• Local Rademacher complexities
• Robust Inference on Average Treatment Effects with Possibly More Covariates than Observations
• Deep Neural Networks for Estimation and Inference
• Semiparametric efficiency in GMM models with auxiliary data
• Universal Function Approximation by Deep Neural Nets with Bounded Width and ReLU Activations
• Robust Inference Using Inverse Probability Weighting

Cited by in corpus (17)

• Deep Neural Networks for Estimation and Inference
• Double Machine Learning based Program Evaluation under Unconfoundedness
• Targeting customers under response-dependent costs
• Factor Augmented Sparse Throughput Deep ReLU Neural Networks for High Dimensional Regression
• Model-robust and efficient covariate adjustment for cluster-randomized experiments
• Nonlinear Causal Discovery with Confounders
• A Flexible Approach for Predictive Biomarker Discovery
• RNN-based counterfactual prediction, with an application to homestead policy and public schooling
• The costs and benefits of uniformly valid causal inference with high-dimensional nuisance parameters
• Estimating Parameters of Structural Models Using Neural Networks
• A nonparametric framework for treatment effect modifier discovery in high dimensions
• Doubly-robust evaluation of high-dimensional surrogate markers
• Design-based theory for Lasso adjustment in randomized block experiments and rerandomized experiments
• Convolutional neural networks for valid and efficient causal inference
• Applications of 0-1 Neural Networks in Prescription and Prediction
• Bayesian Semi-supervised Inference via a Debiased Modeling Approach
• Deep Regression for Repeated Measurements