Minimax Estimation of Conditional Moment Models

Abstract

We develop an approach for estimating models described via conditional moment restrictions, with a prototypical application being non-parametric instrumental variable regression. We introduce a min-max criterion function, under which the estimation problem can be thought of as solving a zero-sum game between a modeler who is optimizing over the hypothesis space of the target model and an adversary who identifies violating moments over a test function space. We analyze the statistical estimation rate of the resulting estimator for arbitrary hypothesis spaces, with respect to an appropriate analogue of the mean squared error metric, for ill-posed inverse problems. We show that when the minimax criterion is regularized with a second moment penalty on the test function and the test function space is sufficiently rich, then the estimation rate scales with the critical radius of the hypothesis and test function spaces, a quantity which typically gives tight fast rates. Our main result follows from a novel localized Rademacher analysis of statistical learning problems defined via minimax objectives. We provide applications of our main results for several hypothesis spaces used in practice such as: reproducing kernel Hilbert spaces, high dimensional sparse linear functions, spaces defined via shape constraints, ensemble estimators such as random forests, and neural networks. For each of these applications we provide computationally efficient optimization methods for solving the corresponding minimax problem (e.g. stochastic first-order heuristics for neural networks). In several applications, we show how our modified mean squared error rate, combined with conditions that bound the ill-posedness of the inverse problem, lead to mean squared error rates. We conclude with an extensive experimental analysis of the proposed methods.

References in corpus (17)

• Nonparametric methods for inference in the presence of instrumental variables
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• A Reliable Effective Terascale Linear Learning System
• Spectrally-normalized margin bounds for neural networks
• On risk bounds in isotonic and other shape restricted regression problems
• High-dimensional instrumental variables regression and confidence sets
• Endogeneity in high dimensions
• Lower bounds on the performance of polynomial-time algorithms for sparse linear regression
• Empirical entropy, minimax regret and minimax risk
• Deep Generalized Method of Moments for Instrumental Variable Analysis
• Near-Optimal Algorithms for Minimax Optimization
• Global Convergence and Variance-Reduced Optimization for a Class of Nonconvex-Nonconcave Minimax Problems
• Dual Instrumental Variable Regression
• Revisiting Stochastic Extragradient
• On the Optimality of Kernel-Embedding Based Goodness-of-Fit Tests
• SGD Learns One-Layer Networks in WGANs
• Kernel Conditional Moment Test via Maximum Moment Restriction

Cited by in corpus (11)

• DoWhy: Addressing Challenges in Expressing and Validating Causal Assumptions
• Causal Inference Under Unmeasured Confounding With Negative Controls: A Minimax Learning Approach
• Finite Sample Analysis of Minimax Offline Reinforcement Learning: Completeness, Fast Rates and First-Order Efficiency
• Instrumental Variable Regression via Kernel Maximum Moment Loss
• Proximal Causal Learning with Kernels: Two-Stage Estimation and Moment Restriction
• Instrumental Variable Value Iteration for Causal Offline Reinforcement Learning
• Mostly Harmless Machine Learning: Learning Optimal Instruments in Linear IV Models
• Provably Efficient Neural Estimation of Structural Equation Model: An Adversarial Approach
• On Instrumental Variable Regression for Deep Offline Policy Evaluation
• Stochastic Extragradient: General Analysis and Improved Rates
• Instrument Space Selection for Kernel Maximum Moment Restriction