On Learning High Dimensional Structured Single Index Models

Abstract

Single Index Models (SIMs) are simple yet flexible semi-parametric models for machine learning, where the response variable is modeled as a monotonic function of a linear combination of features. Estimation in this context requires learning both the feature weights and the nonlinear function that relates features to observations. While methods have been described to learn SIMs in the low dimensional regime, a method that can efficiently learn SIMs in high dimensions, and under general structural assumptions, has not been forthcoming. In this paper, we propose computationally efficient algorithms for SIM inference in high dimensions with structural constraints. Our general approach specializes to sparsity, group sparsity, and low-rank assumptions among others. Experiments show that the proposed method enjoys superior predictive performance when compared to generalized linear models, and achieves results comparable to or better than single layer feedforward neural networks with significantly less computational cost.

7 pages, 3 tables, 1 Figure, substantial text overlap with arXiv:1506.08910; Accepted for publication at AAAI 2017; added new experimental results comparing our method to a single layer neural network

References in corpus (7)

• Adam: A Method for Stochastic Optimization
• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• Large-scale Multi-label Learning with Missing Labels
• Forward - Backward Greedy Algorithms for Atomic Norm Regularization
• High-dimensional estimation with geometric constraints
• Learning Single Index Models in High Dimensions
• Matrix Completion Under Monotonic Single Index Models

Cited by in corpus (2)

• Beyond the Signs: Nonparametric Tensor Completion via Sign Series
• Nonparametric Trace Regression in High Dimensions via Sign Series Representation