Multi-Target Regression via Input Space Expansion: Treating Targets as Inputs

Abstract

In many practical applications of supervised learning the task involves the prediction of multiple target variables from a common set of input variables. When the prediction targets are binary the task is called multi-label classification, while when the targets are continuous the task is called multi-target regression. In both tasks, target variables often exhibit statistical dependencies and exploiting them in order to improve predictive accuracy is a core challenge. A family of multi-label classification methods address this challenge by building a separate model for each target on an expanded input space where other targets are treated as additional input variables. Despite the success of these methods in the multi-label classification domain, their applicability and effectiveness in multi-target regression has not been studied until now. In this paper, we introduce two new methods for multi-target regression, called Stacked Single-Target and Ensemble of Regressor Chains, by adapting two popular multi-label classification methods of this family. Furthermore, we highlight an inherent problem of these methods - a discrepancy of the values of the additional input variables between training and prediction - and develop extensions that use out-of-sample estimates of the target variables during training in order to tackle this problem. The results of an extensive experimental evaluation carried out on a large and diverse collection of datasets show that, when the discrepancy is appropriately mitigated, the proposed methods attain consistent improvements over the independent regressions baseline. Moreover, two versions of Ensemble of Regression Chains perform significantly better than four state-of-the-art methods including regularization-based multi-task learning methods and a multi-objective random forest approach.

Accepted for publication in Machine Learning journal. This replacement contains major improvements compared to the previous version, including a deeper theoretical and experimental analysis and an extended discussion of related work

References in corpus (5)

• Clustered Multi-Task Learning: A Convex Formulation
• Efficient Monte Carlo Methods for Multi-Dimensional Learning with Classifier Chains
• Multi-Target Regression via Random Linear Target Combinations
• Learning Internal Representations (COLT 1995)
• The Landmark Selection Method for Multiple Output Prediction

Cited by in corpus (18)

• A New Learning Paradigm for Random Vector Functional-Link Network: RVFL+
• Measurements of $\mathrm{t}\bar{\mathrm{t}}$H production and the CP structure of the Yukawa interaction between the Higgs boson and the top quark in the diphoton decay channel
• Measurements of Higgs boson production cross sections and couplings in the diphoton decay channel at $\sqrt{s}=$ 13 TeV
• Search for nonresonant Higgs boson pair production in final states with two bottom quarks and two photons in proton-proton collisions at $\sqrt{s}=$ 13 TeV
• Investigating the dark matter signal in the cosmic ray antiproton flux with the machine learning method
• Search for Higgs boson decays to a Z boson and a photon in proton-proton collisions at $\sqrt{s}$ = 13 TeV
• Deep tree-ensembles for multi-output prediction
• Condensed Gradient Boosting
• Weighting Scheme for a Pairwise Multi-label Classifier Based on the Fuzzy Confusion Matrix
• Search for flavor-changing neutral current interactions of the top quark and Higgs boson in final states with two photons in proton-proton collisions at $\sqrt{s}=$ 13 TeV
• Machine learning for predicting thermal power consumption of the Mars Express Spacecraft
• Fast and Accurate Pseudoinverse with Sparse Matrix Reordering and Incremental Approach
• Measurement of the Higgs boson inclusive and differential fiducial production cross sections in the diphoton decay channel with pp collisions at $\sqrt{s}$ = 13 TeV
• A Correction Method of a Binary Classifier Applied to Multi-label Pairwise Models
• Dynamic classifier chains for multi-label learning
• Search for a new resonance decaying into two spin-0 bosons in a final state with two photons and two bottom quarks in proton-proton collisions at $\sqrt{s}$ = 13 TeV
• Search for a cH signal in the associated production of at least one charm quark with a Higgs boson in the diphoton decay channel in pp collisions at $\sqrt{s}$ = 13 TeV
• Search for a narrow resonance with a mass between 10 and 70 GeV decaying to a pair of photons in proton-proton collisions at $\sqrt{s}$ = 13 TeV