Feature-Weighted Linear Stacking

Abstract

Ensemble methods, such as stacking, are designed to boost predictive accuracy by blending the predictions of multiple machine learning models. Recent work has shown that the use of meta-features, additional inputs describing each example in a dataset, can boost the performance of ensemble methods, but the greatest reported gains have come from nonlinear procedures requiring significant tuning and training time. Here, we present a linear technique, Feature-Weighted Linear Stacking (FWLS), that incorporates meta-features for improved accuracy while retaining the well-known virtues of linear regression regarding speed, stability, and interpretability. FWLS combines model predictions linearly using coefficients that are themselves linear functions of meta-features. This technique was a key facet of the solution of the second place team in the recently concluded Netflix Prize competition. Significant increases in accuracy over standard linear stacking are demonstrated on the Netflix Prize collaborative filtering dataset.

17 pages, 1 figure, 2 tables

References in corpus (1)

• Stacking classifiers for anti-spam filtering of e-mail

Cited by in corpus (17)

• Prediction of infectious disease epidemics via weighted density ensembles
• Bayesian hierarchical stacking: Some models are (somewhere) useful
• The NN-Stacking: Feature weighted linear stacking through neural networks
• Multilabel Classification with R Package mlr
• The Monkeytyping Solution to the YouTube-8M Video Understanding Challenge
• Kaggle LSHTC4 Winning Solution
• One-at-a-time: A Meta-Learning Recommender-System for Recommendation-Algorithm Selection on Micro Level
• Max-Margin Stacking and Sparse Regularization for Linear Classifier Combination and Selection
• MetaSelector: Meta-Learning for Recommendation with User-Level Adaptive Model Selection
• On hybrid modular recommendation systems for video streaming
• MNIST-NET10: A heterogeneous deep networks fusion based on the degree of certainty to reach 0.1 error rate. Ensembles overview and proposal
• A Bayes interpretation of stacking for M-complete and M-open settings
• User-Sensitive Recommendation Ensemble with Clustered Multi-Task Learning
• Improving incremental recommenders with online bagging
• Combining parametric, semi-parametric, and non-parametric survival models with stacked survival models
• ReINTEL Challenge 2020: A Comparative Study of Hybrid Deep Neural Network for Reliable Intelligence Identification on Vietnamese SNSs
• Machine Learning and Deep Learning methods for predictive modelling from Raman spectra in bioprocessing