Popular Ensemble Methods: An Empirical Study

Abstract

An ensemble consists of a set of individually trained classifiers (such as neural networks or decision trees) whose predictions are combined when classifying novel instances. Previous research has shown that an ensemble is often more accurate than any of the single classifiers in the ensemble. Bagging (Breiman, 1996c) and Boosting (Freund and Shapire, 1996; Shapire, 1990) are two relatively new but popular methods for producing ensembles. In this paper we evaluate these methods on 23 data sets using both neural networks and decision trees as our classification algorithm. Our results clearly indicate a number of conclusions. First, while Bagging is almost always more accurate than a single classifier, it is sometimes much less accurate than Boosting. On the other hand, Boosting can create ensembles that are less accurate than a single classifier -- especially when using neural networks. Analysis indicates that the performance of the Boosting methods is dependent on the characteristics of the data set being examined. In fact, further results show that Boosting ensembles may overfit noisy data sets, thus decreasing its performance. Finally, consistent with previous studies, our work suggests that most of the gain in an ensemble's performance comes in the first few classifiers combined; however, relatively large gains can be seen up to 25 classifiers when Boosting decision trees.

Cited by in corpus (139)

• Haplotype-based variant detection from short-read sequencing
• A Survey on Distributed Machine Learning
• A deep learning system for differential diagnosis of skin diseases
• Neural network an1alysis of sleep stages enables efficient diagnosis of narcolepsy
• Fully Convolutional Network Ensembles for White Matter Hyperintensities Segmentation in MR Images
• Detecting Anemia from Retinal Fundus Images
• Assessment of supervised machine learning methods for fluid flows
• Uncertainty Quantification in Machine Learning for Engineering Design and Health Prognostics: A Tutorial
• Deep Learning and Glaucoma Specialists: The Relative Importance of Optic Disc Features to Predict Glaucoma Referral in Fundus Photos
• Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning
• Fusion of Probability Density Functions
• Grammatical Error Correction: A Survey of the State of the Art
• The Heterogeneous Ensembles of Standard Classification Algorithms (HESCA): the Whole is Greater than the Sum of its Parts
• BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning
• A novel approach for the automated segmentation and volume quantification of cardiac fats on computed tomography
• Ensemble Deep Learning on Large, Mixed-Site fMRI Datasets in Autism and Other Tasks
• NOMAD 2018 Kaggle Competition: Solving Materials Science Challenges Through Crowd Sourcing
• (Machine) Learning to Do More with Less
• ResKD: Residual-Guided Knowledge Distillation
• An empirical analysis of dropout in piecewise linear networks
• Ensemble Learning of Coarse-Grained Molecular Dynamics Force Fields with a Kernel Approach
• Credit spread approximation and improvement using random forest regression
• Triaging Content Severity in Online Mental Health Forums
• Let Features Decide for Themselves: Feature Mask Network for Person Re-identification
• Defending Model Inversion and Membership Inference Attacks via Prediction Purification
• Multi-label ensemble based on variable pairwise constraint projection
• Uncertainty-Aware Deep Ensembles for Reliable and Explainable Predictions of Clinical Time Series
• A black-box adversarial attack for poisoning clustering
• BERT-based Financial Sentiment Index and LSTM-based Stock Return Predictability
• Finite Versus Infinite Neural Networks: an Empirical Study
• Improved deep learning techniques in gravitational-wave data analysis
• How Far Does the Predictive Decision Impact the Software Project? The Cost, Service Time, and Failure Analysis from a Cross-Project Defect Prediction Model
• Efficient Adaptive Ensembling for Image Classification
• Machine Learning-Driven Process of Alumina Ceramics Laser Machining
• Hyperparameter Ensembles for Robustness and Uncertainty Quantification
• The Evolution of Reinforcement Learning in Quantitative Finance: A Survey
• Autostacker: A Compositional Evolutionary Learning System
• Recent Advances in Deep Learning for Object Detection
• Dalek -- a deep-learning emulator for TARDIS
• Histopathologic Image Processing: A Review
• An Online Ensemble Learning Model for Detecting Attacks in Wireless Sensor Networks
• HyperGAN: A Generative Model for Diverse, Performant Neural Networks
• Pulsars Detection by Machine Learning with Very Few Features
• Succinct Differentiation of Disparate Boosting Ensemble Learning Methods for Prognostication of Polycystic Ovary Syndrome Diagnosis
• Morphological classifiers
• Predicting the redshift of gamma-ray loud AGNs using Supervised Machine Learning: Part 2
• Uncertainty Quantification for Virtual Diagnostic of Particle Accelerators
• Kalman Filter-based Heuristic Ensemble (KFHE): A new perspective on multi-class ensemble classification using Kalman filters
• Fast Sequence Component Analysis for Attack Detection in Synchrophasor Networks
• VisRuler: Visual Analytics for Extracting Decision Rules from Bagged and Boosted Decision Trees
• Double-Base Asymmetric AdaBoost
• The semantic similarity ensemble
• Early Exiting with Ensemble Internal Classifiers
• Cosmic String Detection with Tree-Based Machine Learning
• Person Recognition using Smartphones' Accelerometer Data
• Forecasting COVID-19 spreading trough an ensemble of classical and machine learning models: Spain's case study
• EnsembleNet: End-to-End Optimization of Multi-headed Models
• Morphology-assisted galaxy mass-to-light predictions using deep learning
• Candidate high-redshift protoclusters and lensed galaxies in the Planck list of high-z sources overlapping with Herschel-SPIRE imaging
• Bringing Cartoons to Life: Towards Improved Cartoon Face Detection and Recognition Systems
• Face Detection Using Adaboosted SVM-Based Component Classifier
• Graph Encoder Ensemble for Simultaneous Vertex Embedding and Community Detection
• Algorithm Selection for Combinatorial Search Problems: A Survey
• White paper: The Helix Pathogenicity Prediction Platform
• Authorship Identification of Source Code Segments Written by Multiple Authors Using Stacking Ensemble Method
• On the Certified Robustness for Ensemble Models and Beyond
• Visualizing Ensemble Predictions of Music Mood
• A Data-Driven Two-Phase Multi-Split Causal Ensemble Model for Time Series
• CIFAR-10: KNN-based Ensemble of Classifiers
• Language Detection For Short Text Messages In Social Media
• Improving Retrieval Modeling Using Cross Convolution Networks And Multi Frequency Word Embedding
• Features in Concert: Discriminative Feature Selection meets Unsupervised Clustering
• Strength in Numbers: Trading-off Robustness and Computation via Adversarially-Trained Ensembles
• Elsa: Energy-based learning for semi-supervised anomaly detection
• Rank of Experts: Detection Network Ensemble
• No One Representation to Rule Them All: Overlapping Features of Training Methods
• Dealing with Adversarial Player Strategies in the Neural Network Game iNNk through Ensemble Learning
• Fingerprinting Encrypted Voice Traffic on Smart Speakers with Deep Learning
• Detecting Colorized Images via Convolutional Neural Networks: Toward High Accuracy and Good Generalization
• Sensor-Type Classification in Buildings
• Interpretable Models Capable of Handling Systematic Missingness in Imbalanced Classes and Heterogeneous Datasets
• Inverse renormalization group of spin glasses
• An Empirical Study of the L2-Boost technique with Echo State Networks
• An Outlier Detection-based Tree Selection Approach to Extreme Pruning of Random Forests
• On Extreme Pruning of Random Forest Ensembles for Real-time Predictive Applications
• Artificial Intelligence Powered Material Search Engine
• Comprehensive List Generation for Multi-Generator Reranking
• Relationship between Diversity and Perfomance of Multiple Classifiers for Decision Support
• Per-Instance Algorithm Selection for Recommender Systems via Instance Clustering
• UNISELF: A Unified Network with Instance Normalization and Self-Ensembled Lesion Fusion for Multiple Sclerosis Lesion Segmentation
• Resource Allocation Using Gradient Boosting Aided Deep Q-Network for IoT in C-RANs
• Human Vs. Machines: Who Wins In Semiconductor Market Forecasting?
• Solving physics-based initial value problems with unsupervised machine learning
• Selective Ensembles for Consistent Predictions
• Competing AI: How does competition feedback affect machine learning?
• Can you tell? SSNet -- a Sagittal Stratum-inspired Neural Network Framework for Sentiment Analysis
• Principal Model Analysis Based on Partial Least Squares
• Unifying Ensemble Methods for Q-learning via Social Choice Theory
• Machine Learning Approach to Earthquake Rupture Dynamics
• CoMID: Context-based Multi-Invariant Detection for Monitoring Cyber-Physical Software
• Fused Deep Convolutional Neural Network for Precision Diagnosis of COVID-19 Using Chest X-Ray Images
• Theoretical Investigation of Composite Neural Network
• Web Question Answering with Neurosymbolic Program Synthesis
• Ensemble Pruning via Margin Maximization
• Ubiquitous Indoor Localization and Worldwide Automatic Construction of Floor Plans
• On the Current State of Research in Explaining Ensemble Performance Using Margins
• Ensemble of Distributed Learners for Online Classification of Dynamic Data Streams
• IEA: Inner Ensemble Average within a convolutional neural network
• A Machine Learning Perspective on Predictive Coding with PAQ
• Sparse MoEs meet Efficient Ensembles
• Quantum Boosting using Domain-Partitioning Hypotheses
• Osteoporosis Prescreening using Panoramic Radiographs through a Deep Convolutional Neural Network with Attention Mechanism
• Predictive No-Reference Assessment of Video Quality
• Personalization in Human-AI Teams: Improving the Compatibility-Accuracy Tradeoff
• On the Insufficiency of the Large Margins Theory in Explaining the Performance of Ensemble Methods
• Minimally Supervised Feature Selection for Classification (Master's Thesis, University Politehnica of Bucharest)
• Multi-Scale Distributed Representation for Deep Learning and its Application to b-Jet Tagging
• Data-informed Emulators for Multi-Physics Simulations
• Predictive Modeling of Hospital Readmission: Challenges and Solutions
• Federated Bayesian Network Ensembles
• Exploiting a Zoo of Checkpoints for Unseen Tasks
• Combining Different V1 Brain Model Variants to Improve Robustness to Image Corruptions in CNNs
• Bayesian Model Averaging for Ensemble-Based Estimates of Solvation Free Energies
• Probabilistic Diagnostic Tests for Degradation Problems in Supervised Learning
• Combining Breast Cancer Risk Prediction Models
• Portfolio optimization using local linear regression ensembles in RapidMiner
• Inner Ensemble Networks: Average Ensemble as an Effective Regularizer
• Stacked Generalization for Human Activity Recognition
• Building an NCAA mens basketball predictive model and quantifying its success
• SEPP: Similarity Estimation of Predicted Probabilities for Defending and Detecting Adversarial Text
• Trained Model Fusion for Object Detection using Gating Network
• Model ensembles of artificial neural networks and support vector regression for improved accuracy in the prediction of vegetation conditions
• Ensembles of Random Sphere Cover Classifiers
• Data Quality Measures and Efficient Evaluation Algorithms for Large-Scale High-Dimensional Data
• Continuous-time Markov-switching GARCH Process with Robust and Efficient State Path and Volatility Estimation
• Place-specific Background Modeling Using Recursive Autoencoders
• Unsupervised Cross-Lingual Speech Emotion Recognition Using Pseudo Multilabel
• Duration-of-Stay Storage Assignment under Uncertainty
• Vote-boosting ensembles