Simpler Hyperparameter Optimization for Software Analytics: Why, How, When?
arXiv:1912.04061 · doi:10.1109/TSE.2021.3073242
Abstract
How can we make software analytics simpler and faster? One method is to match the complexity of analysis to the intrinsic complexity of the data being explored. For example, hyperparameter optimizers find the control settings for data miners that improve the predictions generated via software analytics. Sometimes, very fast hyperparameter optimization can be achieved by "DODGE-ing"; i.e. simply steering way from settings that lead to similar conclusions. But when is it wise to use that simple approach and when must we use more complex (and much slower) optimizers?} To answer this, we applied hyperparameter optimization to 120 SE data sets that explored bad smell detection, predicting Github issue close time, bug report analysis, defect prediction, and dozens of other non-SE problems. We find that the simple DODGE works best for data sets with low "intrinsic dimensionality" (u ~ 3) and very poorly for higher-dimensional data (u > 8). Nearly all the SE data seen here was intrinsically low-dimensional, indicating that DODGE is applicable for many SE analytics tasks.
arXiv comment15 pages
References in corpus (10)
- Scikit-learn: Machine Learning in Python
- NLTK: The Natural Language Toolkit
- What is Wrong with Topic Modeling? (and How to Fix it Using Search-based Software Engineering)
- Tuning for Software Analytics: is it Really Necessary?
- Is "Better Data" Better than "Better Data Miners"? (On the Benefits of Tuning SMOTE for Defect Prediction)
- How to "DODGE" Complex Software Analytics?
- Why is Differential Evolution Better than Grid Search for Tuning Defect Predictors?
- Less is More: Minimizing Code Reorganization using XTREE
- Secure Deep Learning Engineering: A Software Quality Assurance Perspective
- Better Predictors for Issue Lifetime