AutoML: A Survey of the State-of-the-Art

Abstract

Deep learning (DL) techniques have penetrated all aspects of our lives and brought us great convenience. However, building a high-quality DL system for a specific task highly relies on human expertise, hindering the applications of DL to more areas. Automated machine learning (AutoML) becomes a promising solution to build a DL system without human assistance, and a growing number of researchers focus on AutoML. In this paper, we provide a comprehensive and up-to-date review of the state-of-the-art (SOTA) in AutoML. First, we introduce AutoML methods according to the pipeline, covering data preparation, feature engineering, hyperparameter optimization, and neural architecture search (NAS). We focus more on NAS, as it is currently very hot sub-topic of AutoML. We summarize the performance of the representative NAS algorithms on the CIFAR-10 and ImageNet datasets and further discuss several worthy studying directions of NAS methods: one/two-stage NAS, one-shot NAS, and joint hyperparameter and architecture optimization. Finally, we discuss some open problems of the existing AutoML methods for future research.

automated machine learning (AutoML), published in journal of Knowledge-Based Systems

References in corpus (121)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Scikit-learn: Machine Learning in Python
• SMOTE: Synthetic Minority Over-sampling Technique
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Distilling the Knowledge in a Neural Network
• TensorFlow: A system for large-scale machine learning
• Practical Bayesian Optimization of Machine Learning Algorithms
• mixup: Beyond Empirical Risk Minimization
• Neural Architecture Search with Reinforcement Learning
• How transferable are features in deep neural networks?
• Categorical Reparameterization with Gumbel-Softmax
• Improved Regularization of Convolutional Neural Networks with Cutout
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• Neural Architecture Search: A Survey
• DARTS: Differentiable Architecture Search
• AMC: AutoML for Model Compression and Acceleration on Mobile Devices
• Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• Synthetic Data and Artificial Neural Networks for Natural Scene Text Recognition
• Data Augmentation Generative Adversarial Networks
• Searching for Activation Functions
• OpenAI Gym
• Efficient Neural Architecture Search via Parameter Sharing
• Data Synthesis based on Generative Adversarial Networks
• Large-Scale Evolution of Image Classifiers
• Neural Architecture Optimization
• Hyper-Parameter Optimization: A Review of Algorithms and Applications
• Scalable Bayesian Optimization Using Deep Neural Networks
• Designing Neural Network Architectures using Reinforcement Learning
• PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search
• Gradient-based Hyperparameter Optimization through Reversible Learning
• SMASH: One-Shot Model Architecture Search through HyperNetworks
• A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets
• GAN Augmentation: Augmenting Training Data using Generative Adversarial Networks
• Efficient Architecture Search by Network Transformation
• Neural Architecture Search with Bayesian Optimisation and Optimal Transport
• Understanding and Robustifying Differentiable Architecture Search
• Automated Machine Learning: From Principles to Practices
• DARTS+: Improved Differentiable Architecture Search with Early Stopping
• NAS-Bench-101: Towards Reproducible Neural Architecture Search
• Synthesizing Tabular Data using Generative Adversarial Networks
• One-Shot Neural Architecture Search via Self-Evaluated Template Network
• A Survey on Neural Architecture Search
• A Comprehensive Survey of Neural Architecture Search: Challenges and Solutions
• Mixed Precision Quantization of ConvNets via Differentiable Neural Architecture Search
• DetNAS: Backbone Search for Object Detection
• Data Noising as Smoothing in Neural Network Language Models
• Convolutional Neural Fabrics
• Evaluating the Search Phase of Neural Architecture Search
• Automated Machine Learning: State-of-The-Art and Open Challenges
• Path-Level Network Transformation for Efficient Architecture Search
• MONAS: Multi-Objective Neural Architecture Search using Reinforcement Learning
• Population Based Augmentation: Efficient Learning of Augmentation Policy Schedules
• DeepArchitect: Automatically Designing and Training Deep Architectures
• Network Pruning via Transformable Architecture Search
• Hyperparameter optimization with approximate gradient
• AutoML-Zero: Evolving Machine Learning Algorithms From Scratch
• Graph HyperNetworks for Neural Architecture Search
• Fixup Initialization: Residual Learning Without Normalization
• FRAGE: Frequency-Agnostic Word Representation
• NAS evaluation is frustratingly hard
• Adversarial AutoAugment
• BayesNAS: A Bayesian Approach for Neural Architecture Search
• Towards Automated Deep Learning: Efficient Joint Neural Architecture and Hyperparameter Search
• Fast, Accurate and Lightweight Super-Resolution with Neural Architecture Search
• Scalable Neural Architecture Search for 3D Medical Image Segmentation
• Peephole: Predicting Network Performance Before Training
• Simulation assisted machine learning
• FairNAS: Rethinking Evaluation Fairness of Weight Sharing Neural Architecture Search
• AutoGAN-Distiller: Searching to Compress Generative Adversarial Networks
• Auto-GNN: Neural Architecture Search of Graph Neural Networks
• MaskConnect: Connectivity Learning by Gradient Descent
• BoostClean: Automated Error Detection and Repair for Machine Learning
• Improving Neural Language Modeling via Adversarial Training
• Raiders of the Lost Architecture: Kernels for Bayesian Optimization in Conditional Parameter Spaces
• Semi-Supervised Neural Architecture Search
• AtomNAS: Fine-Grained End-to-End Neural Architecture Search
• Transfer Learning with Neural AutoML
• You Only Search Once: Single Shot Neural Architecture Search via Direct Sparse Optimization
• DADA: Differentiable Automatic Data Augmentation
• BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Models
• UniformAugment: A Search-free Probabilistic Data Augmentation Approach
• Layer-compensated Pruning for Resource-constrained Convolutional Neural Networks
• sharpDARTS: Faster and More Accurate Differentiable Architecture Search
• AlphaClean: Automatic Generation of Data Cleaning Pipelines
• Tabular Benchmarks for Joint Architecture and Hyperparameter Optimization
• Reinforced Evolutionary Neural Architecture Search
• Deep Active Learning with a Neural Architecture Search
• Early Stopping without a Validation Set
• Connectivity Learning in Multi-Branch Networks
• Auto-Meta: Automated Gradient Based Meta Learner Search
• Convolution by Evolution: Differentiable Pattern Producing Networks
• Bayesian Neural Architecture Search using A Training-Free Performance Metric
• Adversarial Text Generation Without Reinforcement Learning
• FBNetV3: Joint Architecture-Recipe Search using Predictor Pretraining
• AutoEmb: Automated Embedding Dimensionality Search in Streaming Recommendations
• Gradient Descent: The Ultimate Optimizer
• Differentiable Neural Input Search for Recommender Systems
• Evolving Robust Neural Architectures to Defend from Adversarial Attacks
• Real-time Federated Evolutionary Neural Architecture Search
• Evolutionary-Neural Hybrid Agents for Architecture Search
• Fast Neural Network Adaptation via Parameter Remapping and Architecture Search
• Exploring Shared Structures and Hierarchies for Multiple NLP Tasks
• Efficient Progressive Neural Architecture Search
• Deep Clustered Convolutional Kernels
• Modularized Morphing of Neural Networks
• Learning data augmentation policies using augmented random search
• Overcoming Multi-Model Forgetting
• NAS-Bench-NLP: Neural Architecture Search Benchmark for Natural Language Processing
• Bayesian Learning of Neural Network Architectures
• Towards modular and programmable architecture search
• IRS: A Large Naturalistic Indoor Robotics Stereo Dataset to Train Deep Models for Disparity and Surface Normal Estimation
• Neural Architecture Search using Deep Neural Networks and Monte Carlo Tree Search
• A Hybrid Differential Evolution Approach to Designing Deep Convolutional Neural Networks for Image Classification
• Automated Model Design and Benchmarking of 3D Deep Learning Models for COVID-19 Detection with Chest CT Scans
• VEGA: Towards an End-to-End Configurable AutoML Pipeline
• Style transfer-based image synthesis as an efficient regularization technique in deep learning
• EENA: Efficient Evolution of Neural Architecture
• Improving NeuroEvolution Efficiency by Surrogate Model-based Optimization with Phenotypic Distance Kernels
• Greedy AutoAugment

Cited by in corpus (135)

• The Medical Segmentation Decathlon
• Deep Neural Networks and Tabular Data: A Survey
• Medical Image Segmentation Using Deep Learning: A Survey
• EEG based Emotion Recognition: A Tutorial and Review
• Graph-based Deep Learning for Communication Networks: A Survey
• Galaxy Zoo DECaLS: Detailed Visual Morphology Measurements from Volunteers and Deep Learning for 314,000 Galaxies
• AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data
• Automated data processing and feature engineering for deep learning and big data applications: a survey
• Deep Learning in fNIRS: A review
• Automated machine learning: AI-driven decision making in business analytics
• Enable Deep Learning on Mobile Devices: Methods, Systems, and Applications
• Review: Deep Learning in Electron Microscopy
• Multi-Objective Hyperparameter Optimization in Machine Learning -- An Overview
• IoT Data Analytics in Dynamic Environments: From An Automated Machine Learning Perspective
• Learning to Optimize: A Primer and A Benchmark
• General Purpose Artificial Intelligence Systems (GPAIS): Properties, Definition, Taxonomy, Societal Implications and Responsible Governance
• Challenging the Security of Logic Locking Schemes in the Era of Deep Learning: A Neuroevolutionary Approach
• Automated Evolutionary Approach for the Design of Composite Machine Learning Pipelines
• Automated Machine Learning on Graphs: A Survey
• Universal Activation Function For Machine Learning
• Model fusion with physics-guided machine learning
• Telling Stories from Computational Notebooks: AI-Assisted Presentation Slides Creation for Presenting Data Science Work
• ktrain: A Low-Code Library for Augmented Machine Learning
• DUMA: Reading Comprehension with Transposition Thinking
• PaSca: a Graph Neural Architecture Search System under the Scalable Paradigm
• Recommender systems based on graph embedding techniques: A comprehensive review
• Artificial Intelligence in Industry 4.0: A Review of Integration Challenges for Industrial Systems
• Systematic Feature Design for Cycle Life Prediction of Lithium-Ion Batteries During Formation
• A Population-based Hybrid Approach to Hyperparameter Optimization for Neural Networks
• Multi-Microgrid Collaborative Optimization Scheduling Using an Improved Multi-Agent Soft Actor-Critic Algorithm
• An AutoML-based Approach to Multimodal Image Sentiment Analysis
• Data augmentation with automated machine learning: approaches and performance comparison with classical data augmentation methods
• Enabling AutoML for Zero-Touch Network Security: Use-Case Driven Analysis
• SliceMamba with Neural Architecture Search for Medical Image Segmentation
• Efficient End-to-End AutoML via Scalable Search Space Decomposition
• Towards Zero Touch Networks: Cross-Layer Automated Security Solutions for 6G Wireless Networks
• Physical Informed-Inspired Deep Reinforcement Learning Based Bi-Level Programming for Microgrid Scheduling
• Hubble Asteroid Hunter: I. Identifying asteroid trails in Hubble Space Telescope images
• Embedding Compression in Recommender Systems: A Survey
• DiffPrep: Differentiable Data Preprocessing Pipeline Search for Learning over Tabular Data
• Evaluating Generic Auto-ML Tools for Computational Pathology
• SubStrat: A Subset-Based Strategy for Faster AutoML
• PSViT: Better Vision Transformer via Token Pooling and Attention Sharing
• VolcanoML: Speeding up End-to-End AutoML via Scalable Search Space Decomposition
• Evolving Robust Neural Architectures to Defend from Adversarial Attacks
• Symbolic Regression as Feature Engineering Method for Machine and Deep Learning Regression Tasks
• A machine learning classifier for LOFAR radio galaxy cross-matching techniques
• AutoML using Metadata Language Embeddings
• Neural Feature Search for RGB-Infrared Person Re-Identification
• Why Should I Choose You? AutoXAI: A Framework for Selecting and Tuning eXplainable AI Solutions
• Evolutionary Architecture Search for Graph Neural Networks
• Towards Autonomous Cybersecurity: An Intelligent AutoML Framework for Autonomous Intrusion Detection
• Feature construction using explanations of individual predictions
• Vehicle Suspension Recommendation System: Multi-Fidelity Neural Network-based Mechanism Design Optimization
• Auto Seg-Loss: Searching Metric Surrogates for Semantic Segmentation
• Automating Outlier Detection via Meta-Learning
• SapientML: Synthesizing Machine Learning Pipelines by Learning from Human-Written Solutions
• Unified Question Generation with Continual Lifelong Learning
• Exploring Lottery Ticket Hypothesis in Media Recommender Systems
• Deep Ensembles for Low-Data Transfer Learning
• PEANUT: A Human-AI Collaborative Tool for Annotating Audio-Visual Data
• Automated Model Design and Benchmarking of 3D Deep Learning Models for COVID-19 Detection with Chest CT Scans
• Exploring the Intersection between Neural Architecture Search and Continual Learning
• Exploring hyper-parameter spaces of neuroscience models on high performance computers with Learning to Learn
• Multi-Objective Evolutionary Neural Architecture Search for Recurrent Neural Networks
• Experimental Learning of a Hyperelastic Behavior with a Physics-Augmented Neural Network
• Building Domain-Specific Machine Learning Workflows: A Conceptual Framework for the State-of-the-Practice
• Semantic-Based Neural Network Repair
• DIFER: Differentiable Automated Feature Engineering
• Neural Architecture Search for Generative Adversarial Networks: A Comprehensive Review and Critical Analysis
• Improving Machine Reading Comprehension with Single-choice Decision and Transfer Learning
• AINeedsPlanner: A Workbook to Support Effective Collaboration Between AI Experts and Clients
• Amazon SageMaker Autopilot: a white box AutoML solution at scale
• AutoAI-TS: AutoAI for Time Series Forecasting
• Towards AutoQML: A Cloud-Based Automated Circuit Architecture Search Framework
• AutoML to Date and Beyond: Challenges and Opportunities
• Capturing Dependencies within Machine Learning via a Formal Process Model
• Modyn: Data-Centric Machine Learning Pipeline Orchestration
• VEGA: Towards an End-to-End Configurable AutoML Pipeline
• Hybrid Self-Attention NEAT: A novel evolutionary approach to improve the NEAT algorithm
• Conceptualization and Framework of Hybrid Intelligence Systems
• Graph Neural Network Architecture Search for Molecular Property Prediction
• Human-Centered AI for Data Science: A Systematic Approach
• Achieving Real-Time LiDAR 3D Object Detection on a Mobile Device
• An Overview and Solution for Democratizing AI Workflows at the Network Edge
• FADNet++: Real-Time and Accurate Disparity Estimation with Configurable Networks
• Predicting delays in Indian lower courts using AutoML and Decision Forests
• Automating Generative Deep Learning for Artistic Purposes: Challenges and Opportunities
• Automated Contrastive Learning Strategy Search for Time Series
• ARDA: Automatic Relational Data Augmentation for Machine Learning
• Genetic Neural Architecture Search for automatic assessment of human sperm images
• Intelligent Learning Rate Distribution to reduce Catastrophic Forgetting in Transformers
• AutoLoss-Zero: Searching Loss Functions from Scratch for Generic Tasks
• Using Meta-learning to Recommend Process Discovery Methods
• Benchmarking Multimodal AutoML for Tabular Data with Text Fields
• Parameter Prediction for Unseen Deep Architectures
• Towards Automatic Actor-Critic Solutions to Continuous Control
• FlatNAS: optimizing Flatness in Neural Architecture Search for Out-of-Distribution Robustness
• The Deep Promotion Time Cure Model
• Classification of integers based on residue classes via modern deep learning algorithms
• SAFE: Scalable Automatic Feature Engineering Framework for Industrial Tasks
• MS-NAS: Multi-Scale Neural Architecture Search for Medical Image Segmentation
• Automated machine learning for borehole resistivity measurements
• Interpret-able feedback for AutoML systems
• "I see models being a whole other thing": An Empirical Study of Pre-Trained Model Naming Conventions and A Tool for Enhancing Naming Consistency
• MONCAE: Multi-Objective Neuroevolution of Convolutional Autoencoders
• Automated Machine Learning, Bounded Rationality, and Rational Metareasoning
• Chameleon: A Semi-AutoML framework targeting quick and scalable development and deployment of production-ready ML systems for SMEs
• Transfer Learning Based Automatic Model Creation Tool For Resource Constraint Devices
• Privileged Zero-Shot AutoML
• TND-NAS: Towards Non-differentiable Objectives in Progressive Differentiable NAS Framework
• Capturing and Anticipating User Intents in Data Analytics via Knowledge Graphs
• AIPerf: Automated machine learning as an AI-HPC benchmark
• Dataset-Adaptive Dimensionality Reduction
• Man versus Machine: AutoML and Human Experts' Role in Phishing Detection
• Selecting Optimal Trace Clustering Pipelines with AutoML
• AutoSmart: An Efficient and Automatic Machine Learning framework for Temporal Relational Data
• Data science and AI in FinTech: An overview
• DriveML: An R Package for Driverless Machine Learning
• Fast IR Drop Estimation with Machine Learning
• Design of borehole resistivity measurement acquisition systems using deep learning
• The Efficiency Misnomer
• Tightening the Approximation Error of Adversarial Risk with Auto Loss Function Search
• DIVA: Dataset Derivative of a Learning Task
• AutoDC: Automated data-centric processing
• CUAB: Convolutional Uncertainty Attention Block Enhanced the Chest X-ray Image Analysis
• Neural Architecture Search based on Cartesian Genetic Programming Coding Method
• FLIM Networks with Bag of Feature Points
• Constraint-Aware Discrete Black-Box Optimization Using Tensor Decomposition
• Neural Architecture Search for Sentence Classification with BERT
• Computer-Aided Clinical Skin Disease Diagnosis Using CNN and Object Detection Models
• Dynamic Surrogate Switching: Sample-Efficient Search for Factorization Machine Configurations in Online Recommendations
• How saccadic vision might help with theinterpretability of deep networks
• LETI: Latency Estimation Tool and Investigation of Neural Networks inference on Mobile GPU
• Towards autonomic orchestration of machine learning pipelines in future networks