DARTS: Differentiable Architecture Search

Abstract

This paper addresses the scalability challenge of architecture search by formulating the task in a differentiable manner. Unlike conventional approaches of applying evolution or reinforcement learning over a discrete and non-differentiable search space, our method is based on the continuous relaxation of the architecture representation, allowing efficient search of the architecture using gradient descent. Extensive experiments on CIFAR-10, ImageNet, Penn Treebank and WikiText-2 show that our algorithm excels in discovering high-performance convolutional architectures for image classification and recurrent architectures for language modeling, while being orders of magnitude faster than state-of-the-art non-differentiable techniques. Our implementation has been made publicly available to facilitate further research on efficient architecture search algorithms.

arXiv commentPublished at ICLR 2019; Code and pretrained models available at https://github.com/quark0/darts

Cited by in corpus (587)

• Neural Architecture Search: A Survey
• AutoML: A Survey of the State-of-the-Art
• Automatically designing CNN architectures using genetic algorithm for image classification
• A Survey of Model Compression and Acceleration for Deep Neural Networks
• Multi-Task Learning for Dense Prediction Tasks: A Survey
• Generalizing from a Few Examples: A Survey on Few-Shot Learning
• PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search
• A Survey on Deep Learning Techniques for Stereo-based Depth Estimation
• ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware
• SNAS: Stochastic Neural Architecture Search
• MCUNet: Tiny Deep Learning on IoT Devices
• NAS-FAS: Static-Dynamic Central Difference Network Search for Face Anti-Spoofing
• Understanding and Robustifying Differentiable Architecture Search
• DARTS+: Improved Differentiable Architecture Search with Early Stopping
• Random Search and Reproducibility for Neural Architecture Search
• Revealing the Invisible with Model and Data Shrinking for Composite-database Micro-expression Recognition
• HAT: Hardware-Aware Transformers for Efficient Natural Language Processing
• One-Shot Neural Architecture Search via Self-Evaluated Template Network
• Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis
• A Survey on Neural Architecture Search
• True Few-Shot Learning with Language Models
• Mixed Precision Quantization of ConvNets via Differentiable Neural Architecture Search
• Neural Architecture Transfer
• DetNAS: Backbone Search for Object Detection
• NATS-Bench: Benchmarking NAS Algorithms for Architecture Topology and Size
• Evaluating the Search Phase of Neural Architecture Search
• Differentiable Quantum Architecture Search
• Automated Machine Learning: State-of-The-Art and Open Challenges
• Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentation
• MicroNets: Neural Network Architectures for Deploying TinyML Applications on Commodity Microcontrollers
• AdaShare: Learning What To Share For Efficient Deep Multi-Task Learning
• Network Pruning via Transformable Architecture Search
• Searching Multi-Rate and Multi-Modal Temporal Enhanced Networks for Gesture Recognition
• A Practical Survey on Faster and Lighter Transformers
• Carbon Emissions and Large Neural Network Training
• A Comprehensive Study of Deep Video Action Recognition
• Validation of the rapid detection approach for enhancing the electronic nose systems performance, using different deep learning models and support vector machines
• NAS evaluation is frustratingly hard
• BRP-NAS: Prediction-based NAS using GCNs
• Once-for-All: Train One Network and Specialize it for Efficient Deployment
• MetaPruning: Meta Learning for Automatic Neural Network Channel Pruning
• Trellis Networks for Sequence Modeling
• Recent Progress in the CUHK Dysarthric Speech Recognition System
• Neural Predictor based Quantum Architecture Search
• Best Practices for Scientific Research on Neural Architecture Search
• Coresets via Bilevel Optimization for Continual Learning and Streaming
• An overview of mixing augmentation methods and augmentation strategies
• SpArSe: Sparse Architecture Search for CNNs on Resource-Constrained Microcontrollers
• NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search
• Stabilizing Differentiable Architecture Search via Perturbation-based Regularization
• Exploring Randomly Wired Neural Networks for Image Recognition
• NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Search
• Bayesian Bits: Unifying Quantization and Pruning
• Applications and Techniques for Fast Machine Learning in Science
• Deep Learning for Image Super-resolution: A Survey
• Dynamic Convolution: Attention over Convolution Kernels
• NSGA-Net: Neural Architecture Search using Multi-Objective Genetic Algorithm
• GraphNAS: Graph Neural Architecture Search with Reinforcement Learning
• Probabilistic Neural Architecture Search
• Automated Machine Learning on Graphs: A Survey
• BNAS:An Efficient Neural Architecture Search Approach Using Broad Scalable Architecture
• Bridging the Gap between Sample-based and One-shot Neural Architecture Search with BONAS
• A System for Massively Parallel Hyperparameter Tuning
• Zero-Cost Proxies for Lightweight NAS
• Efficient Multi-objective Neural Architecture Search via Lamarckian Evolution
• DiffMG: Differentiable Meta Graph Search for Heterogeneous Graph Neural Networks
• EEEA-Net: An Early Exit Evolutionary Neural Architecture Search
• ASAP: Architecture Search, Anneal and Prune
• Auto-GNN: Neural Architecture Search of Graph Neural Networks
• CrypTen: Secure Multi-Party Computation Meets Machine Learning
• On Neural Architecture Search for Resource-Constrained Hardware Platforms
• A Comprehensive Survey on Hardware-Aware Neural Architecture Search
• Effective and Efficient Dropout for Deep Convolutional Neural Networks
• NAS-BERT: Task-Agnostic and Adaptive-Size BERT Compression with Neural Architecture Search
• Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective
• Toward Extremely Lightweight Distracted Driver Recognition With Distillation-Based Neural Architecture Search and Knowledge Transfer
• MCUNetV2: Memory-Efficient Patch-based Inference for Tiny Deep Learning
• Towards Green Automated Machine Learning: Status Quo and Future Directions
• Learning Generalisable Omni-Scale Representations for Person Re-Identification
• Inspect, Understand, Overcome: A Survey of Practical Methods for AI Safety
• Winning the Lottery with Continuous Sparsification
• Model Rubik's Cube: Twisting Resolution, Depth and Width for TinyNets
• EPE-NAS: Efficient Performance Estimation Without Training for Neural Architecture Search
• Language Models with Transformers
• 3D-ANAS: 3D Asymmetric Neural Architecture Search for Fast Hyperspectral Image Classification
• Tuning Hyperparameters without Grad Students: Scalable and Robust Bayesian Optimisation with Dragonfly
• BinaryBERT: Pushing the Limit of BERT Quantization
• Stabilizing DARTS with Amended Gradient Estimation on Architectural Parameters
• DAIS: Automatic Channel Pruning via Differentiable Annealing Indicator Search
• Neural Architecture Search without Training
• Auto-ReID: Searching for a Part-aware ConvNet for Person Re-Identification
• AtomNAS: Fine-Grained End-to-End Neural Architecture Search
• Learning to Branch for Multi-Task Learning
• Neural Architecture Search For Keyword Spotting
• DADA: Differentiable Automatic Data Augmentation
• Dynamic Neural Networks: A Survey
• ResizeMix: Mixing Data with Preserved Object Information and True Labels
• Cream of the Crop: Distilling Prioritized Paths For One-Shot Neural Architecture Search
• Noisy Differentiable Architecture Search
• AutoFIS: Automatic Feature Interaction Selection in Factorization Models for Click-Through Rate Prediction
• ISTA-NAS: Efficient and Consistent Neural Architecture Search by Sparse Coding
• Searching Efficient 3D Architectures with Sparse Point-Voxel Convolution
• Learning Deformable Image Registration from Optimization: Perspective, Modules, Bilevel Training and Beyond
• Deeper Insights into Weight Sharing in Neural Architecture Search
• sharpDARTS: Faster and More Accurate Differentiable Architecture Search
• Searching for Low-Bit Weights in Quantized Neural Networks
• Does Unsupervised Architecture Representation Learning Help Neural Architecture Search?
• Towards Better Accuracy-efficiency Trade-offs: Divide and Co-training
• Tabular Benchmarks for Joint Architecture and Hyperparameter Optimization
• Reinforced Evolutionary Neural Architecture Search
• Rethinking Architecture Selection in Differentiable NAS
• Exploring Architectural Ingredients of Adversarially Robust Deep Neural Networks
• Fair DARTS: Eliminating Unfair Advantages in Differentiable Architecture Search
• DARTS-: Robustly Stepping out of Performance Collapse Without Indicators
• Computation Reallocation for Object Detection
• Improving Auto-Augment via Augmentation-Wise Weight Sharing
• GOLD-NAS: Gradual, One-Level, Differentiable
• Is it enough to optimize CNN architectures on ImageNet?
• Understanding Neural Architecture Search Techniques
• A Survey of Modern Object Detection Literature using Deep Learning
• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• EvoPose2D: Pushing the Boundaries of 2D Human Pose Estimation using Accelerated Neuroevolution with Weight Transfer
• Triple-level Model Inferred Collaborative Network Architecture for Video Deraining
• Learning Graph Convolutional Network for Skeleton-based Human Action Recognition by Neural Searching
• Constrained Design of Deep Iris Networks
• One Proxy Device Is Enough for Hardware-Aware Neural Architecture Search
• Sub-Architecture Ensemble Pruning in Neural Architecture Search
• Search to aggregate neighborhood for graph neural network
• CARS: Continuous Evolution for Efficient Neural Architecture Search
• AutoHAS: Efficient Hyperparameter and Architecture Search
• Theory-Inspired Path-Regularized Differential Network Architecture Search
• Involution: Inverting the Inherence of Convolution for Visual Recognition
• AdaBERT: Task-Adaptive BERT Compression with Differentiable Neural Architecture Search
• Joint Multi-Dimension Pruning via Numerical Gradient Update
• VarGNet: Variable Group Convolutional Neural Network for Efficient Embedded Computing
• HW-NAS-Bench:Hardware-Aware Neural Architecture Search Benchmark
• How to Train Your Super-Net: An Analysis of Training Heuristics in Weight-Sharing NAS
• Diverse Branch Block: Building a Convolution as an Inception-like Unit
• Locally Free Weight Sharing for Network Width Search
• EfficientTDNN: Efficient Architecture Search for Speaker Recognition
• Accelerate CNNs from Three Dimensions: A Comprehensive Pruning Framework
• Probabilistic Dual Network Architecture Search on Graphs
• A Closer Look at Structured Pruning for Neural Network Compression
• Towards Efficient Post-training Quantization of Pre-trained Language Models
• Model-based Asynchronous Hyperparameter and Neural Architecture Search
• CurveLane-NAS: Unifying Lane-Sensitive Architecture Search and Adaptive Point Blending
• DCNAS: Densely Connected Neural Architecture Search for Semantic Image Segmentation
• SpineNet: Learning Scale-Permuted Backbone for Recognition and Localization
• Blurring the Line Between Structure and Learning to Optimize and Adapt Receptive Fields
• Progressive DARTS: Bridging the Optimization Gap for NAS in the Wild
• STEERAGE: Synthesis of Neural Networks Using Architecture Search and Grow-and-Prune Methods
• Learnable Embedding Sizes for Recommender Systems
• Differentiable Neural Input Search for Recommender Systems
• When NAS Meets Robustness: In Search of Robust Architectures against Adversarial Attacks
• Domain-aware Visual Bias Eliminating for Generalized Zero-Shot Learning
• Splitting Steepest Descent for Growing Neural Architectures
• Joint Neural Architecture Search and Quantization
• C2FNAS: Coarse-to-Fine Neural Architecture Search for 3D Medical Image Segmentation
• MANAS: Multi-Agent Neural Architecture Search
• Neural Ensemble Search for Uncertainty Estimation and Dataset Shift
• Learned Low Precision Graph Neural Networks
• Neural Feature Search for RGB-Infrared Person Re-Identification
• Neural Architecture Generator Optimization
• Interpretable Neural Architecture Search via Bayesian Optimisation with Weisfeiler-Lehman Kernels
• Rethinking Co-design of Neural Architectures and Hardware Accelerators
• Self-supervised Cross-silo Federated Neural Architecture Search
• How Powerful are Performance Predictors in Neural Architecture Search?
• Multi-Objective Evolutionary Design of Deep Convolutional Neural Networks for Image Classification
• Evolutionary Neural Architecture Search Supporting Approximate Multipliers
• Bag of Baselines for Multi-objective Joint Neural Architecture Search and Hyperparameter Optimization
• Knowledge distillation via adaptive instance normalization
• Real-time Federated Evolutionary Neural Architecture Search
• The Deep Bootstrap Framework: Good Online Learners are Good Offline Generalizers
• How degenerate is the parametrization of neural networks with the ReLU activation function?
• Multi-modal Visual Tracking: Review and Experimental Comparison
• Lower Bounds and Accelerated Algorithms for Bilevel Optimization
• MixPath: A Unified Approach for One-shot Neural Architecture Search
• AutoOD: Automated Outlier Detection via Curiosity-guided Search and Self-imitation Learning
• One-Shot Neural Architecture Search via Compressive Sensing
• Multi-branch Attentive Transformer
• Evolutionary-Neural Hybrid Agents for Architecture Search
• Hit-Detector: Hierarchical Trinity Architecture Search for Object Detection
• NSGANetV2: Evolutionary Multi-Objective Surrogate-Assisted Neural Architecture Search
• MixStyle Neural Networks for Domain Generalization and Adaptation
• Interstellar: Searching Recurrent Architecture for Knowledge Graph Embedding
• SM-NAS: Structural-to-Modular Neural Architecture Search for Object Detection
• HALF: Holistic Auto Machine Learning for FPGAs
• Rethinking Channel Dimensions for Efficient Model Design
• AutoPose: Searching Multi-Scale Branch Aggregation for Pose Estimation
• Searching for Winograd-aware Quantized Networks
• PILAE: A Non-gradient Descent Learning Scheme for Deep Feedforward Neural Networks
• Convolution Neural Network Architecture Learning for Remote Sensing Scene Classification
• DARTS-ASR: Differentiable Architecture Search for Multilingual Speech Recognition and Adaptation
• Search for Better Students to Learn Distilled Knowledge
• SparseBERT: Rethinking the Importance Analysis in Self-attention
• Learning Dynamic Routing for Semantic Segmentation
• Densely Connected Search Space for More Flexible Neural Architecture Search
• On Training Implicit Models
• Sanity-Checking Pruning Methods: Random Tickets can Win the Jackpot
• Speedy Performance Estimation for Neural Architecture Search
• BlockSwap: Fisher-guided Block Substitution for Network Compression on a Budget
• Auto Seg-Loss: Searching Metric Surrogates for Semantic Segmentation
• V-NAS: Neural Architecture Search for Volumetric Medical Image Segmentation
• Pruning In Time (PIT): A Lightweight Network Architecture Optimizer for Temporal Convolutional Networks
• Co-Exploration of Neural Architectures and Heterogeneous ASIC Accelerator Designs Targeting Multiple Tasks
• Learnable Boundary Guided Adversarial Training
• FL-AGCNS: Federated Learning Framework for Automatic Graph Convolutional Network Search
• DHP: Differentiable Meta Pruning via HyperNetworks
• Zen-NAS: A Zero-Shot NAS for High-Performance Deep Image Recognition
• A Second look at Exponential and Cosine Step Sizes: Simplicity, Adaptivity, and Performance
• Customizable Architecture Search for Semantic Segmentation
• LightTrack: Finding Lightweight Neural Networks for Object Tracking via One-Shot Architecture Search
• Searching for Efficient Multi-Stage Vision Transformers
• Analyzing and Mitigating Interference in Neural Architecture Search
• All You Need is a Few Shifts: Designing Efficient Convolutional Neural Networks for Image Classification
• Differentiable Architecture Search with Ensemble Gumbel-Softmax
• Differentiable Fine-grained Quantization for Deep Neural Network Compression
• Neural Architecture Design for GPU-Efficient Networks
• Can weight sharing outperform random architecture search? An investigation with TuNAS
• GreedyNAS: Towards Fast One-Shot NAS with Greedy Supernet
• Self-supervised Neural Architecture Search
• Federated Hyperparameter Tuning: Challenges, Baselines, and Connections to Weight-Sharing
• Not All Attention Is All You Need
• Batch Group Normalization
• Neural Architecture Search for Deep Face Recognition
• Fast Task-Aware Architecture Inference
• Efficient Neural Architecture Search via Proximal Iterations
• Cyclic Differentiable Architecture Search
• Efficient Deep Neural Networks
• NASGEM: Neural Architecture Search via Graph Embedding Method
• EcoNAS: Finding Proxies for Economical Neural Architecture Search
• Techniques for Automated Machine Learning
• Efficient Forward Architecture Search
• Standing on the Shoulders of Giants: Hardware and Neural Architecture Co-Search with Hot Start
• An Introduction to Neural Architecture Search for Convolutional Networks
• EDD: Efficient Differentiable DNN Architecture and Implementation Co-search for Embedded AI Solutions
• Improving the HardNet Descriptor
• Few-shot Neural Architecture Search
• Auto-PyTorch Tabular: Multi-Fidelity MetaLearning for Efficient and Robust AutoDL
• MTL-NAS: Task-Agnostic Neural Architecture Search towards General-Purpose Multi-Task Learning
• Searching to Exploit Memorization Effect in Learning from Corrupted Labels
• Hierarchical Neural Architecture Search via Operator Clustering
• Balanced One-shot Neural Architecture Optimization
• Understanding Architectures Learnt by Cell-based Neural Architecture Search
• Effective Training of Convolutional Neural Networks with Low-bitwidth Weights and Activations
• Automated Model Design and Benchmarking of 3D Deep Learning Models for COVID-19 Detection with Chest CT Scans
• CATE: Computation-aware Neural Architecture Encoding with Transformers
• DHA: End-to-End Joint Optimization of Data Augmentation Policy, Hyper-parameter and Architecture
• Circumventing Outliers of AutoAugment with Knowledge Distillation
• Multi-Objective Meta Learning
• Towards Improving the Consistency, Efficiency, and Flexibility of Differentiable Neural Architecture Search
• S2DNAS:Transforming Static CNN Model for Dynamic Inference via Neural Architecture Search
• Sharpness-aware Quantization for Deep Neural Networks
• Data Augmentation for Object Detection via Differentiable Neural Rendering
• FSNet: Compression of Deep Convolutional Neural Networks by Filter Summary
• Learning Versatile Neural Architectures by Propagating Network Codes
• SwiftNet: Using Graph Propagation as Meta-knowledge to Search Highly Representative Neural Architectures
• AutoSF: Searching Scoring Functions for Knowledge Graph Embedding
• Anycost GANs for Interactive Image Synthesis and Editing
• TF-NAS: Rethinking Three Search Freedoms of Latency-Constrained Differentiable Neural Architecture Search
• PENNI: Pruned Kernel Sharing for Efficient CNN Inference
• Prune and Replace NAS
• Flexible Multi-task Networks by Learning Parameter Allocation
• NASI: Label- and Data-agnostic Neural Architecture Search at Initialization
• Genetic Deep Learning for Lung Cancer Screening
• Deep Learning for Click-Through Rate Estimation
• DrNAS: Dirichlet Neural Architecture Search
• AttentionNAS: Spatiotemporal Attention Cell Search for Video Classification
• Rethinking Graph Neural Architecture Search from Message-passing
• Teachers Do More Than Teach: Compressing Image-to-Image Models
• Manipulating SGD with Data Ordering Attacks
• A Survey On Anti-Spoofing Methods For Face Recognition with RGB Cameras of Generic Consumer Devices
• DIFER: Differentiable Automated Feature Engineering
• Automatically Searching for U-Net Image Translator Architecture
• Joint Search of Data Augmentation Policies and Network Architectures
• Learning Differentiable Programs with Admissible Neural Heuristics
• HM-NAS: Efficient Neural Architecture Search via Hierarchical Masking
• Stochastic Training is Not Necessary for Generalization
• AutoKWS: Keyword Spotting with Differentiable Architecture Search
• Accuracy Prediction with Non-neural Model for Neural Architecture Search
• Neural Architecture Search by Estimation of Network Structure Distributions
• Rethinking the Number of Channels for the Convolutional Neural Network
• ModuleNet: Knowledge-inherited Neural Architecture Search
• Neural Epitome Search for Architecture-Agnostic Network Compression
• Learning Index Selection with Structured Action Spaces
• Multi-objective Neural Architecture Search via Non-stationary Policy Gradient
• Classifying the classifier: dissecting the weight space of neural networks
• Do Normalization Layers in a Deep ConvNet Really Need to Be Distinct?
• Ordering Chaos: Memory-Aware Scheduling of Irregularly Wired Neural Networks for Edge Devices
• Inductive Transfer for Neural Architecture Optimization
• Graph-guided Architecture Search for Real-time Semantic Segmentation
• Online Meta-Learning for Multi-Source and Semi-Supervised Domain Adaptation
• Model Complexity of Deep Learning: A Survey
• HourNAS: Extremely Fast Neural Architecture Search Through an Hourglass Lens
• Efficient Decoupled Neural Architecture Search by Structure and Operation Sampling
• Rapid Neural Architecture Search by Learning to Generate Graphs from Datasets
• TransTailor: Pruning the Pre-trained Model for Improved Transfer Learning
• Self-supervised Representation Learning for Evolutionary Neural Architecture Search
• Pay Attention when Required
• DKM: Differentiable K-Means Clustering Layer for Neural Network Compression
• Auto-MVCNN: Neural Architecture Search for Multi-view 3D Shape Recognition
• Integrating Large Circular Kernels into CNNs through Neural Architecture Search
• Discovering Robust Convolutional Architecture at Targeted Capacity: A Multi-Shot Approach
• Exploiting Channel Similarity for Accelerating Deep Convolutional Neural Networks
• Differentiable Sparsification for Deep Neural Networks
• Contrastive Neural Architecture Search with Neural Architecture Comparators
• Towards Learning of Filter-Level Heterogeneous Compression of Convolutional Neural Networks
• VA-RED${}^2$: Video Adaptive Redundancy Reduction
• Generative Zero-shot Network Quantization
• Learning to Rank Learning Curves
• DOTS: Decoupling Operation and Topology in Differentiable Architecture Search
• EH-DNAS: End-to-End Hardware-aware Differentiable Neural Architecture Search
• UFO-BLO: Unbiased First-Order Bilevel Optimization
• EvoGrad: Efficient Gradient-Based Meta-Learning and Hyperparameter Optimization
• An Asymptotically Optimal Multi-Armed Bandit Algorithm and Hyperparameter Optimization
• OPANAS: One-Shot Path Aggregation Network Architecture Search for Object Detection
• Towards Oracle Knowledge Distillation with Neural Architecture Search
• NPAS: A Compiler-aware Framework of Unified Network Pruning and Architecture Search for Beyond Real-Time Mobile Acceleration
• Deep Neural Architecture Search with Deep Graph Bayesian Optimization
• MixSearch: Searching for Domain Generalized Medical Image Segmentation Architectures
• Task-Adaptive Neural Network Search with Meta-Contrastive Learning
• On Evolving Attention Towards Domain Adaptation
• MUXConv: Information Multiplexing in Convolutional Neural Networks
• Geometry-Aware Gradient Algorithms for Neural Architecture Search
• Learning Architectures for Binary Networks
• SOTERIA: In Search of Efficient Neural Networks for Private Inference
• Generalized Data Weighting via Class-level Gradient Manipulation
• Controlling Model Complexity in Probabilistic Model-Based Dynamic Optimization of Neural Network Structures
• How can AI Automate End-to-End Data Science?
• Are Labels Necessary for Neural Architecture Search?
• Efficient Relation-aware Scoring Function Search for Knowledge Graph Embedding
• FADNet++: Real-Time and Accurate Disparity Estimation with Configurable Networks
• Single-Path Mobile AutoML: Efficient ConvNet Design and NAS Hyperparameter Optimization
• Intra-Ensemble in Neural Networks
• A Value-Function-based Interior-point Method for Non-convex Bi-level Optimization
• Genetic Neural Architecture Search for automatic assessment of human sperm images
• The Authors Matter: Understanding and Mitigating Implicit Bias in Deep Text Classification
• AutoSTR: Efficient Backbone Search for Scene Text Recognition
• Self-Assembling Modular Networks for Interpretable Multi-Hop Reasoning
• The Nonlinearity Coefficient - A Practical Guide to Neural Architecture Design
• Model Similarity Mitigates Test Set Overuse
• Bag of Tricks for Neural Architecture Search
• MoGA: Searching Beyond MobileNetV3
• RepNAS: Searching for Efficient Re-parameterizing Blocks
• Searching for Stage-wise Neural Graphs In the Limit
• Neural Architecture Search Over a Graph Search Space
• Deep learning for pedestrians: backpropagation in CNNs
• Pareto-Frontier-aware Neural Architecture Generation for Diverse Budgets
• Rethinking Performance Estimation in Neural Architecture Search
• FNA++: Fast Network Adaptation via Parameter Remapping and Architecture Search
• Auto-Classifier: A Robust Defect Detector Based on an AutoML Head
• Multi-Modality Information Fusion for Radiomics-based Neural Architecture Search
• DAAS: Differentiable Architecture and Augmentation Policy Search
• Randomized Stochastic Variance-Reduced Methods for Multi-Task Stochastic Bilevel Optimization
• ArchRepair: Block-Level Architecture-Oriented Repairing for Deep Neural Networks
• Searching Collaborative Agents for Multi-plane Localization in 3D Ultrasound
• Enhanced Bilevel Optimization via Bregman Distance
• Automated Discovery of Adaptive Attacks on Adversarial Defenses
• Optimizing Data Usage via Differentiable Rewards
• Multi-Objective Neural Architecture Search Based on Diverse Structures and Adaptive Recommendation
• Efficient hyperparameter optimization by way of PAC-Bayes bound minimization
• TransNAS-Bench-101: Improving Transferability and Generalizability of Cross-Task Neural Architecture Search
• NAS-DIP: Learning Deep Image Prior with Neural Architecture Search
• Binarized Neural Architecture Search for Efficient Object Recognition
• PONAS: Progressive One-shot Neural Architecture Search for Very Efficient Deployment
• Neural Architecture Search for Lightweight Non-Local Networks
• GLiT: Neural Architecture Search for Global and Local Image Transformer
• Transfer NAS: Knowledge Transfer between Search Spaces with Transformer Agents
• Search For Deep Graph Neural Networks
• On the Bounds of Function Approximations
• SCARLET-NAS: Bridging the Gap between Stability and Scalability in Weight-sharing Neural Architecture Search
• EPNAS: Efficient Progressive Neural Architecture Search
• MemNet: Memory-Efficiency Guided Neural Architecture Search with Augment-Trim learning
• Efficient Neural Interaction Function Search for Collaborative Filtering
• Scheduled Differentiable Architecture Search for Visual Recognition
• Binarized Neural Architecture Search
• On a Sparse Shortcut Topology of Artificial Neural Networks
• WeNet: Weighted Networks for Recurrent Network Architecture Search
• Tiny Adversarial Mulit-Objective Oneshot Neural Architecture Search
• AutoAdapt: Automated Segmentation Network Search for Unsupervised Domain Adaptation
• HR-NAS: Searching Efficient High-Resolution Neural Architectures with Lightweight Transformers
• Bilevel Optimization for Machine Learning: Algorithm Design and Convergence Analysis
• Full-attention based Neural Architecture Search using Context Auto-regression
• NAS-HPO-Bench-II: A Benchmark Dataset on Joint Optimization of Convolutional Neural Network Architecture and Training Hyperparameters
• ViPNAS: Efficient Video Pose Estimation via Neural Architecture Search
• Learning Graph Representation of Person-specific Cognitive Processes from Audio-visual Behaviours for Automatic Personality Recognition
• Fitness Landscape Footprint: A Framework to Compare Neural Architecture Search Problems
• Efficient Neural Architecture Search for End-to-end Speech Recognition via Straight-Through Gradients
• Effective, Efficient and Robust Neural Architecture Search
• How Does Supernet Help in Neural Architecture Search?
• Online Hyperparameter Meta-Learning with Hypergradient Distillation
• FEAR: A Simple Lightweight Method to Rank Architectures
• Separable Layers Enable Structured Efficient Linear Substitutions
• Optimal Gradient Checkpoint Search for Arbitrary Computation Graphs
• Generic Neural Architecture Search via Regression
• Automated Deepfake Detection
• AutoLoss-Zero: Searching Loss Functions from Scratch for Generic Tasks
• Deep-n-Cheap: An Automated Search Framework for Low Complexity Deep Learning
• Disturbance-immune Weight Sharing for Neural Architecture Search
• NetAdaptV2: Efficient Neural Architecture Search with Fast Super-Network Training and Architecture Optimization
• A Design Space Study for LISTA and Beyond
• Searching to Sparsify Tensor Decomposition for N-ary Relational Data
• Neighborhood-Aware Neural Architecture Search
• CompOFA: Compound Once-For-All Networks for Faster Multi-Platform Deployment
• DS-Net++: Dynamic Weight Slicing for Efficient Inference in CNNs and Transformers
• StyleNAS: An Empirical Study of Neural Architecture Search to Uncover Surprisingly Fast End-to-End Universal Style Transfer Networks
• G-DARTS-A: Groups of Channel Parallel Sampling with Attention
• Memory-Efficient Differentiable Transformer Architecture Search
• Interleaving Learning, with Application to Neural Architecture Search
• Understanding the Effects of Pre-Training for Object Detectors via Eigenspectrum
• MS-NAS: Multi-Scale Neural Architecture Search for Medical Image Segmentation
• ENAS4D: Efficient Multi-stage CNN Architecture Search for Dynamic Inference
• NAS-TC: Neural Architecture Search on Temporal Convolutions for Complex Action Recognition
• BNAS v2: Learning Architectures for Binary Networks with Empirical Improvements
• ISP4ML: Understanding the Role of Image Signal Processing in Efficient Deep Learning Vision Systems
• Genetic Network Architecture Search
• Anti-Bandit Neural Architecture Search for Model Defense
• Hypernetwork-Based Augmentation
• Lifelong Learning with Searchable Extension Units
• Large Scale Neural Architecture Search with Polyharmonic Splines
• ElixirNet: Relation-aware Network Architecture Adaptation for Medical Lesion Detection
• Feature Gradients: Scalable Feature Selection via Discrete Relaxation
• Rethinking Neural Operations for Diverse Tasks
• Optimization-Inspired Learning with Architecture Augmentations and Control Mechanisms for Low-Level Vision
• Pi-NAS: Improving Neural Architecture Search by Reducing Supernet Training Consistency Shift
• A Scalable and Cloud-Native Hyperparameter Tuning System
• SpeechNAS: Towards Better Trade-off between Latency and Accuracy for Large-Scale Speaker Verification
• Plug-in, Trainable Gate for Streamlining Arbitrary Neural Networks
• Exploring the Loss Landscape in Neural Architecture Search
• MetaXL: Meta Representation Transformation for Low-resource Cross-lingual Learning
• Direct Differentiable Augmentation Search
• HMCNAS: Neural Architecture Search using Hidden Markov Chains and Bayesian Optimization
• SuperShaper: Task-Agnostic Super Pre-training of BERT Models with Variable Hidden Dimensions
• Variational Depth Search in ResNets
• ProxyBO: Accelerating Neural Architecture Search via Bayesian Optimization with Zero-cost Proxies
• Multi-objective Search of Robust Neural Architectures against Multiple Types of Adversarial Attacks
• Ranking architectures using meta-learning
• Knowledge Transfer Graph for Deep Collaborative Learning
• Searching for A Robust Neural Architecture in Four GPU Hours
• Software/Hardware Co-design for Multi-modal Multi-task Learning in Autonomous Systems
• Neural Mask Generator: Learning to Generate Adaptive Word Maskings for Language Model Adaptation
• MS-DARTS: Mean-Shift Based Differentiable Architecture Search
• Dynamic Graph: Learning Instance-aware Connectivity for Neural Networks
• Structured Sparsification with Joint Optimization of Group Convolution and Channel Shuffle
• EENA: Efficient Evolution of Neural Architecture
• NeuralScale: Efficient Scaling of Neurons for Resource-Constrained Deep Neural Networks
• Disentangled Neural Architecture Search
• Dynamic Routing Networks
• Disentangling Neural Architectures and Weights: A Case Study in Supervised Classification
• Data Summarization via Bilevel Optimization
• TinyDefectNet: Highly Compact Deep Neural Network Architecture for High-Throughput Manufacturing Visual Quality Inspection
• You Only Search Once: A Fast Automation Framework for Single-Stage DNN/Accelerator Co-design
• BaLeNAS: Differentiable Architecture Search via the Bayesian Learning Rule
• Differentiable Feature Aggregation Search for Knowledge Distillation
• Adaptive Divergence for Rapid Adversarial Optimization
• Learned Transferable Architectures Can Surpass Hand-Designed Architectures for Large Scale Speech Recognition
• FATNN: Fast and Accurate Ternary Neural Networks
• Real Image Super Resolution Via Heterogeneous Model Ensemble using GP-NAS
• What and Where: Learn to Plug Adapters via NAS for Multi-Domain Learning
• Lessons from the Clustering Analysis of a Search Space: A Centroid-based Approach to Initializing NAS
• MetaAugment: Sample-Aware Data Augmentation Policy Learning
• Neural Architecture Search as Sparse Supernet
• EfficientPose: Efficient Human Pose Estimation with Neural Architecture Search
• iDARTS: Improving DARTS by Node Normalization and Decorrelation Discretization
• Value-Function-based Sequential Minimization for Bi-level Optimization
• Heart-Darts: Classification of Heartbeats Using Differentiable Architecture Search
• Learning by Self-Explanation, with Application to Neural Architecture Search
• Meta Label Correction for Noisy Label Learning
• Hyperparameter Optimization in Neural Networks via Structured Sparse Recovery
• Rapid Elastic Architecture Search under Specialized Classes and Resource Constraints
• Multi-objective Neural Architecture Search with Almost No Training
• Parametric Contrastive Learning
• The Heterogeneity Hypothesis: Finding Layer-Wise Differentiated Network Architectures
• Learning Architectures from an Extended Search Space for Language Modeling
• Direct Federated Neural Architecture Search
• Towards Accurate and Compact Architectures via Neural Architecture Transformer
• Fully Quantized Image Super-Resolution Networks
• TND-NAS: Towards Non-differentiable Objectives in Progressive Differentiable NAS Framework
• SAR-NAS: Skeleton-based Action Recognition via Neural Architecture Searching
• BN-NAS: Neural Architecture Search with Batch Normalization
• MetaXT: Meta Cross-Task Transfer between Disparate Label Spaces
• ADDS: Adaptive Differentiable Sampling for Robust Multi-Party Learning
• FLASH: Fast Neural Architecture Search with Hardware Optimization
• Mixed Variable Bayesian Optimization with Frequency Modulated Kernels
• Mining the Weights Knowledge for Optimizing Neural Network Structures
• Dynamic Slimmable Network
• A3C-S: Automated Agent Accelerator Co-Search towards Efficient Deep Reinforcement Learning
• BiX-NAS: Searching Efficient Bi-directional Architecture for Medical Image Segmentation
• Learning Elastic Embeddings for Customizing On-Device Recommenders
• Real-Time Semantic Segmentation via Auto Depth, Downsampling Joint Decision and Feature Aggregation
• Zero-Cost Operation Scoring in Differentiable Architecture Search
• Rapid Structural Pruning of Neural Networks with Set-based Task-Adaptive Meta-Pruning
• Automated Deep Abstractions for Stochastic Chemical Reaction Networks
• An Empirical Study on Regularization of Deep Neural Networks by Local Rademacher Complexity
• ViTAS: Vision Transformer Architecture Search
• Accelerating Neural Architecture Search via Proxy Data
• Neural Architecture Search via Bregman Iterations
• Raw Differentiable Architecture Search for Speech Deepfake and Spoofing Detection
• DARTS-PRIME: Regularization and Scheduling Improve Constrained Optimization in Differentiable NAS
• Differentiable Architecture Pruning for Transfer Learning
• Tensor Yard: One-Shot Algorithm of Hardware-Friendly Tensor-Train Decomposition for Convolutional Neural Networks
• Cascade Bagging for Accuracy Prediction with Few Training Samples
• Self-Supervised Neural Architecture Search for Multimodal Deep Neural Networks
• Learning from Mistakes based on Class Weighting with Application to Neural Architecture Search
• Training BatchNorm Only in Neural Architecture Search and Beyond
• Improving Differentiable Architecture Search with a Generative Model
• Predicting the success of Gradient Descent for a particular Dataset-Architecture-Initialization (DAI)
• DIVA: Dataset Derivative of a Learning Task
• T-AutoML: Automated Machine Learning for Lesion Segmentation using Transformers in 3D Medical Imaging
• Searching for TrioNet: Combining Convolution with Local and Global Self-Attention
• Meta-Teacher For Face Anti-Spoofing
• A Survey on Green Deep Learning
• FOX-NAS: Fast, On-device and Explainable Neural Architecture Search
• Single-DARTS: Towards Stable Architecture Search
• Prioritized Subnet Sampling for Resource-Adaptive Supernet Training
• Domain Adaptation by Maximizing Population Correlation with Neural Architecture Search
• Scene-adaptive Knowledge Distillation for Sequential Recommendation via Differentiable Architecture Search
• Connection Sensitivity Matters for Training-free DARTS: From Architecture-Level Scoring to Operation-Level Sensitivity Analysis
• Neural Architecture Search in operational context: a remote sensing case-study
• RankNAS: Efficient Neural Architecture Search by Pairwise Ranking
• Self-Supervised Neural Architecture Search for Imbalanced Datasets
• Partially-Connected Differentiable Architecture Search for Deepfake and Spoofing Detection
• Network Space Search for Pareto-Efficient Spaces
• Learning by Examples Based on Multi-level Optimization
• An Analysis of Super-Net Heuristics in Weight-Sharing NAS
• On the Security Risks of AutoML
• Balancing Average and Worst-case Accuracy in Multitask Learning
• NAS-FCOS: Efficient Search for Object Detection Architectures
• Guided Evolution for Neural Architecture Search
• ES-ENAS: Efficient Evolutionary Optimization for Large Hybrid Search Spaces
• Global2Local: Efficient Structure Search for Video Action Segmentation
• AUTOKD: Automatic Knowledge Distillation Into A Student Architecture Family
• MOFA: Modular Factorial Design for Hyperparameter Optimization
• Revisiting Neural Architecture Search
• Effective Algorithm-Accelerator Co-design for AI Solutions on Edge Devices
• LETI: Latency Estimation Tool and Investigation of Neural Networks inference on Mobile GPU
• Small-Group Learning, with Application to Neural Architecture Search
• AdvantageNAS: Efficient Neural Architecture Search with Credit Assignment
• AutoBSS: An Efficient Algorithm for Block Stacking Style Search
• Grow-Push-Prune: aligning deep discriminants for effective structural network compression
• MSR-DARTS: Minimum Stable Rank of Differentiable Architecture Search
• Evolving Neural Architecture Using One Shot Model
• BNAS-v2: Memory-efficient and Performance-collapse-prevented Broad Neural Architecture Search
• Neural Architecture Search via Combinatorial Multi-Armed Bandit
• Recoding latent sentence representations -- Dynamic gradient-based activation modification in RNNs
• Gradient-based Hyperparameter Optimization Over Long Horizons
• Joslim: Joint Widths and Weights Optimization for Slimmable Neural Networks
• FIVES: Feature Interaction Via Edge Search for Large-Scale Tabular Data
• Semi-Supervised Learning with Meta-Gradient
• Probabilistic Metric Learning with Adaptive Margin for Top-K Recommendation
• AutoSNAP: Automatically Learning Neural Architectures for Instrument Pose Estimation
• GIID-Net: Generalizable Image Inpainting Detection via Neural Architecture Search and Attention
• In-Loop Meta-Learning with Gradient-Alignment Reward
• Differentiable Segmentation of Sequences
• Inner Ensemble Networks: Average Ensemble as an Effective Regularizer
• Searching by Generating: Flexible and Efficient One-Shot NAS with Architecture Generator
• Evolutionary NAS with Gene Expression Programming of Cellular Encoding
• Enhanced Gradient for Differentiable Architecture Search
• Trustless parallel local search for effective distributed algorithm discovery
• On the Synergies between Machine Learning and Binocular Stereo for Depth Estimation from Images: a Survey
• Improving Neural Language Models by Segmenting, Attending, and Predicting the Future
• Joint Learning of Neural Transfer and Architecture Adaptation for Image Recognition
• Searching Efficient Model-guided Deep Network for Image Denoising
• ENOS: Energy-Aware Network Operator Search for Hybrid Digital and Compute-in-Memory DNN Accelerators
• Reconsidering CO2 emissions from Computer Vision
• Balancing Accuracy and Latency in Multipath Neural Networks
• Analyzing Neural Networks Based on Random Graphs
• Adaptive Interaction Modeling via Graph Operations Search
• Learning, transferring, and recommending performance knowledge with Monte Carlo tree search and neural networks
• CP-NAS: Child-Parent Neural Architecture Search for Binary Neural Networks
• Plug-and-Play Anomaly Detection with Expectation Maximization Filtering
• Discretization-Aware Architecture Search
• RNN Architecture Learning with Sparse Regularization
• Malleable 2.5D Convolution: Learning Receptive Fields along the Depth-axis for RGB-D Scene Parsing
• Neural Architecture Search in Embedding Space
• Learnable Parameter Similarity
• Shape Adaptor: A Learnable Resizing Module
• Differentiable TAN Structure Learning for Bayesian Network Classifiers
• BOML: A Modularized Bilevel Optimization Library in Python for Meta Learning
• Customized Graph Embedding: Tailoring Embedding Vectors to different Applications
• Efficient Structured Pruning and Architecture Searching for Group Convolution
• Nucleus Neural Network: A Data-driven Self-organized Architecture
• AmoebaContact and GDFold: a new pipeline for rapid prediction of protein structures
• LV-BERT: Exploiting Layer Variety for BERT
• Quantized Neural Networks via {-1, +1} Encoding Decomposition and Acceleration
• Efficient Data-specific Model Search for Collaborative Filtering
• Debiasing a First-order Heuristic for Approximate Bi-level Optimization