FermiNets: Learning generative machines to generate efficient neural networks via generative synthesis

Abstract

The tremendous potential exhibited by deep learning is often offset by architectural and computational complexity, making widespread deployment a challenge for edge scenarios such as mobile and other consumer devices. To tackle this challenge, we explore the following idea: Can we learn generative machines to automatically generate deep neural networks with efficient network architectures? In this study, we introduce the idea of generative synthesis, which is premised on the intricate interplay between a generator-inquisitor pair that work in tandem to garner insights and learn to generate highly efficient deep neural networks that best satisfies operational requirements. What is most interesting is that, once a generator has been learned through generative synthesis, it can be used to generate not just one but a large variety of different, unique highly efficient deep neural networks that satisfy operational requirements. Experimental results for image classification, semantic segmentation, and object detection tasks illustrate the efficacy of generative synthesis in producing generators that automatically generate highly efficient deep neural networks (which we nickname FermiNets) with higher model efficiency and lower computational costs (reaching >10x more efficient and fewer multiply-accumulate operations than several tested state-of-the-art networks), as well as higher energy efficiency (reaching >4x improvements in image inferences per joule consumed on a Nvidia Tegra X2 mobile processor). As such, generative synthesis can be a powerful, generalized approach for accelerating and improving the building of deep neural networks for on-device edge scenarios.

arXiv comment4 pages

References in corpus (10)

• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
• Deep Residual Learning for Image Recognition
• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• An Analysis of Deep Neural Network Models for Practical Applications
• RefineNet: Multi-Path Refinement Networks for High-Resolution Semantic Segmentation
• ProjectionNet: Learning Efficient On-Device Deep Networks Using Neural Projections
• Two-Bit Networks for Deep Learning on Resource-Constrained Embedded Devices
• SquishedNets: Squishing SqueezeNet further for edge device scenarios via deep evolutionary synthesis

Cited by in corpus (14)

• COVID-Net: A Tailored Deep Convolutional Neural Network Design for Detection of COVID-19 Cases from Chest X-Ray Images
• Do Explanations Reflect Decisions? A Machine-centric Strategy to Quantify the Performance of Explainability Algorithms
• EdgeSpeechNets: Highly Efficient Deep Neural Networks for Speech Recognition on the Edge
• TinySpeech: Attention Condensers for Deep Speech Recognition Neural Networks on Edge Devices
• AttendNets: Tiny Deep Image Recognition Neural Networks for the Edge via Visual Attention Condensers
• DepthNet Nano: A Highly Compact Self-Normalizing Neural Network for Monocular Depth Estimation
• DeepLABNet: End-to-end Learning of Deep Radial Basis Networks with Fully Learnable Basis Functions
• Human-Machine Collaborative Design for Accelerated Design of Compact Deep Neural Networks for Autonomous Driving
• Progressive Label Distillation: Learning Input-Efficient Deep Neural Networks
• TinyDefectNet: Highly Compact Deep Neural Network Architecture for High-Throughput Manufacturing Visual Quality Inspection
• State of Compact Architecture Search For Deep Neural Networks
• PolyNeuron: Automatic Neuron Discovery via Learned Polyharmonic Spline Activations
• COVID-Net MLSys: Designing COVID-Net for the Clinical Workflow
• Seeing Convolution Through the Eyes of Finite Transformation Semigroup Theory: An Abstract Algebraic Interpretation of Convolutional Neural Networks