Multi-objective Evolutionary Approach for Efficient Kernel Size and Shape for CNN

Abstract

While state-of-the-art development in CNN topology, such as VGGNet and ResNet, have become increasingly accurate, these networks are computationally expensive involving billions of arithmetic operations and parameters. To improve the classification accuracy, state-of-the-art CNNs usually involve large and complex convolutional layers. However, for certain applications, e.g. Internet of Things (IoT), where such CNNs are to be implemented on resource-constrained platforms, the CNN architectures have to be small and efficient. To deal with this problem, reducing the resource consumption in convolutional layers has become one of the most significant solutions. In this work, a multi-objective optimisation approach is proposed to trade-off between the amount of computation and network accuracy by using Multi-Objective Evolutionary Algorithms (MOEAs). The number of convolution kernels and the size of these kernels are proportional to computational resource consumption of CNNs. Therefore, this paper considers optimising the computational resource consumption by reducing the size and number of kernels in convolutional layers. Additionally, the use of unconventional kernel shapes has been investigated and results show these clearly outperform the commonly used square convolution kernels. The main contributions of this paper are therefore a methodology to significantly reduce computational cost of CNNs, based on unconventional kernel shapes, and provide different trade-offs for specific use cases. The experimental results further demonstrate that the proposed method achieves large improvements in resource consumption with no significant reduction in network performance. Compared with the benchmark CNN, the best trade-off architecture shows a reduction in multiplications of up to 6X and with slight increase in classification accuracy on CIFAR-10 dataset.

arXiv comment13 pages paper, plus 17 papers supplementary materials

References in corpus (12)

• Adam: A Method for Stochastic Optimization
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• Compressing Deep Convolutional Networks using Vector Quantization
• An Analysis of Deep Neural Network Models for Practical Applications
• Automatically designing CNN architectures using genetic algorithm for image classification
• Predicting Parameters in Deep Learning
• Speeding up Convolutional Neural Networks with Low Rank Expansions
• Flattened Convolutional Neural Networks for Feedforward Acceleration
• ACNet: Strengthening the Kernel Skeletons for Powerful CNN via Asymmetric Convolution Blocks
• A Genetic Programming Approach to Designing Convolutional Neural Network Architectures