Pruning and Slicing Neural Networks using Formal Verification

Abstract

Deep neural networks (DNNs) play an increasingly important role in various computer systems. In order to create these networks, engineers typically specify a desired topology, and then use an automated training algorithm to select the network's weights. While training algorithms have been studied extensively and are well understood, the selection of topology remains a form of art, and can often result in networks that are unnecessarily large - and consequently are incompatible with end devices that have limited memory, battery or computational power. Here, we propose to address this challenge by harnessing recent advances in DNN verification. We present a framework and a methodology for discovering redundancies in DNNs - i.e., for finding neurons that are not needed, and can be removed in order to reduce the size of the DNN. By using sound verification techniques, we can formally guarantee that our simplified network is equivalent to the original, either completely, or up to a prescribed tolerance. Further, we show how to combine our technique with slicing, which results in a family of very small DNNs, which are together equivalent to the original. Our approach can produce DNNs that are significantly smaller than the original, rendering them suitable for deployment on additional kinds of systems, and even more amenable to subsequent formal verification. We provide a proof-of-concept implementation of our approach, and use it to evaluate our techniques on several real-world DNNs.

References in corpus (13)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• End to End Learning for Self-Driving Cars
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• Binarized Neural Networks
• Formal Security Analysis of Neural Networks using Symbolic Intervals
• An approach to reachability analysis for feed-forward ReLU neural networks
• A Unified View of Piecewise Linear Neural Network Verification
• Towards Proving the Adversarial Robustness of Deep Neural Networks
• Algorithms for Verifying Deep Neural Networks
• Parallelization Techniques for Verifying Neural Networks
• Toward Scalable Verification for Safety-Critical Deep Networks
• Towards Scalable Verification of Deep Reinforcement Learning

Cited by in corpus (1)

• Towards Scalable Verification of Deep Reinforcement Learning