Fuzzing Automatic Differentiation in Deep-Learning Libraries

Abstract

Deep learning (DL) has attracted wide attention and has been widely deployed in recent years. As a result, more and more research efforts have been dedicated to testing DL libraries and frameworks. However, existing work largely overlooked one crucial component of any DL system, automatic differentiation (AD), which is the basis for the recent development of DL. To this end, we propose $\nabla$Fuzz, the first general and practical approach specifically targeting the critical AD component in DL libraries. Our key insight is that each DL library API can be abstracted into a function processing tensors/vectors, which can be differentially tested under various execution scenarios (for computing outputs/gradients with different implementations). We have implemented $\nabla$Fuzz as a fully automated API-level fuzzer targeting AD in DL libraries, which utilizes differential testing on different execution scenarios to test both first-order and high-order gradients, and also includes automated filtering strategies to remove false positives caused by numerical instability. We have performed an extensive study on four of the most popular and actively-maintained DL libraries, PyTorch, TensorFlow, JAX, and OneFlow. The result shows that $\nabla$Fuzz substantially outperforms state-of-the-art fuzzers in terms of both code coverage and bug detection. To date, $\nabla$Fuzz has detected 173 bugs for the studied DL libraries, with 144 already confirmed by developers (117 of which are previously unknown bugs and 107 are related to AD). Remarkably, $\nabla$Fuzz contributed 58.3% (7/12) of all high-priority AD bugs for PyTorch and JAX during a two-month period. None of the confirmed AD bugs were detected by existing fuzzers.

arXiv commentAccepted at ICSE'23

References in corpus (10)

• Evaluating Large Language Models Trained on Code
• DeepXplore: Automated Whitebox Testing of Deep Learning Systems
• On Evaluating Adversarial Robustness
• Practical Program Repair in the Era of Large Pre-trained Language Models
• Less Training, More Repairing Please: Revisiting Automated Program Repair via Zero-shot Learning
• Natural Attack for Pre-trained Models of Code
• NNSmith: Generating Diverse and Valid Test Cases for Deep Learning Compilers
• DocTer: Documentation Guided Fuzzing for Testing Deep Learning API Functions
• Automatic differentiation in ML: Where we are and where we should be going
• OneFlow: Redesign the Distributed Deep Learning Framework from Scratch

Cited by in corpus (3)

• WhiteFox: White-Box Compiler Fuzzing Empowered by Large Language Models
• KernelGPT: Enhanced Kernel Fuzzing via Large Language Models
• ACETest: Automated Constraint Extraction for Testing Deep Learning Operators