Benchmarking Edge Inference Strategies for Deep Learning Models in Industrial Machine Vision
arXiv:2607.11356
The paper compares the inference performance of four popular frameworks—PyTorch, ONNX Runtime, OpenVINO, and TensorRT—on CPU and GPU edge devices for industrial machine‑vision tasks, using both CNN and vision‑transformer models.
Abstract
Edge deployment is often the preferred solution for industrial machine vision systems when low latency, data security, or limited connectivity are critical requirements. Several frameworks are available to optimise inference on edge devices; however, relatively few studies have systematically compared their inference-time performance under industrial deployment conditions. In this work, we present a comparative study of four widely used approaches for machine vision inference in industrial settings: plain PyTorch, ONNX Runtime, OpenVINO, and TensorRT. The evaluation focuses on inference time, covers several CPU- and GPU-based hardware platforms, and includes both conventional convolutional neural networks and a transformer-based vision model. For the hardware platforms and models evaluated, the results show that OpenVINO achieves the lowest inference time on CPUs, while TensorRT achieves the lowest inference time on GPUs. However, TensorRT does not outperform plain PyTorch for the transformer-based model considered in this study.
6 pages, 6 figures, accepted at IEEE COINS 2026. Preprint version; camera-ready version to appear in the conference proceedings