PyTorchPipe: a framework for rapid prototyping of pipelines combining language and vision

Abstract

Access to vast amounts of data along with affordable computational power stimulated the reincarnation of neural networks. The progress could not be achieved without adequate software tools, lowering the entry bar for the next generations of researchers and developers. The paper introduces PyTorchPipe (PTP), a framework built on top of PyTorch. Answering the recent needs and trends in machine learning, PTP facilitates building and training of complex, multi-modal models combining language and vision (but is not limited to those two modalities). At its core, PTP employs a component-oriented approach and relies on the concept of a pipeline, defined as a directed acyclic graph of loosely coupled components. A user defines a pipeline using yaml-based (thus human-readable) configuration files, whereas PTP provides generic workers for their loading, training, and testing using all the computational power (CPUs and GPUs) that is available to the user. The paper covers the main concepts of PyTorchPipe, discusses its key features and briefly presents the currently implemented tasks, models and components.

arXiv commentPaper accepted for SysML 2019 workshop at 33rd Conference on Neural Information Processing Systems (NeurIPS 2019)

References in corpus (12)

• VQA: Visual Question Answering
• Theano: new features and speed improvements
• Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
• A Multi-World Approach to Question Answering about Real-World Scenes based on Uncertain Input
• Hadamard Product for Low-rank Bilinear Pooling
• Pythia v0.1: the Winning Entry to the VQA Challenge 2018
• Show, Ask, Attend, and Answer: A Strong Baseline For Visual Question Answering
• Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
• Mixed-Precision Training for NLP and Speech Recognition with OpenSeq2Seq
• Ludwig: a type-based declarative deep learning toolbox
• The Visual QA Devil in the Details: The Impact of Early Fusion and Batch Norm on CLEVR
• Leveraging Medical Visual Question Answering with Supporting Facts