BigDL: A Distributed Deep Learning Framework for Big Data

Abstract

This paper presents BigDL (a distributed deep learning framework for Apache Spark), which has been used by a variety of users in the industry for building deep learning applications on production big data platforms. It allows deep learning applications to run on the Apache Hadoop/Spark cluster so as to directly process the production data, and as a part of the end-to-end data analysis pipeline for deployment and management. Unlike existing deep learning frameworks, BigDL implements distributed, data parallel training directly on top of the functional compute model (with copy-on-write and coarse-grained operations) of Spark. We also share real-world experience and "war stories" of users that have adopted BigDL to address their challenges(i.e., how to easily build end-to-end data analysis and deep learning pipelines for their production data).

In ACM Symposium of Cloud Computing conference (SoCC) 2019

References in corpus (12)

• SSD: Single Shot MultiBox Detector
• Sequence to Sequence Learning with Neural Networks
• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• TensorFlow: A system for large-scale machine learning
• Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• Revisiting Distributed Synchronous SGD
• RLlib: Abstractions for Distributed Reinforcement Learning
• ChainerMN: Scalable Distributed Deep Learning Framework
• ImageNet Training in Minutes

Cited by in corpus (16)

• Review: Deep Learning in Electron Microscopy
• Distributed Training of Deep Learning Models: A Taxonomic Perspective
• Cluster Pruning: An Efficient Filter Pruning Method for Edge AI Vision Applications
• Video Big Data Analytics in the Cloud: A Reference Architecture, Survey, Opportunities, and Open Research Issues
• Toward automatic comparison of visualization techniques: Application to graph visualization
• A Scalable Framework for Multilevel Streaming Data Analytics using Deep Learning
• Distributed Deep Learning Using Volunteer Computing-Like Paradigm
• Machine Learning Pipelines with Modern Big Data Tools for High Energy Physics
• Democratizing Production-Scale Distributed Deep Learning
• Towards a Scalable and Distributed Infrastructure for Deep Learning Applications
• Tensor Relational Algebra for Machine Learning System Design
• Quantifying and Improving Performance of Distributed Deep Learning with Cloud Storage
• Gear Training: A new way to implement high-performance model-parallel training
• A Survey on Proactive Customer Care: Enabling Science and Steps to Realize it
• Apply Distributed CNN on Genomics to accelerate Transcription-Factor TAL1 Motif Prediction
• PipeTune: Pipeline Parallelism of Hyper and System Parameters Tuning for Deep Learning Clusters