A Selective Review on Statistical Methods for Massive Data Computation: Distributed Computing, Subsampling, and Minibatch Techniques

Abstract

This paper presents a selective review of statistical computation methods for massive data analysis. A huge amount of statistical methods for massive data computation have been rapidly developed in the past decades. In this work, we focus on three categories of statistical computation methods: (1) distributed computing, (2) subsampling methods, and (3) minibatch gradient techniques. The first class of literature is about distributed computing and focuses on the situation, where the dataset size is too huge to be comfortably handled by one single computer. In this case, a distributed computation system with multiple computers has to be utilized. The second class of literature is about subsampling methods and concerns about the situation, where the sample size of dataset is small enough to be placed on one single computer but too large to be easily processed by its memory as a whole. The last class of literature studies those minibatch gradient related optimization techniques, which have been extensively used for optimizing various deep learning models.

References in corpus (43)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• One-step sparse estimates in nonconcave penalized likelihood models
• A Stochastic Gradient Method with an Exponential Convergence Rate for Finite Training Sets
• Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descent
• Optimal Subsampling for Large Sample Logistic Regression
• Non-strongly-convex smooth stochastic approximation with convergence rate O(1/n)
• Local SGD Converges Fast and Communicates Little
• Information-Based Optimal Subdata Selection for Big Data Linear Regression
• Quantile Regression Under Memory Constraint
• A Statistical Perspective on Algorithmic Leveraging
• Decentralized Quasi-Newton Methods
• Variants of RMSProp and Adagrad with Logarithmic Regret Bounds
• Why Random Reshuffling Beats Stochastic Gradient Descent
• The Step Decay Schedule: A Near Optimal, Geometrically Decaying Learning Rate Procedure For Least Squares
• Characterizing Deep-Learning I/O Workloads in TensorFlow
• Least Squares Approximation for a Distributed System
• First-order Newton-type Estimator for Distributed Estimation and Inference
• Distributed Simultaneous Inference in Generalized Linear Models via Confidence Distribution
• Efficient Distributed Learning with Sparsity
• Understanding the Role of Momentum in Stochastic Gradient Methods
• Second-Order Stochastic Optimization for Machine Learning in Linear Time
• SGD: General Analysis and Improved Rates
• Distributed High-dimensional Regression Under a Quantile Loss Function
• Random Reshuffling: Simple Analysis with Vast Improvements
• GIANT: Globally Improved Approximate Newton Method for Distributed Optimization
• An Improved Analysis of Stochastic Gradient Descent with Momentum
• Distributed Inference for Linear Support Vector Machine
• Distributed Feature Screening via Componentwise Debiasing
• Asynchronous Decentralized SGD with Quantized and Local Updates
• Divide-and-Conquer Information-Based Optimal Subdata Selection Algorithm
• A Distributed and Integrated Method of Moments for High-Dimensional Correlated Data Analysis
• On Linear Stochastic Approximation: Fine-grained Polyak-Ruppert and Non-Asymptotic Concentration
• Communication-Constrained Distributed Quantile Regression with Optimal Statistical Guarantees
• An Analysis of Constant Step Size SGD in the Non-convex Regime: Asymptotic Normality and Bias
• DAve-QN: A Distributed Averaged Quasi-Newton Method with Local Superlinear Convergence Rate
• A Distributed Quasi-Newton Algorithm for Empirical Risk Minimization with Nonsmooth Regularization
• Fast and Robust Sparsity Learning over Networks: A Decentralized Surrogate Median Regression Approach
• On the Convergence of Nesterov's Accelerated Gradient Method in Stochastic Settings
• Doubly Distributed Supervised Learning and Inference with High-Dimensional Correlated Outcomes
• Logistic Regression for Massive Data with Rare Events
• Decentralised Learning with Random Features and Distributed Gradient Descent
• An Asymptotic Analysis of Minibatch-Based Momentum Methods for Linear Regression Models