3 papers
cs.DC2025
Oases: Efficient Large-Scale Model Training on Commodity Servers via Overlapped and Automated Tensor Model Parallelism
Shengwei Li, Zhiquan Lai, Dongsheng Li +5
Deep learning is experiencing a rise in large-scale models. Training large-scale models is costly, prompting researchers to train large-scale models on commodity servers that more…
cs.LG2025
Towards Understanding the Generalizability of Delayed Stochastic Gradient Descent
Xiaoge Deng, Li Shen, Shengwei Li +3
Stochastic gradient descent (SGD) performed in an asynchronous manner plays a crucial role in training large-scale machine learning models. However, the generalization performance…
cs.LG2024
Sharpness-Aware Minimization with Adaptive Regularization for Training Deep Neural Networks
Jinping Zou, Xiaoge Deng, Tao Sun
Sharpness-Aware Minimization (SAM) has proven highly effective in improving model generalization in machine learning tasks. However, SAM employs a fixed hyperparameter associated w…