1 citations · 1 across the 9 of their papers we have counts for
7 papers · 1 filter
Optimal low-rank stochastic gradient estimation for LLM training
Zehao Li, Tao Ren, Zishi Zhang +2
Large language model (LLM) training is often bottlenecked by memory constraints and stochastic gradient noise in extremely high-dimensional parameter spaces. Motivated by empirical…
Omni-Masked Gradient Descent: Memory-Efficient Optimization via Mask Traversal with Improved Convergence
Hui Yang, Tao Ren, Jinyang Jiang +2
Memory-efficient optimization methods have recently gained increasing attention for scaling full-parameter training of large language models under the GPU-memory bottleneck. Existi…
RiskPO: Risk-based Policy Optimization via Verifiable Reward for LLM Post-Training
Tao Ren, Jinyang Jiang, Hui Yang +10
Reinforcement learning with verifiable reward has recently emerged as a central paradigm for post-training large language models (LLMs); however, prevailing mean-based methods, suc…
Forward Learning with Differential Privacy
Mingqian Feng, Zeliang Zhang, Jinyang Jiang +2
Differential privacy (DP) in deep learning is a critical concern as it ensures the confidentiality of training data while maintaining model utility. Existing DP training algorithms…
CoNNect: Connectivity-Based Regularization for Structural Pruning
Christian Franssen, Jinyang Jiang, Yijie Peng +1
Pruning encompasses a range of techniques aimed at increasing the sparsity of neural networks (NNs). These techniques can generally be framed as minimizing a loss function subject…
FLOPS: Forward Learning with OPtimal Sampling
Tao Ren, Zishi Zhang, Jinyang Jiang +4
Given the limitations of backpropagation, perturbation-based gradient computation methods have recently gained focus for learning with only forward passes, also referred to as quer…
Approximated Likelihood Ratio: A Forward-Only and Parallel Framework for Boosting Neural Network Training
Zeliang Zhang, Jinyang Jiang, Zhuo Liu +3
Efficient and biologically plausible alternatives to backpropagation in neural network training remain a challenge due to issues such as high computational complexity and additiona…