1 citations · 3 across the 7 of their papers we have counts for
7 papers
Cautious Weight Decay
Lizhang Chen, Jonathan Li, Kaizhao Liang +6
We introduce Cautious Weight Decay (CWD), a one-line, optimizer-agnostic modification that applies weight decay only to parameter coordinates whose signs align with the optimizer u…
Train Large, Deploy Compact: Structured Compression for Compact Low-Rank Adaptation
Xin Yu, Cong Xie, Xunmei Liu +3
Low-rank adaptation (LoRA) has become a widely used paradigm for parameter-efficient fine-tuning of large language models, yet its representational capacity often lags behind full…
Truncated Proximal Policy Optimization
Tiantian Fan, Lingjun Liu, Yu Yue +20
Recently, test-time scaling Large Language Models (LLMs) have demonstrated exceptional reasoning capabilities across scientific and professional tasks by generating long chains-of-…
SDP4Bit: Toward 4-bit Communication Quantization in Sharded Data Parallelism for LLM Training
Jinda Jia, Cong Xie, Hanlin Lu +8
Recent years have witnessed a clear trend towards language models with an ever-increasing number of parameters, as well as the growing training overhead and memory usage. Distribut…
Distributed Sign Momentum with Local Steps for Training Transformers
Shuhua Yu, Ding Zhou, Cong Xie +4
Pre-training Transformer models is resource-intensive, and recent studies have shown that sign momentum is an efficient technique for training large-scale deep learning models, par…
MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Router
Yanyue Xie, Zhi Zhang, Ding Zhou +6
Mixture-of-Experts (MoE) architectures face challenges such as high memory consumption and redundancy in experts. Pruning MoE can reduce network weights while maintaining model per…
Object-Driven One-Shot Fine-tuning of Text-to-Image Diffusion with Prototypical Embedding
Jianxiang Lu, Cong Xie, Hui Guo
As large-scale text-to-image generation models have made remarkable progress in the field of text-to-image generation, many fine-tuning methods have been proposed. However, these m…