5 papers · 1 filter
FlashSinkhorn 2: Block-Sparse Entropic Optimal Transport
Felix X. -F. Ye, Yu Chin Fabian Lim, Naigang Wang +1
Streaming GPU solvers for entropic optimal transport (EOT), such as FlashSinkhorn, avoid storing the dense kernel but still evaluate all point pairs in every Sinkhorn i…
XPress: Parallel Refinement for Diffusion Drafters in Speculative Decoding
Zheng Wang, Davis Wertheimer, Yu Chin Fabian Lim +4
Block-diffusion drafters like dFlash generate an entire block of draft tokens in a single forward pass, drastically reducing the overhead of multiple-token drafting in speculative…
ZO-Act: Efficient Zeroth-Order Fine-Tuning via One-Shot Activation-Informed Low-Rank Subspaces
Xun Dong, Yibo Xu, Naigang Wang +3
Zeroth-order (ZO) optimization enables fine-tuning large language models when backpropagation is unavailable or memory-prohibitive, but existing methods often perturb full model we…
LoDAdaC: a unified local training-based decentralized framework with adaptive gradients and compressed communication
Wei Liu, Anweshit Panda, Ujwal Pandey +5
In the decentralized distributed learning, achieving fast convergence and low communication cost is essential for scalability and high efficiency. Adaptive gradient methods, such a…
Frayed RoPE and Long Inputs: A Geometric Perspective
Davis Wertheimer, Aozhong Zhang, Derrick Liu +2
Rotary Positional Embedding (RoPE) is a widely adopted technique for encoding position in language models, which, while effective, causes performance breakdown when input length ex…