From the 1 of 13 linked papers with an AI index.
13 papers
Layer-Parallel Inference Reduces Encrypted Nonlinear Depth in Transformers
Ligong Han, Kai Xu, Hao Wang +3
The paper introduces Structured Newton Layer Parallelism (SNLP) to reduce the sequential nonlinear depth of encrypted Transformer inference under fully homomorphic encryption, achi…
DMuon: Efficient Distributed Muon Training with Near-Adam Overhead
Vincent Chen, Starrick Liu, Regis Cheng +8
Matrix-orthogonalization-based optimizers, exemplified by Muon, have demonstrated strong convergence behavior across a wide range of modern deep learning workloads. The matrix-awar…
SARA: Unlocking Multilingual Knowledge in Mixture-of-Experts via Semantically Anchored Routing Alignment
Tianyu Dong, Yangyang Liu, Jiang Zhou +9
Sparse Mixture-of-Experts (MoE) architectures have emerged as an increasingly influential paradigm as they offer a strategic balance between parameter scalability and computational…
PACT: Preserving Anchored Cores in Task-vectors for Model Merging
Ningyuan Shi, Zhipeng Zhou, Hao Wang +2
Model merging has emerged as a training-free alternative to multi-task learning, aiming to combine multiple task-specific fine-tuned models into a single multi-task model. Most exi…
Uncertainty-Aware Reward Modeling for Stable RLHF
Licheng Pan, Haocheng Yang, Haoxuan Li +7
Reinforcement learning from human feedback (RLHF) aligns large language models by training reward models on preference data and optimizing policies to maximize predicted rewards. H…
S2D2: Fast Decoding for Diffusion LLMs via Training-Free Self-Speculation
Ligong Han, Hao Wang, Han Gao +2
Block-diffusion language models offer a promising path toward faster-than-autoregressive generation by combining block-wise autoregressive decoding with within-block parallel denoi…