3 citations · 3 across the 14 of their papers we have counts for
14 papers
Layer-Parallel Inference Reduces Encrypted Nonlinear Depth in Transformers
Ligong Han, Kai Xu, Hao Wang +3
Fully homomorphic encryption (FHE) enables computation on encrypted data, but practical encrypted Transformer inference is bottlenecked by the sequential composition of many nonlin…
DMuon: Efficient Distributed Muon Training with Near-Adam Overhead
Vincent Chen, Starrick Liu, Regis Cheng +8
Matrix-orthogonalization-based optimizers, exemplified by Muon, have demonstrated strong convergence behavior across a wide range of modern deep learning workloads. The matrix-awar…
SARA: Unlocking Multilingual Knowledge in Mixture-of-Experts via Semantically Anchored Routing Alignment
Tianyu Dong, Yangyang Liu, Jiang Zhou +9
Sparse Mixture-of-Experts (MoE) architectures have emerged as an increasingly influential paradigm as they offer a strategic balance between parameter scalability and computational…
PACT: Preserving Anchored Cores in Task-vectors for Model Merging
Ningyuan Shi, Zhipeng Zhou, Hao Wang +2
Model merging has emerged as a training-free alternative to multi-task learning, aiming to combine multiple task-specific fine-tuned models into a single multi-task model. Most exi…
Uncertainty-Aware Reward Modeling for Stable RLHF
Licheng Pan, Haocheng Yang, Haoxuan Li +7
Reinforcement learning from human feedback (RLHF) aligns large language models by training reward models on preference data and optimizing policies to maximize predicted rewards. H…
JoyAI-LLM Flash: Advancing Mid-Scale LLMs with Token Efficiency
Aichen Cai, Anmeng Zhang, Anyu Li +66
We introduce JoyAI-LLM Flash, an efficient Mixture-of-Experts (MoE) language model designed to redefine the trade-off between strong performance and token efficiency in the sub-50B…
DIET: Learning to Distill Dataset Continually for Recommender Systems
Jiaqing Zhang, Hao Wang, Mingjia Yin +6
Modern deep recommender models are trained under a continual learning paradigm, relying on massive and continuously growing streaming behavioral logs. In large-scale platforms, ret…
The Diminishing Returns of Early-Exit Decoding in Modern LLMs
Rui Wei, Rui Du, Hanfei Yu +4
In Large Language Model (LLM) inference, early-exit refers to stopping computation at an intermediate layer once the prediction is sufficiently confident, thereby reducing latency…
S2D2: Fast Decoding for Diffusion LLMs via Training-Free Self-Speculation
Ligong Han, Hao Wang, Han Gao +2
Block-diffusion language models offer a promising path toward faster-than-autoregressive generation by combining block-wise autoregressive decoding with within-block parallel denoi…
Hopscotch: Discovering and Skipping Redundancies in Language Models
Mustafa Eyceoz, Nikhil Shivakumar Nayak, Hao Wang +2
Modern causal language models stack many attention blocks to improve performance, but not all blocks are necessary for every task. We propose Hopscotch, a simple yet effective meth…
Sculpting Subspaces: Constrained Full Fine-Tuning in LLMs for Continual Learning
Nikhil Shivakumar Nayak, Krishnateja Killamsetty, Ligong Han +8
Continual learning in large language models (LLMs) is prone to catastrophic forgetting, where adapting to new tasks significantly degrades performance on previously learned ones. E…
SQuat: Subspace-orthogonal KV Cache Quantization
Hao Wang, Ligong Han, Kai Xu +1
The key-value (KV) cache accelerates LLMs decoding by storing KV tensors from previously generated tokens. It reduces redundant computation at the cost of increased memory usage. T…
Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMs
Aldo Pareja, Nikhil Shivakumar Nayak, Hao Wang +10
The rise of large language models (LLMs) has created a significant disparity: industrial research labs with their computational resources, expert teams, and advanced infrastructure…
Privacy without Noisy Gradients: Slicing Mechanism for Generative Model Training
Kristjan Greenewald, Yuancheng Yu, Hao Wang +1
Training generative models with differential privacy (DP) typically involves injecting noise into gradient updates or adapting the discriminator's training procedure. As a result,…