4 papers
ProbMoE: Differentiable Probabilistic Routing for Mixture-of-Experts
Heng Zhao, Zilei Shao, Guy Van den Broeck +1
Mixture-of-Experts (MoE) models scale by activating only a small subset of experts per token. However, training such models remains challenging because top- routing is discrete…
Breaking the Factorization Barrier in Diffusion Language Models
Ian Li, Zilei Shao, Benjie Wang +3
Diffusion language models theoretically allow for efficient parallel generation but are practically hindered by the ``factorization barrier'': the assumption that simultaneously pr…
Zero-Variance Gradients for Variational Autoencoders
Zilei Shao, Anji Liu, Guy Van den Broeck
Training deep generative models like Variational Autoencoders (VAEs) requires propagating gradients through stochastic latent variables, which introduces estimation variance that c…
Adversarial Tokenization
Renato Lui Geh, Zilei Shao, Guy Van den Broeck
Current LLM pipelines account for only one possible tokenization for a given string, ignoring exponentially many alternative tokenizations during training and inference. For exampl…