7 papers
ShellElastic: Scalable Elastic Pretraining with Simultaneous Maximal Updates and Hyperparameter Transfer
Can Jin, Yan Kang, Yifan Gong +5
Foundation models need to support diverse inference budgets, yet training a separate model for each budget duplicates pretraining effort. Elastic models train several nested sub-ne…
Complete-muE: Optimal Hyperparameter Transfer and Scaling for MoE Models
Hongwu Peng, Ohiremen Dibua, Yuanjun Xiong +8
We propose Complete-muE, a framework which targets hyperparameter transfer across dense FFN and any Mixture-of-Experts (MoE) setups in transformer blocks. Existing tools such as $μ…
Soft Tail-dropping for Adaptive Visual Tokenization
Zeyuan Chen, Kai Zhang, Zhuowen Tu +1
We present Soft Tail-dropping Adaptive Tokenizer (STAT), a 1D discrete visual tokenizer that adaptively chooses the number of output tokens per image according to its structural co…
AlignTok: Aligning Visual Foundation Encoders to Tokenizers for Diffusion Models
Bowei Chen, Sai Bi, Hao Tan +6
In this work, we propose aligning pretrained visual encoders to serve as tokenizers for latent diffusion models in image generation. Unlike training a variational autoencoder (VAE)…
KnapFormer: An Online Load Balancer for Efficient Diffusion Transformers Training
Kai Zhang, Peng Wang, Sai Bi +2
We present KnapFormer, an efficient and versatile framework to combine workload balancing and sequence parallelism in distributed training of Diffusion Transformers (DiT). KnapForm…
RetinaGS: Scalable Training for Dense Scene Rendering with Billion-Scale 3D Gaussians
Bingling Li, Shengyi Chen, Luchao Wang +3
In this work, we explore the possibility of training high-parameter 3D Gaussian splatting (3DGS) models on large-scale, high-resolution datasets. We design a general model parallel…
A Full-duplex Speech Dialogue Scheme Based On Large Language Models
Peng Wang, Songshuo Lu, Yaohua Tang +3
We present a generative dialogue system capable of operating in a full-duplex manner, allowing for seamless interaction. It is based on a large language model (LLM) carefully align…