5 papers
GIFT: Guided Importance-Aware Fine-Tuning for Diffusion Language Models
Guowei Xu, Wenxin Xu, Jiawang Zhao +1
Diffusion models have recently shown strong potential in language modeling, offering faster generation compared to traditional autoregressive approaches. However, applying supervis…
InfiniPipe: Elastic Pipeline Parallelism for Efficient Variable-Length Long-Context LLM Training
Shiju Wang, Yujie Wang, Ao Sun +5
Long context training is crucial for LLM's context extension. Existing schemes, such as sequence parallelism, incur substantial communication overhead. Pipeline parallelism (PP) re…
ES-dLLM: Efficient Inference for Diffusion Large Language Models by Early-Skipping
Zijian Zhu, Fei Ren, Zhanhong Tan +1
Diffusion large language models (dLLMs) are emerging as a promising alternative to autoregressive models (ARMs) due to their ability to capture bidirectional context and the potent…
RailX: A Flexible, Scalable, and Low-Cost Network Architecture for Hyper-Scale LLM Training Systems
Yinxiao Feng, Tiancheng Chen, Yuchen Wei +5
Increasingly large AI workloads are calling for hyper-scale infrastructure; however, traditional interconnection network architecture is neither scalable nor cost-effective enough.…
SoMa: Identifying, Exploring, and Understanding the DRAM Communication Scheduling Space for DNN Accelerators
Jingwei Cai, Xuan Wang, Mingyu Gao +5
Modern Deep Neural Network (DNN) accelerators are equipped with increasingly larger on-chip buffers to provide more opportunities to alleviate the increasingly severe DRAM bandwidt…