3 papers
cs.LG2025
Flash Inference: Near Linear Time Inference for Long Convolution Sequence Models and Beyond
Costin-Andrei Oncescu, Sanket Purandare, Stratos Idreos +1
While transformers have been at the core of most recent advancements in sequence generative models, their computational cost remains quadratic in sequence length. Several subquadra…
cs.CL2025
TorchTitan: One-stop PyTorch native solution for production ready LLM pre-training
Wanchao Liang, Tianyu Liu, Less Wright +10
The development of large language models (LLMs) has been instrumental in advancing state-of-the-art natural language processing applications. Training LLMs with billions of paramet…
cs.DC2024
SimpleFSDP: Simpler Fully Sharded Data Parallel with torch.compile
Ruisi Zhang, Tianyu Liu, Will Feng +4
Distributed training of large models consumes enormous computation resources and requires substantial engineering efforts to compose various training techniques. This paper present…