SimpleFSDP: Simpler Fully Sharded Data Parallel with torch.compile

Ruisi Zhang, Tianyu Liu, Will Feng +4

Distributed training of large models consumes enormous computation resources and requires substantial engineering efforts to compose various training techniques. This paper present…