4 papers
Skrull: Towards Efficient Long Context Fine-tuning through Dynamic Data Scheduling
Hongtao Xu, Wenting Shen, Yuanxin Wei +6
Long-context supervised fine-tuning (Long-SFT) plays a vital role in enhancing the performance of large language models (LLMs) on long-context tasks. To smoothly adapt LLMs to long…
TAPAS: Fast and Automatic Derivation of Tensor Parallel Strategies for Large Neural Networks
Ziji Shi, Le Jiang, Ang Wang +6
Tensor parallelism is an essential technique for distributed training of large neural networks. However, automatically determining an optimal tensor parallel strategy is challengin…
Efficient Long Context Fine-tuning with Chunk Flow
Xiulong Yuan, Hongtao Xu, Wenting Shen +10
Long context fine-tuning of large language models(LLMs) involves training on datasets that are predominantly composed of short sequences and a small proportion of longer sequences.…
BladeDISC++: Memory Optimizations Based On Symbolic Shape
Xiulong Yuan, Xu Yan, Wenting Shen +5
Recent deep learning workloads exhibit dynamic characteristics, leading to the rising adoption of dynamic shape compilers. These compilers can generate efficient kernels for dynami…