5 papers
NSP: Accelerating Variable-Length LLM Training via Nested Sequence Parallelism
Yi'ou Wang, Xiaoyang Li, Yijie Zheng +2
Long-context LLM training on long-tailed corpora faces a central communication--balance tradeoff. Such sequence-length heterogeneity makes any single sequence-parallelism (SP) degr…
ScenePilot: Controllable Boundary-Driven Critical Scenario Generation for Autonomous Driving
Qiyu Ruan, Yuxuan Wang, He Li +2
Safety-critical scenarios are central to evaluating autonomous driving systems, yet their rarity in naturalistic logs makes simulation-based stress testing indispensable. Most scen…
Mosaic: Towards Efficient Training of Multimodal Models with Spatial Resource Multiplexing
Yanbo Wang, Yuxuan Wang, Chen Chen +6
With the wide adoption of Multimodal Models (MMs) in real-world scenarios, it is significant to efficiently train emerging MMs that exhibit increasingly complex module architecture…
Seaweed-7B: Cost-Effective Training of Video Generation Foundation Model
Team Seawead, Ceyuan Yang, Zhijie Lin +52
This technical report presents a cost-efficient strategy for training a video generation foundation model. We present a mid-sized research model with approximately 7 billion parame…
OrchMLLM: Orchestrate Multimodal Data with Batch Post-Balancing to Accelerate Multimodal Large Language Model Training
Yijie Zheng, Bangjun Xiao, Lei Shi +7
Multimodal large language models (MLLMs), such as GPT-4o, are garnering significant attention. During the exploration of MLLM training, we identified Modality Composition Incoheren…