Redox: Improving I/O Efficiency of Model Training Through File Redirection

Yuhao Li, Xuanhua Shi, Yunfei Zhao +3

This paper proposes Redox, a training data management system designed to achieve high I/O efficiency. The key insight is a new observation of file redirection: for model training,…