10 papers
LeJOT-AutoML: LLM-Driven Feature Engineering for Job Execution Time Prediction in Databricks Cost Optimization
Lizhi Ma, Yi-Xiang Hu, Yihui Ren +2
Databricks job orchestration systems (e.g., LeJOT) reduce cloud costs by selecting low-priced compute configurations while meeting latency and dependency constraints. Accurate exec…
LeJOT: An Intelligent Job Cost Orchestration Solution for Databricks Platform
Lizhi Ma, Yi-Xiang Hu, Yuke Wang +5
With the rapid advancements in big data technologies, the Databricks platform has become a cornerstone for enterprises and research institutions, offering high computational effici…
Machine Learning-Driven Predictive Resource Management in Complex Science Workflows
Tasnuva Chowdhury, Tadashi Maeno, Fatih Furkan Akman +23
The collaborative efforts of large communities in science experiments, often comprising thousands of global members, reflect a monumental commitment to exploration and discovery. R…
Data Management System Analysis for Distributed Computing Workloads
Kuan-Chieh Hsu, Sairam Sri Vatsavai, Ozgur O. Kilic +20
Large-scale international collaborations such as ATLAS rely on globally distributed workflows and data management to process, move, and store vast volumes of data. ATLAS's Producti…
CGSim: A Simulation Framework for Large Scale Distributed Computing Environment
Sairam Sri Vatsavai, Raees Khan, Kuan-Chieh Hsu +20
Large-scale distributed computing infrastructures such as the Worldwide LHC Computing Grid (WLCG) require comprehensive simulation tools for evaluating performance, testing new alg…
TPCpp-10M: Simulated proton-proton collisions in a Time Projection Chamber for AI Foundation Models
Shuhang Li, Yi Huang, David Park +10
Scientific foundation models hold great promise for advancing nuclear and particle physics by improving analysis precision and accelerating discovery. Yet, progress in this field i…