Publications (36)
xGFabric: Coupling Sensor Networks and HPC Facilities with Private 5G Wireless Networks for Real-Time Digital Agriculture
Liubov Kurafeeva, Alan Subedi, Ryan Hartung +9
Advanced scientific applications require coupling distributed sensor networks with centralized high-performance computing facilities. Citrus Under Protective Screening (CUPS) exemp…
Scalable Runtime Architecture for Data-driven, Hybrid HPC and ML Workflow Applications
Andre Merzky, Mikhail Titov, Matteo Turilli +3
Hybrid workflows combining traditional HPC and novel ML methodologies are transforming scientific computing. This paper presents the architecture and implementation of a scalable r…
Synapse: Synthetic Application Profiler and Emulator
Andre Merzky, Ming Tai Ha, Matteo Turilli +1
Motivated by the need to emulate workload execution characteristics on high-performance and distributed heterogeneous resources, we introduce Synapse. Synapse is used as a proxy ap…
Integrating and Characterizing HPC Task Runtime Systems for hybrid AI-HPC workloads
Andre Merzky, Mikhail Titov, Matteo Turilli +1
Scientific workflows increasingly involve both HPC and machine-learning tasks, combining MPI-based simulations, training, and inference in a single execution. Launchers such as Slu…
Exascale Workflow Applications and Middleware: An ExaWorks Retrospective
Aymen Alsaadi, Mihael Hategan-Marandiuc, Ketan Maheshwari +9
Exascale computers offer transformative capabilities to combine data-driven and learning-based approaches with traditional simulation applications to accelerate scientific discover…
ExaWorks Software Development Kit: A Robust and Scalable Collection of Interoperable Workflow Technologies
Matteo Turilli, Mihael Hategan-Marandiuc, Mikhail Titov +13
Scientific discovery increasingly requires executing heterogeneous scientific workflows on high-performance computing (HPC) platforms. Heterogeneous workflows contain different typ…
Pandemic Drugs at Pandemic Speed: Infrastructure for Accelerating COVID-19 Drug Discovery with Hybrid Machine Learning- and Physics-based Simulations on High Performance Computers
Agastya P. Bhati, Shunzhou Wan, Dario Alfè +26
The race to meet the challenges of the global pandemic has served as a reminder that the existing drug discovery process is expensive, inefficient and slow. There is a major bottle…
P*: A Model of Pilot-Abstractions
Andre Luckow, Mark Santcroos, Ole Weidner +3
Pilot-Jobs support effective distributed resource utilization, and are arguably one of the most widely-used distributed computing abstractions - as measured by the number and types…
Protein-Ligand Docking Surrogate Models: A SARS-CoV-2 Benchmark for Deep Learning Accelerated Virtual Screening
Austin Clyde, Thomas Brettin, Alexander Partin +8
We propose a benchmark to study surrogate model accuracy for protein-ligand docking. We share a dataset consisting of 200 million 3D complex structures and 2D structure scores acro…
Workflow Mini-Apps: Portable, Scalable, Tunable & Faithful Representations of Scientific Workflows
Ozgur Ozan Kilic, Tianle Wang, Matteo Turilli +4
Workflows are critical for scientific discovery. However, the sophistication, heterogeneity, and scale of workflows make building, testing, and optimizing them increasingly challen…
Scalable HPC and AI Infrastructure for COVID-19 Therapeutics
Hyungro Lee, Andre Merzky, Li Tan +15
COVID-19 has claimed more 1 million lives and resulted in over 40 million infections. There is an urgent need to identify drugs that can inhibit SARS-CoV-2. In response, the DOE re…
Adaptive Protein Design Protocols and Middleware
Aymen Alsaadi, Jonathan Ash, Mikhail Titov +4
Computational protein design is experiencing a transformation driven by AI/ML. However, the range of potential protein sequences and structures is astronomically vast, even for mod…
Evaluating Distributed Execution of Workloads
Matteo Turilli, Yadu Nand Babuji, Andre Merzky +4
Resource selection and task placement for distributed execution poses conceptual and implementation difficulties. Although resource selection and task placement are at the core of…
Integrating Abstractions to Enhance the Execution of Distributed Applications
Matteo Turilli, Feng Liu, Zhao Zhang +5
One of the factors that limits the scale, performance, and sophistication of distributed applications is the difficulty of concurrently executing them on multiple distributed compu…
Design and Implementation of an Analysis Pipeline for Heterogeneous Data
Arup Kumar Sarker, Aymen Alsaadi, Niranda Perera +8
Managing and preparing complex data for deep learning, a prevalent approach in large-scale data science can be challenging. Data transfer for model training also presents difficult…
Characterizing the Performance of Executing Many-tasks on Summit
Matteo Turilli, Andre Merzky, Thomas Naughton +2
Many scientific workloads are comprised of many tasks, where each task is an independent simulation or analysis of data. The execution of millions of tasks on heterogeneous HPC pla…
A Scalable Solution for Running Ensemble Simulations for Photovoltaic Energy
Weiming Hu, Guido Cervone, Matteo Turilli +2
This chapter proposes and provides an in-depth discussion of a scalable solution for running ensemble simulation for solar energy production. Generating a forecast ensemble is comp…
Workflows Community Summit: Advancing the State-of-the-art of Scientific Workflows Management Systems Research and Development
Rafael Ferreira da Silva, Henri Casanova, Kyle Chard +55
Scientific workflows are a cornerstone of modern scientific computing, and they have underpinned some of the most significant discoveries of the last decade. Many of these workflow…
ExaWorks: Workflows for Exascale
Aymen Al-Saadi, Dong H. Ahn, Yadu Babuji +12
Exascale computers will offer transformative capabilities to combine data-driven and learning-based approaches with traditional simulation applications to accelerate scientific dis…
RADICAL-Cybertools: Middleware Building Blocks for Scalable Science
Vivek Balasubramanian, Shantenu Jha, Andre Merzky +1
RADICAL-Cybertools (RCT) are a set of software systems that serve as middleware to develop efficient and effective tools for scientific computing. Specifically, RCT enable executin…
IMPECCABLE: Integrated Modeling PipelinE for COVID Cure by Assessing Better LEads
Aymen Al Saadi, Dario Alfe, Yadu Babuji +33
The drug discovery process currently employed in the pharmaceutical industry typically requires about 10 years and $2-3 billion to deliver one new drug. This is both too expensive…
Towards General Distributed Resource Selection
Ming Tai Ha, Matteo Turilli, Andre Merzky +1
The advantages of distributing workloads and utilizing multiple distributed resources are now well established. The type and degree of heterogeneity of distributed resources is inc…
Synapse: Synthetic Application Profiler and Emulator
Andre Merzky, Shantenu Jha
We introduce Synapse motivated by the needs to estimate and emulate workload execution characteristics on high-performance and distributed heterogeneous resources. Synapse has a pl…
RepEx: A Flexible Framework for Scalable Replica Exchange Molecular Dynamics Simulations
Antons Treikalis, Andre Merzky, Haoyuan Chen +3
Replica Exchange (RE) simulations have emerged as an important algorithmic tool for the molecular sciences. RE simulations involve the concurrent execution of independent simulatio…
PSI/J: A Portable Interface for Submitting, Monitoring, and Managing Jobs
Mihael Hategan-Marandiuc, Andre Merzky, Nicholson Collier +10
It is generally desirable for high-performance computing (HPC) applications to be portable between HPC systems, for example to make use of more performant hardware, make effective…
Using Pilot Systems to Execute Many Task Workloads on Supercomputers
Andre Merzky, Matteo Turilli, Manuel Maldonado +2
High performance computing systems have historically been designed to support applications comprised of mostly monolithic, single-job workloads. Pilot systems decouple workload spe…
Design and Performance Characterization of RADICAL-Pilot on Leadership-class Platforms
Andre Merzky, Matteo Turilli, Mikhail Titov +2
Many extreme scale scientific applications have workloads comprised of a large number of individual high-performance tasks. The Pilot abstraction decouples workload specification,…
Grid Computing: The Next Decade -- Report and Summary
Jarek Nabrzyski, Krzysztof Kurowski, Daniel S. Katz +1
The evolution of the global scientific cyberinfrastructure (CI) has, over the last 10+ years, led to a large diversity of CI instances. While specialized, competing and alternative…
Design and Performance Characterization of RADICAL-Pilot on Titan
Andre Merzky, Matteo Turilli, Manuel Maldonado +1
Many extreme scale scientific applications have workloads comprised of a large number of individual high-performance tasks. The Pilot abstraction decouples workload specification,…
RHAPSODY: Execution of Hybrid AI-HPC Workflows at Scale
Aymen Alsaadi, Mason Hooten, Mariya Goliyad +14
Hybrid AI-HPC workflows combine large-scale simulation, training, high-throughput inference, and tightly coupled, agent-driven control within a single execution campaign. These wor…
Designing Workflow Systems Using Building Blocks
Matteo Turilli, Andre Merzky, Vivek Balasubramanian +2
We suggest there is a need for a fresh perspective on the design and development of workflow systems and argue for a building blocks approach. We outline a description of this appr…
Hybrid Edge-HPC Systems for Low-Latency Data-Driven Inference
Liubov Kurafeeva, Ryan Hartung, Benjamin Carter +9
Emerging cyber-physical systems increasingly require low-latency inference from streaming sensor data while maintaining models that reflect complex and evolving physical processes.…
Middleware Building Blocks for Workflow Systems
Matteo Turilli, Vivek Balasubramanian, Andre Merzky +2
This paper describes a building blocks approach to the design of scientific workflow systems. We discuss RADICAL-Cybertools as one implementation of the building blocks concept, sh…
Pipeline for Automating Compliance-based Elimination and Extension (PACE2): A Systematic Framework for High-throughput Biomolecular Material Simulation Workflows
Srinivas C. Mushnoori, Ethan Zang, Akash Banerjee +5
The formation of biomolecular materials via dynamical interfacial processes such as self-assembly and fusion, for diverse compositions and external conditions, can be efficiently p…
RADICAL-Pilot and Parsl: Executing Heterogeneous Workflows on HPC Platforms
Aymen Alsaadi, Logan Ward, Andre Merzky +4
Workflows applications are becoming increasingly important to support scientific discovery. That is leading to a proliferation of workflow management systems and, thus, to a fragme…
RAPTOR: Ravenous Throughput Computing
Andre Merzky, Matteo Turilli, Shantenu Jha
We describe the design, implementation and performance of the RADICAL-Pilot task overlay (RAPTOR). RAPTOR enables the execution of heterogeneous tasks -- i.e., functions and execut…