Skip to main content

ENABLING DYNAMIC AND INTELLIGENT WORKFLOWS IN THE FUTURE EUROHPCECOSYSTEM

PCI2021-121957

•
Funding agency name Agencia Estatal de Investigación
Funding agency acronym AEI
Program Programa Estatal para Afrontar las Prioridades de Nuestro Entorno
Subprogram Subprograma Estatal de Internacionalización
Call Proyectos Programación Conjunta Internacional
Call year 2021
State Plan 2017-2020 Plan Estatal de Investigación Científica y Técnica y de Innovación
Beneficiary centre BARCELONA SUPERCOMPUTING CENTER CENTRO NACIONAL DE SUPERCOMPUTACION

Resultados relacionados

Resultados totales (Incluyendo duplicados): 23
Encontrada(s) 1 página(s)

RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publication Artículo científico (JournalArticle)

Parallel reduced-order modeling for digital twins using high-performance computing workflows

  • Ares de Parga, S.
  • Bravo, J. R.
  • Sibuet, N.
  • Hernandez, J. A.
  • Rossi, R.
  • Boschert, Stefan
  • Tatu, Cristian Catalin
  • Vázquez-Novoa, Fernando
  • Ejarque, Jorge
  • Badia, Rosa M.
  • Quintana-Ortí, Enrique S.|||0000-0002-5454-165X
  • Tomás Domínguez, Andrés Enrique
[EN] The integration of reduced-order models with high-performance computing is critical for developing digital twins, particularly for real-time monitoring and predictive maintenance of industrial systems. This paper presents a comprehensive, high-performance computing-enabled workflow for developing and deploying projection-based reduced-order models for large-scale mechanical simulations. We use PyCOMPSs' parallel framework to efficiently execute reduced-order model training simulations, employing parallel singular value decomposition algorithms such as randomized singular value decomposition, Lanczos singular value decomposition, and full singular value decomposition based on tall-skinny QR. Moreover, we introduce a partitioned version of the hyperreduction scheme known as the Empirical Cubature Method to further enhance computational efficiency in projection-based reduced-order models for mechanical systems. Despite the widespread use of high-performance computing for projection-based reduced-order models, there is a significant lack of publications detailing comprehensive workflows for building and deploying end-to-end projection-based reduced-order models in high-performance computing environments. Our workflow is validated through a case study focusing on the thermal dynamics of a motor, a multiphysics problem involving convective heat transfer and mechanical components. The projection-based reduced-order model is designed to deliver a real-time prognosis tool that could enable rapid and safe motor restarts post-emergency shutdowns under different operating conditions, demonstrating its potential impact on the practice of simulations in engineering mechanics. To facilitate deployment, we use the High-Performance Computing Workflow as a Service strategy and Functional Mock-Up Units to ensure compatibility and ease of integration across high-performance computing, edge, and cloud environments. The outcomes illustrate the efficacy of combining projection-based reduced-order models and high-performance computing, establishing a precedent for scalable, real-time digital twin applications in computational mechanics across multiple industries., The authors acknowledge financial support from the Spanish Ministry of Economy and Competitiveness, through the Severo Ochoa Programme for Centres of Excellence in R&D (CEX2018-000797-S and CEX2021-001148-S) , as well as to the project PID2019-107255GB (BSC authors). This project has received funding from the European High-Performance Computing Joint Undertaking (JU) under grant agreement No 955558. The JU receives support from the European Union s Horizon 2020 research and innovation programme and Spain, Germany, France, Italy, Poland, Switzerland, Norway. This publication is part of the R&D project PCI2021-121944 and PCI2021-121957, financed by MCIN/AEI/10.13039/501100011033 and by the European Union NextGenerationEU/PRTR . Sebastian Ares de Parga Regalado acknowledges financial support from the Generalitat de Catalunya through the FI_SDUR-2021 grant (2021 FISDU 00142), which provided funding for his predoctoral training. BSC authors acknowledge financial support from the Departament de Recerca i Universitats de la Generalitat de Catalunya, research group MPiEDist (2021 SGR 00412). Author Fernando Vázquez is supported by PRE2022-104134 funded by MICIU/AEI /10.13039/501100011033 and by the FSE+.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publication Artículo científico (JournalArticle)

A machine learning estimator trained on synthetic data for real-time earthquake ground-shaking predictions in Southern California

  • Monterrubio-Velasco, Marisol
  • Callaghan, Scott
  • Modesto, David
  • Carrasco, Jose Carlos
  • Badía, Rosa M.
  • Vázquez-Novoa, Fernando
  • Pienkowska, Marta
  • de la Puente, Josep
  • Pallarés-Font de Mora, Pablo|||0000-0002-0120-3251
  • Quintana-Ortí, Enrique S.|||0000-0002-5454-165X
[EN] After large-magnitude earthquakes, a crucial task for impact assessment is to rapidly and accurately estimate the ground shaking in the affected region. To satisfy real-time constraints, intensity measures are traditionally evaluated with empirical Ground Motion Models that can drastically limit the accuracy of the estimated values. As an alternative, here we present Machine Learning strategies trained on physics-based simulations that require similar evaluation times. We trained and validated the proposed Machine Learning-based Estimator for ground shaking maps with one of the largest existing datasets (<100M simulated seismograms) from CyberShake developed by the Southern California Earthquake Center covering the Los Angeles basin. For a well-tailored synthetic database, our predictions outperform empirical Ground Motion Models provided that the events considered are compatible with the training data. Using the proposed strategy we show significant error reductions not only for synthetic, but also for five real historical earthquakes, relative to empirical Ground Motion Models., This work has been funded by the European Commission's Horizon 2020 Framework program and the European High-Performance Computing Joint Undertaking (JU) under grant agreement No 955558 and by MCIN/AEI/10.13039/501100011033 and the European Union NextGenerationEU/PRTR (PCI2021-121957), project eFlows4HPC. This research has been supported by the European High-Performance Computing Joint Undertaking (JU) as well as Spain, Italy, Iceland, Germany, Norway, France, Finland, and Croatia under grant agreement no. 101093038, (ChEESE-CoE) The authors acknowledge the Center for Advanced Research Computing (CARC) at the University of Southern California for providing computing resources that have contributed to the research results reported within this publication. URL: https://carc.usc.edu. This research used resources of the Oak Ridge Leadership Computing Facility, which is a DOE Office of Science User Facility supported under Contract DE-AC05-00OR22725, and the National Center for Supercomputer Applications under the Blue Waters Sustained-Petascale Computing Project. SCEC is funded by USGS Cooperative Agreement G17AC00047 and NSF Cooperative Agreement EAR-1600087. The authors also thank Dr. Kevin Milner, who provided the ASK-14 estimations needed for the comparisons presented in this paper. The authors thank Dr. Arnau Folch who reviewed the draft of the paper, improving it with his comments and suggestions.
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Artículo científico (JournalArticle)

Enabling dynamic and intelligent workflows for HPC, data analytics, and AI convergence

  • Ejarque, Jorge
  • Badia, Rosa M.
  • Albertin, Loïc
  • Aloisio, Giovanni
  • Baglione, Enrico
  • Becerra, Yolanda
  • Boschert, Stefan
  • Berlin, Julian R.
  • D’Anca, Alessandro
  • Elia, Donatello
  • Exertier, François
  • Fiore, Sandro
  • Flich, José
  • Folch, Arnau
  • Gibbons, Steven J.
  • Koldunov, Nikolay
  • Lordan, Francesc
  • Lorito, Stefano
  • Løvholt, Finn
  • Macías, Jorge
  • Volpe, Manuela
The evolution of High-Performance Computing (HPC) platforms enables the design and execution of progressively larger and more complex workflow applications in these systems. The complexity comes not only from the number of elements that compose the workflows but also from the type of computations they perform. While traditional HPC workflows target simulations and modelling of physical phenomena, current needs require in addition data analytics (DA) and artificial intelligence (AI) tasks. However, the development of these workflows is hampered by the lack of proper programming models and environments that support the integration of HPC, DA, and AI, as well as the lack of tools to easily deploy and execute the workflows in HPC systems. To progress in this direction, this paper presents use cases where complex workflows are required and investigates the main issues to be addressed for the HPC/DA/AI convergence. Based on this study, the paper identifies the challenges of a new workflow platform to manage complex workflows. Finally, it proposes a development approach for such a workflow platform addressing these challenges in two directions: first, by defining a software stack that provides the functionalities to manage these complex workflows; and second, by proposing the HPC Workflow as a Service (HPCWaaS) paradigm, which leverages the software stack to facilitate the reusability of complex workflows in federated HPC infrastructures. Proposals presented in this work are subject to study and development as part of the EuroHPC eFlows4HPC project., This work has received funding from the European High-Performance Computing Joint Undertaking (JU) under grant agreement No 955558. The JU receives support from the European Union’s Horizon 2020 research and innovation programme and Spain, Germany, France, Italy, Poland, Switzerland and Norway. In Spain, it has received complementary funding from MCIN/AEI/10.13039/501100011033, Spain and the European Union NextGenerationEU/PRTR (contracts PCI2021-121957, PCI2021-121931, PCI2021-121944, and PCI2021-121927). In Germany, it has received complementary funding from the German Federal Ministry of Education and Research (contracts 16HPC016K, 6GPC016K, 16HPC017 and 16HPC018). In France, it has received financial support from Caisse des dépôts et consignations (CDC) under the action PIA ADEIP (project Calculateurs). In Italy, it has been preliminary approved for complimentary funding by Ministero dello Sviluppo Economico (MiSE) (ref. project prop. 2659). In Norway, it has received complementary funding from the Norwegian Research Council, Norway under project number 323825. In Switzerland, it has been preliminary approved for complimentary funding by the State Secretariat for Education, Research, and Innovation (SERI), Norway. In Poland, it is partially supported by the National Centre for Research and Development under decision DWM/EuroHPCJU/4/2021. The authors also acknowledge financial support by MCIN/AEI /10.13039/501100011033, Spain through the “Severo Ochoa Programme for Centres of Excellence in R&D” under Grant CEX2018-000797-S, the Spanish Government, Spain (contract PID2019-107255 GB) and by Generalitat de Catalunya, Spain (contract 2017-SGR-01414). Anna Queralt is a Serra Húnter Fellow., With funding from the Spanish government through the ‘Severo Ochoa Centre of Excellence’ accreditation (CEX2018-000797-S).
UPCommons. Portal del coneixement obert de la UPC
Publication Artículo científico (JournalArticle)

Enhancing iteration performance on distributed task-based workflows

  • Barceló Cuerda, Alex|||0000-0002-0877-5738
  • Queralt Calafat, Anna|||0000-0003-2782-2955
  • Cortés, Toni|||0000-0002-2537-8937
Task-based programming models have proven to be a robust and versatile way to approach development of applications for distributed environments. They provide natural programming patterns with high performance. However, execution on this paradigm can be very sensitive to granularity –i.e., the quantity and execution length of tasks. Granularity is often linked with the block size of the data, and finding the optimal block size has several challenges, as it requires inner knowledge of the computing environment. Our proposal is to supplement the task-based programming model with a new mechanism –our SplIter proposal. At its core, the SplIter provides a transparent way to split a collection into partitions (logical groups of blocks, obtained without any transfers nor data rearrangement), which can then be iterated. Tasks are linked to those partitions, which means that SplIter breaks the dependency between block size and task granularity. The evaluation shows that the SplIter is able to achieve performance improvements of over one order of magnitude when compared to the baseline, and it is either competitive or strictly better (depending on application characteristics) to the competitor alternative. We have chosen different applications covering a wide variety of scenarios; those applications are representatives of a broader set of applications and domains. The changes required in the source code of a task-based application are minimal, preserving the high programmability of the programming model. Two different state-of-the-art task-based frameworks have been evaluated for all the applications: COMPSs and Dask, showing that the SplIter can be effectively used within different frameworks., This work has been funded by the European Commission’s Horizon 2020 Framework program and the European High-Performance Computing Joint Undertaking (JU) under grant agreement No 955558 and by MCIN/AEI/ 10.13039/501100011033 and the European Union NextGenerationEU/PRTR (PCI2021-121957). Anna Queralt is a Serra Húnter Fellow., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Artículo científico (JournalArticle)

Boosting HPC data analysis performance with the ParSoDA-Py library

  • Belcastro, Loris
  • Giampà, Salvatore
  • Marozzo, Fabrizio
  • Talia, Domenico
  • Trunfio, Paolo
  • Badia Sala, Rosa Maria|||0000-0003-2941-5499
  • Ejarque, Jorge|||0000-0003-4725-5097
  • Mammadli, Nihad
Developing and executing large-scale data analysis applications in parallel and distributed environments can be a complex and time-consuming task. Developers often find themselves diverted from their application logic to handle technical details about the underlying runtime and related issues. To simplify this process, ParSoDA, a Java library, has been proposed to facilitate the development of parallel data mining applications executed on HPC systems. It simplifies the process by providing built-in scalability mechanisms relying on the Hadoop and Spark frameworks. This paper presents ParSoDA-Py, the Python version of the ParSoDA library, which allows for further support of commonly used runtimes and libraries for big data analysis. After a complete library redesign, ParSoDA can be now easily integrated with other Python-based distributed runtimes for HPC systems, such as COMPSs and Apache Spark, and with the large ecosystem of Python-based data processing libraries. The paper discusses the adaptation process, which takes into consideration the new technical requirements, and evaluates both usability and scalability through some case study applications., This work has been partially funded by the European Commission’s Horizon 2020 Framework program and the European High-Performance Computing Joint Undertaking (JU) under Grant agreement No 955558 and by MCIN/AEI/10.13039/501100011033 and the European Union NextGenerationEU/PRTR (PCI2021-121957), project eFlows4HPC. It has also been supported by the Spanish Government (PID2019-107255GB) and by the Departament de Recerca i Universitats de la Generalitat de Catalunya to the Research Group MPiEDist (2021 SGR 00412) We also acknowledge financial support from “National Centre for HPC, Big Data and Quantum Computing," CN00000013 - CUP H23C22000360005, and from “FAIR - Future Artificial Intelligence Research" Project - CUP H23C22000860006., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Artículo científico (JournalArticle)

A machine learning estimator trained on synthetic data for real-time earthquake ground-shaking predictions in Southern California

  • Monterrubio Velasco, Marisol|||0000-0003-0790-1832
  • Callaghan, Scott
  • Modesto Galende, David|||0000-0001-9540-8815
  • Carrasco Jiménez, José Carlos
  • Badia Sala, Rosa Maria|||0000-0003-2941-5499
  • Pallarés Font de Mora, Pablo
  • Vázquez Novoa, Fernando
  • Quintana Ortí, Enrique Salvador
  • Pienkowska, Marta
  • de la Puente, Josep
After large-magnitude earthquakes, a crucial task for impact assessment is to rapidly and accurately estimate the ground shaking in the affected region. To satisfy real-time constraints, intensity measures are traditionally evaluated with empirical Ground Motion Models that can drastically limit the accuracy of the estimated values. As an alternative, here we present Machine Learning strategies trained on physics-based simulations that require similar evaluation times. We trained and validated the proposed Machine Learning-based Estimator for ground shaking maps with one of the largest existing datasets (<100M simulated seismograms) from CyberShake developed by the Southern California Earthquake Center covering the Los Angeles basin. For a well-tailored synthetic database, our predictions outperform empirical Ground Motion Models provided that the events considered are compatible with the training data. Using the proposed strategy we show significant error reductions not only for synthetic, but also for five real historical earthquakes, relative to empirical Ground Motion Models., This work has been funded by the European Commission’s Horizon 2020 Framework program and the European High-Performance Computing Joint Undertaking (JU) under grant agreement No 955558 and by MCIN/AEI/ 10.13039/501100011033 and the European Union NextGeneration EU/PRTR(PCI2021-121957), project eFlows4HPC. This research has been supported by the European High-Performance Computing Joint Undertaking (JU) as well as Spain, Italy, Iceland, Germany, Norway, France, Finland, and Croatia under grant agreement no. 101093038, (ChEESE-CoE). The authors acknowledge the Center for Advanced Research Computing (CARC) at the University of Southern California for providing computing resources that have contributed to the research results reported within this publication. URL: https://carc.usc.edu. This research used resources of the Oak Ridge Leadership Computing Facility, which is a DOE Office of Science User Facility supported under Contract DE-AC05-00OR22725, and the National Center for Supercomputer Applications under the Blue Waters Sustained-Petascale Computing Project. SCEC is funded by USGS Cooperative Agreement G17AC00047 and NSF Cooperative Agreement EAR1600087., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Managing workflow malleability in urgent computing for earthquake alerts

  • Ejarque Artigas, Jorge
  • Monterrubio Velasco, Marisol|||0000-0003-0790-1832
  • Bhihe, Cedric
  • Pienkowska, Marta
  • de la Puente, Josep
  • Badia Sala, Rosa Maria|||0000-0003-2941-5499
When large earthquakes happen, first responders need fast and accurate information regarding their impact. UCIS4EQ is an urgent computing platform that estimates ground shaking based on high-performance parallel 3D simulations. In this work, we present the PyCOMPSs implementation of UCIS4EQ towards urgent high-performance computing with a particular focus on providing a malleability mechanism. This allows UCIS4EQ to scan for data updates during runtime and dynamically incorporate said data into its execution. This involves, in particular, leveraging several HPC jobs. The implementation has been validated with data from the Samos Izmir earthquake. The malleable version of the workflow can reduce the use of computational resources by 40% while reducing the variability of the results and keeping the same time-to-solution as a non-malleable run., This work has been supported by the Spanish Government (PID2019-107255GB) and by MCIN/AEI/ 10.13039/501100011033 (CEX2021-001148-S), by Generalitat de Catalunya (contract 2021-SGR-00412), and by the European Commission’s Horizon 2020 Framework program and the European High-Performance Computing Joint Undertaking (JU) under grant agreement No 955558 and by MCIN/AEI/ 10.13039/501100011033 and the European Union NextGenerationEU/ PRTR (project eFlows4HPC, PCI2021-121957)., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Artículo científico (JournalArticle)

Extreme-scale workflows: a perspective from the JLESC international community

  • Yildiz, Orcun
  • Gueroudji, Amal
  • Bigot, Julien
  • Raffin, Bruno
  • Badia Sala, Rosa Maria|||0000-0003-2941-5499
  • Peterka, Tom
The Joint Laboratory for Extreme-Scale Computing (JLESC) focuses on software challenges in high-performance computing systems to meet the needs of today’s science campaigns, which often require large resources, consist of multiple tasks, and generate vast amounts of data. In this context, extreme-scale workflows have been the key factor in enabling scientific discoveries by helping scientists automate the dependencies and data exchanges between workflow tasks, instead of managing those manually. In this paper we present representative extreme-scale workflows and feature workflow systems developed by JLESC participating institutions. We present lessons learned while developing these tools, alongside with the open challenges and future research directions in the field of extreme-scale workflows., This research is supported by the U.S. Department of Energy, Office of Science, Office of Advanced Scientific Computing Research, under contract number DE-AC02- 06CH11357, program manager Margaret Lentz. The BSC author acknowledges the support of the Spanish Government (PID2019-107255GB) and by MCIN/AEI /10.13039 /501100011033 (CEX2021-001148-S), by the Departament de Recerca i Universitats de la Generalitat de Catalunya to the Research Group MPiEDist (2021 SGR 00412). The same author acknowledges the European Commission’s Horizon 2020 Framework program and the European High-Performance Computing Joint Undertaking (JU) under grant agreement No. 955558 and by MCIN/AEI/10.13039/501100011033 and the European Union NextGenerationEU/PRTR (PCI2021-121957), project eFlows4HPC. The INRIA author acknowledges the European Union’s Horizon research and innovation programme under grant agreement No 101144014., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

GPU cache system for COMPSs: A task-based distributed computing framework

  • Tatu, Cristian Catalin
  • Conejero Bañón, Francisco Javier
  • Vázquez Novoa, Fernando
  • Badia Sala, Rosa Maria|||0000-0003-2941-5499
In this paper, we propose a novel GPU cache system for COMPSs, a task-based distributed computing framework that enables the execution of parallel applications on heterogeneous clusters. GPU COMPSs tasks can exploit the computational power of NVIDIA GPUs to process large data blocks. However, the current implementation of COMPSs requires each task to write its output data to disk and the subsequent tasks to read them from disk, which introduces significant overhead. To overcome this limitation, we design and implement a GPU cache system that allows tasks to store and retrieve data from the GPU memory, avoiding unnecessary disk operations and reducing data transfer time. We conducted extensive experiments on several benchmarks and demonstrated that our GPU cache system can achieve significant speedups compared to the baseline COMPSs implementation., This work has been partially funded by the Spanish Government (PID2019-107255GB) y MCIN/AEI /10.13039/501100011033 (CEX2021-001148S), and by the Departament de Recerca i Universitats de la Generalitat de Catalunya, research group MPiEDist (2021 SGR 00412), and by the European Commission through the program Horizon 2020 and the European High-Performance Computing Joint Undertaking (JU) with contract 955558, project eFlows4HPC and by the MCIN/AEI/10.13039/501100011033 y European Union NextGenerationEU/PRTR (PCI2021-121957)., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Artículo científico (JournalArticle)

Block size estimation for data partitioning in HPC applications using machine learning techniques

  • Cantini, Riccardo
  • Marozzo, Fabrizio
  • Orsino, Alessio
  • Talia, Domenico
  • Trunfio, Paolo
  • Badia Sala, Rosa Maria|||0000-0003-2941-5499
  • Ejarque Artigas, Jorge
  • Vázquez-Novoa, Fernando
The extensive use of HPC infrastructures and frameworks for running data-intensive applications has led to a growing interest in data partitioning techniques and strategies. In fact, application performance can be heavily affected by how data are partitioned, which in turn depends on the selected size for data blocks, i.e. the block size. Therefore, finding an effective partitioning, i.e. a suitable block size, is a key strategy to speed-up parallel data-intensive applications and increase scalability. This paper describes a methodology, namely BLEST-ML (BLock size ESTimation through Machine Learning), for block size estimation that relies on supervised machine learning techniques. The proposed methodology was evaluated by designing an implementation tailored to dislib, a distributed computing library highly focused on machine learning algorithms built on top of the PyCOMPSs framework. We assessed the effectiveness of the provided implementation through an extensive experimental evaluation considering different algorithms from dislib, datasets, and infrastructures, including the MareNostrum 4 supercomputer. The results we obtained show the ability of BLEST-ML to efficiently determine a suitable way to split a given dataset, thus providing a proof of its applicability to enable the efficient execution of data-parallel applications in high performance environments., This work has been partially supported by the European Commission through the Horizon 2020 Research and Innovation program and the EuroHPC JU under contract 955558 (eFlows4HPC project) and by MCIN/AEI/10.13039/501100011033 and the European Union NextGenerationEU/PRTR (PCI2021‑121957 and CEX2021‑ 001148‑S) and by the Spanish Government (PID2019‑107255GB), and Generalitat de Catalunya (contract 2021‑SGR‑00412). We also acknowledge financial support from “National Centre for HPC, Big Data and Quantum Computing”, CN00000013 ‑ CUP H23C22000360005, and from “FAIR ‑ Future Artificial Intelligence Research” project ‑ CUP H23C22000860006., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Artículo científico (JournalArticle)

Parallel reduced-order modeling for digital twins using high-performance computing workflows

  • Ares de Parga Regalado, Sebastian|||0000-0001-5709-4683
  • Bravo Martínez, José Raúl|||0000-0002-4465-7536
  • Sibuet Ruiz, Nicolás|||0000-0003-2968-3653
  • Hernández Ortega, Joaquín Alberto|||0000-0001-9334-4002
  • Rossi, Riccardo|||0000-0003-0528-7074
  • Boschert, Stefan
  • Quintana Ortí, Enrique Salvador
  • Tomás Domínguez, Andrés Enrique
  • Tatu, Cristian Catalin
  • Vázquez Novoa, Fernando
  • Ejarque Artigas, Jorge
  • Badia Sala, Rosa Maria|||0000-0003-2941-5499
The integration of reduced-order models with high-performance computing is critical for developing digital twins, particularly for real-time monitoring and predictive maintenance of industrial systems. This paper presents a comprehensive, high-performance computing-enabled workflow for developing and deploying projection-based reduced-order models for large-scale mechanical simulations. We use PyCOMPSs’ parallel framework to efficiently execute reduced-order model training simulations, employing parallel singular value decomposition algorithms such as randomized singular value decomposition, Lanczos singular value decomposition, and full singular value decomposition based on tall-skinny QR. Moreover, we introduce a partitioned version of the hyperreduction scheme known as the Empirical Cubature Method to further enhance computational efficiency in projection-based reduced-order models for mechanical systems. Despite the widespread use of high-performance computing for projection-based reduced-order models, there is a significant lack of publications detailing comprehensive workflows for building and deploying end-to-end projection-based reduced-order models in high-performance computing environments. Our workflow is validated through a case study focusing on the thermal dynamics of a motor, a multiphysics problem involving convective heat transfer and mechanical components. The projection-based reduced-order model is designed to deliver a real-time prognosis tool that could enable rapid and safe motor restarts post-emergency shutdowns under different operating conditions, demonstrating its potential impact on the practice of simulations in engineering mechanics. To facilitate deployment, we use the High-Performance Computing Workflow as a Service strategy and Functional Mock-Up Units to ensure compatibility and ease of integration across high-performance computing, edge, and cloud environments. The outcomes illustrate the efficacy of combining projection-based reduced-order models and high-performance computing, establishing a precedent for scalable, real-time digital twin applications in computational mechanics across multiple industries., The authors acknowledge financial support from the Spanish Ministry of Economy and Competitiveness, through the “Severo Ochoa Programme for Centres of Excellence in R&D” (CEX2018-000797-S and CEX2021-001148-S)”, as well as to the project PID2019-107255GB (BSC authors). This project has received funding from the European High-Performance Computing Joint Undertaking (JU) under grant agreement No 955558. The JU receives support from the European Union’s Horizon 2020 research and innovation programme and Spain, Germany, France, Italy, Poland, Switzerland, Norway. This publication is part of the R&D project PCI2021-121944 and PCI2021-121957, financed by MCIN/AEI/10.13039/501100011033 and by the “European Union NextGenerationEU/PRTR”. Sebastian Ares de Parga Regalado acknowledges financial support from the Generalitat de Catalunya through the FI_SDUR-2021 grant (2021 FISDU 00142), which provided funding for his predoctoral training. BSC authors acknowledge financial support from the Departament de Recerca i Universitats de la Generalitat de Catalunya, research group MPiEDist (2021 SGR 00412). Author Fernando Vázquez is supported by PRE2022-104134 funded by MICIU/AEI /10.13039/501100011033 and by the FSE+., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Artículo científico (JournalArticle)

Model generation architecture for digital twins and deployment system

  • Vázquez Novoa, Fernando
  • Conejero Bañón, Francisco Javier
  • Badia Sala, Rosa Maria|||0000-0003-2941-5499
A Digital Twin is a virtual copy or representation of a physical object. Given the same input, both should produce the same output. A bidirectional data exchange between both of them should keep the objects in the same state by automatically reflecting changes from one of the objects to the other. Despite the popularity of the topic, there is a lack of standardization and no widespread adoption of techniques or architectures, resulting in ad-hoc implementations for each use case. This paper proposes a methodology for the generation of digital twins based on the definition of their life cycle, with phases that include training, operation and retraining. To allow efficient management of resources, each phase of the Digital Twin is implemented as a PyCOMPSs workflow. The proposed methodologies and architectures aims to simplify and strengthen the design and development of digital twins. It also allows for the extension of existing ones, the replacement of their components with more advanced ones, and the support of their reuse. We also consider workflow deployment, which is often a difficult task requiring specialized personnel and time. This article focuses on presenting the training phase of a Digital Twin life-cycle and demonstrates its operation by applying it to two real-world use case scenarios in ongoing projects. The use of the proposal led to a simplification of the developed code, allowed a distribution of the computational work among the available resources in a user-agnostic manner and the digital models created faithfully represent the physical object. Additionally, the paper presents the design of the operating life-cycle phase of a Digital Twin., Author Fernando Vázquez is supported by PRE2022-104134 funded by MICIU/AEI /10.13039/501100011033 and by the FSE+. This work has been partially funded by the the projects PID2019- 107255GB, CEX2021-001148-S, and PID2023-147979NB-C21 from the MCIN/AEI and MICIU/AEI /10.13039/501100011033 and by FEDER, UE, and by the Departament de Recerca i Universitats de la Generalitat de Catalunya, research group MPiEDist (2021 SGR 00412), and by the European Commission through the program Horizon 2020 and the European High-Performance Computing Joint Undertaking (JU) with contract 955558 and by the MCIN/AEI/10.13039/501100011033 y European Union NextGenerationEU/PRTR (PCI2021-121957), project eFlows4HPC; as well as by the European Commission under program Horizon Action under contract 101056886 (CAELESTIS). This work has also received funding from the European High Performance Computing Joint Undertaking (JU) and Spain, Italy, Iceland, Germany, Norway, France, Finland and Croatia under grant agreement No 101093038., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

End-to-End Workflows for Climate Science: Integrating HPC Simulations, Big Data Processing, and Machine Learning

  • Elia, Donatello
  • Scardigno, Sonia
  • Ejarque, Jorge|||0000-0003-4725-5097
  • D'anca, Alessandro
  • Accarino, Gabriele
  • Scoccimarro, Enrico
  • Donno, Davide
  • Peano, Daniele
  • Immorlano, Francesco
  • Aloisio, Giovanni
Current scientific workflow systems do not typically integrate simulation-centric and data-centric aspects due to their very different software/infrastructure requirements. A transparent integration of such components into a single end-to-end workflow would lead to a more efficient and automated way for generating insights from large simulation data. This work presents a complex case study related to extreme events analysis of future climate data that integrates in the same workflow numerical simulations, Big Data analytics and Machine Learning models. The case study is being implemented in the context of the eFlows4HPC project using the project’s software stack for deployment and orchestration of the workflow. The solution implemented in the project has shown to simplify the development and execution of end-to-end climate workflows with heterogeneous software requirements. Moreover, such an approach can, in the long term, increase the reuse of workflows by scientists and their portability over different HPC infrastructures., Thiswork has received funding from the European High-Performance Computing Joint Undertaking (JU) under grant agreement No 955558. The JU receives support from the European Union’s Horizon 2020 research and innovation programme and Spain, Germany, France, Italy, Poland, Switzerland and Norway. In Spain, it has received complementary funding from MCIN/AEI/10.13039/501100011033, Spain and the European Union NextGenerationEU/PRTR (contracts PCI2021-121957, PCI2021-121931, PCI2021-121944, and PCI2021- 121927). In Italy, it has been preliminary approved for complimentary funding by Ministero dello Sviluppo Economico (MiSE) (ref. project prop. 2659). The authors also acknowledge financial support by MCIN/AEI /10.13039/501100011033, Spain through the "Severo Ochoa Programme for Centres of Excellence in R&D" under Grant CEX2021-001148-S, the Spanish Government (contract PID2019- 107255 GB) and by Generalitat de Catalunya (contract 2021-SGR-00412)., Peer Reviewed
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Boosting HPC data analysis performance with the ParSoDA-Py library

  • Belcastro, Loris
  • Giampà, Salvatore
  • Marozzo, Fabrizio
  • Talia, Domenico
  • Trunfio, Paolo
  • Badia Sala, Rosa Maria
  • Ejarque, Jorge
  • Mammadli, Nihad
Developing and executing large-scale data analysis applications in parallel and distributed environments can be a complex and time-consuming task. Developers often find themselves diverted from their application logic to handle technical details about the underlying runtime and related issues. To simplify this process, ParSoDA, a Java library, has been proposed to facilitate the development of parallel data mining applications executed on HPC systems. It simplifies the process by providing built-in scalability mechanisms relying on the Hadoop and Spark frameworks. This paper presents ParSoDA-Py, the Python version of the ParSoDA library, which allows for further support of commonly used runtimes and libraries for big data analysis. After a complete library redesign, ParSoDA can be now easily integrated with other Python-based distributed runtimes for HPC systems, such as COMPSs and Apache Spark, and with the large ecosystem of Python-based data processing libraries. The paper discusses the adaptation process, which takes into consideration the new technical requirements, and evaluates both usability and scalability through some case study applications., This work has been partially funded by the European Commission’s Horizon 2020 Framework program and the European High-Performance Computing Joint Undertaking (JU) under Grant agreement No 955558 and by MCIN/AEI/10.13039/501100011033 and the European Union NextGenerationEU/PRTR (PCI2021-121957), project eFlows4HPC. It has also been supported by the Spanish Government (PID2019-107255GB) and by the Departament de Recerca i Universitats de la Generalitat de Catalunya to the Research Group MPiEDist (2021 SGR 00412) We also acknowledge financial support from “National Centre for HPC, Big Data and Quantum Computing," CN00000013 - CUP H23C22000360005, and from “FAIR - Future Artificial Intelligence Research" Project - CUP H23C22000860006., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Model generation architecture for digital twins and deployment system

  • Vázquez Novoa, Fernando
  • Conejero Bañón, Francisco Javier
  • Badia Sala, Rosa Maria
A Digital Twin is a virtual copy or representation of a physical object. Given the same input, both should produce the same output. A bidirectional data exchange between both of them should keep the objects in the same state by automatically reflecting changes from one of the objects to the other. Despite the popularity of the topic, there is a lack of standardization and no widespread adoption of techniques or architectures, resulting in ad-hoc implementations for each use case. This paper proposes a methodology for the generation of digital twins based on the definition of their life cycle, with phases that include training, operation and retraining. To allow efficient management of resources, each phase of the Digital Twin is implemented as a PyCOMPSs workflow. The proposed methodologies and architectures aims to simplify and strengthen the design and development of digital twins. It also allows for the extension of existing ones, the replacement of their components with more advanced ones, and the support of their reuse. We also consider workflow deployment, which is often a difficult task requiring specialized personnel and time. This article focuses on presenting the training phase of a Digital Twin life-cycle and demonstrates its operation by applying it to two real-world use case scenarios in ongoing projects. The use of the proposal led to a simplification of the developed code, allowed a distribution of the computational work among the available resources in a user-agnostic manner and the digital models created faithfully represent the physical object. Additionally, the paper presents the design of the operating life-cycle phase of a Digital Twin., Author Fernando Vázquez is supported by PRE2022-104134 funded by MICIU/AEI /10.13039/501100011033 and by the FSE+. This work has been partially funded by the the projects PID2019- 107255GB, CEX2021-001148-S, and PID2023-147979NB-C21 from the MCIN/AEI and MICIU/AEI /10.13039/501100011033 and by FEDER, UE, and by the Departament de Recerca i Universitats de la Generalitat de Catalunya, research group MPiEDist (2021 SGR 00412), and by the European Commission through the program Horizon 2020 and the European High-Performance Computing Joint Undertaking (JU) with contract 955558 and by the MCIN/AEI/10.13039/501100011033 y European Union NextGenerationEU/PRTR (PCI2021-121957), project eFlows4HPC; as well as by the European Commission under program Horizon Action under contract 101056886 (CAELESTIS). This work has also received funding from the European High Performance Computing Joint Undertaking (JU) and Spain, Italy, Iceland, Germany, Norway, France, Finland and Croatia under grant agreement No 101093038., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Parallel reduced-order modeling for digital twins using high-performance computing workflows

  • Ares de Parga Regalado, Sebastian
  • Bravo Martínez, José Raúl
  • Sibuet Ruiz, Nicolás
  • Hernández Ortega, Joaquín Alberto
  • Rossi, Riccardo
  • Boschert, Stefan
  • Quintana Ortí, Enrique Salvador
  • Tomás Domínguez, Andrés Enrique
  • Tatu, Cristian Catalin
  • Vázquez Novoa, Fernando
  • Ejarque Artigas, Jorge
  • Badia Sala, Rosa Maria
The integration of reduced-order models with high-performance computing is critical for developing digital twins, particularly for real-time monitoring and predictive maintenance of industrial systems. This paper presents a comprehensive, high-performance computing-enabled workflow for developing and deploying projection-based reduced-order models for large-scale mechanical simulations. We use PyCOMPSs’ parallel framework to efficiently execute reduced-order model training simulations, employing parallel singular value decomposition algorithms such as randomized singular value decomposition, Lanczos singular value decomposition, and full singular value decomposition based on tall-skinny QR. Moreover, we introduce a partitioned version of the hyperreduction scheme known as the Empirical Cubature Method to further enhance computational efficiency in projection-based reduced-order models for mechanical systems. Despite the widespread use of high-performance computing for projection-based reduced-order models, there is a significant lack of publications detailing comprehensive workflows for building and deploying end-to-end projection-based reduced-order models in high-performance computing environments. Our workflow is validated through a case study focusing on the thermal dynamics of a motor, a multiphysics problem involving convective heat transfer and mechanical components. The projection-based reduced-order model is designed to deliver a real-time prognosis tool that could enable rapid and safe motor restarts post-emergency shutdowns under different operating conditions, demonstrating its potential impact on the practice of simulations in engineering mechanics. To facilitate deployment, we use the High-Performance Computing Workflow as a Service strategy and Functional Mock-Up Units to ensure compatibility and ease of integration across high-performance computing, edge, and cloud environments. The outcomes illustrate the efficacy of combining projection-based reduced-order models and high-performance computing, establishing a precedent for scalable, real-time digital twin applications in computational mechanics across multiple industries., The authors acknowledge financial support from the Spanish Ministry of Economy and Competitiveness, through the “Severo Ochoa Programme for Centres of Excellence in R&D” (CEX2018-000797-S and CEX2021-001148-S)”, as well as to the project PID2019-107255GB (BSC authors). This project has received funding from the European High-Performance Computing Joint Undertaking (JU) under grant agreement No 955558. The JU receives support from the European Union’s Horizon 2020 research and innovation programme and Spain, Germany, France, Italy, Poland, Switzerland, Norway. This publication is part of the R&D project PCI2021-121944 and PCI2021-121957, financed by MCIN/AEI/10.13039/501100011033 and by the “European Union NextGenerationEU/PRTR”. Sebastian Ares de Parga Regalado acknowledges financial support from the Generalitat de Catalunya through the FI_SDUR-2021 grant (2021 FISDU 00142), which provided funding for his predoctoral training. BSC authors acknowledge financial support from the Departament de Recerca i Universitats de la Generalitat de Catalunya, research group MPiEDist (2021 SGR 00412). Author Fernando Vázquez is supported by PRE2022-104134 funded by MICIU/AEI /10.13039/501100011033 and by the FSE+., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Managing workflow malleability in urgent computing for earthquake alerts

  • Ejarque Artigas, Jorge
  • Monterrubio Velasco, Marisol
  • Bhihe, Cedric
  • Pienkowska, Marta
  • de la Puente, Josep
  • Badia Sala, Rosa Maria
When large earthquakes happen, first responders need fast and accurate information regarding their impact. UCIS4EQ is an urgent computing platform that estimates ground shaking based on high-performance parallel 3D simulations. In this work, we present the PyCOMPSs implementation of UCIS4EQ towards urgent high-performance computing with a particular focus on providing a malleability mechanism. This allows UCIS4EQ to scan for data updates during runtime and dynamically incorporate said data into its execution. This involves, in particular, leveraging several HPC jobs. The implementation has been validated with data from the Samos Izmir earthquake. The malleable version of the workflow can reduce the use of computational resources by 40% while reducing the variability of the results and keeping the same time-to-solution as a non-malleable run., This work has been supported by the Spanish Government (PID2019-107255GB) and by MCIN/AEI/ 10.13039/501100011033 (CEX2021-001148-S), by Generalitat de Catalunya (contract 2021-SGR-00412), and by the European Commission’s Horizon 2020 Framework program and the European High-Performance Computing Joint Undertaking (JU) under grant agreement No 955558 and by MCIN/AEI/ 10.13039/501100011033 and the European Union NextGenerationEU/ PRTR (project eFlows4HPC, PCI2021-121957)., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

A machine learning estimator trained on synthetic data for real-time earthquake ground-shaking predictions in Southern California

  • Monterrubio Velasco, Marisol
  • Callaghan, Scott
  • Modesto Galende, David
  • Carrasco Jiménez, José Carlos
  • Badia Sala, Rosa Maria
  • Pallarés Font de Mora, Pablo
  • Vázquez Novoa, Fernando
  • Quintana Ortí, Enrique Salvador
  • Pienkowska, Marta
  • de la Puente, Josep
After large-magnitude earthquakes, a crucial task for impact assessment is to rapidly and accurately estimate the ground shaking in the affected region. To satisfy real-time constraints, intensity measures are traditionally evaluated with empirical Ground Motion Models that can drastically limit the accuracy of the estimated values. As an alternative, here we present Machine Learning strategies trained on physics-based simulations that require similar evaluation times. We trained and validated the proposed Machine Learning-based Estimator for ground shaking maps with one of the largest existing datasets (<100M simulated seismograms) from CyberShake developed by the Southern California Earthquake Center covering the Los Angeles basin. For a well-tailored synthetic database, our predictions outperform empirical Ground Motion Models provided that the events considered are compatible with the training data. Using the proposed strategy we show significant error reductions not only for synthetic, but also for five real historical earthquakes, relative to empirical Ground Motion Models., This work has been funded by the European Commission’s Horizon 2020 Framework program and the European High-Performance Computing Joint Undertaking (JU) under grant agreement No 955558 and by MCIN/AEI/ 10.13039/501100011033 and the European Union NextGeneration EU/PRTR(PCI2021-121957), project eFlows4HPC. This research has been supported by the European High-Performance Computing Joint Undertaking (JU) as well as Spain, Italy, Iceland, Germany, Norway, France, Finland, and Croatia under grant agreement no. 101093038, (ChEESE-CoE). The authors acknowledge the Center for Advanced Research Computing (CARC) at the University of Southern California for providing computing resources that have contributed to the research results reported within this publication. URL: https://carc.usc.edu. This research used resources of the Oak Ridge Leadership Computing Facility, which is a DOE Office of Science User Facility supported under Contract DE-AC05-00OR22725, and the National Center for Supercomputer Applications under the Blue Waters Sustained-Petascale Computing Project. SCEC is funded by USGS Cooperative Agreement G17AC00047 and NSF Cooperative Agreement EAR1600087., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

End-to-End Workflows for Climate Science: Integrating HPC Simulations, Big Data Processing, and Machine Learning

  • Elia, Donatello
  • Scardigno, Sonia
  • Ejarque, Jorge
  • D'anca, Alessandro
  • Accarino, Gabriele
  • Scoccimarro, Enrico
  • Donno, Davide
  • Peano, Daniele
  • Immorlano, Francesco
  • Aloisio, Giovanni
Current scientific workflow systems do not typically integrate simulation-centric and data-centric aspects due to their very different software/infrastructure requirements. A transparent integration of such components into a single end-to-end workflow would lead to a more efficient and automated way for generating insights from large simulation data. This work presents a complex case study related to extreme events analysis of future climate data that integrates in the same workflow numerical simulations, Big Data analytics and Machine Learning models. The case study is being implemented in the context of the eFlows4HPC project using the project’s software stack for deployment and orchestration of the workflow. The solution implemented in the project has shown to simplify the development and execution of end-to-end climate workflows with heterogeneous software requirements. Moreover, such an approach can, in the long term, increase the reuse of workflows by scientists and their portability over different HPC infrastructures., Thiswork has received funding from the European High-Performance Computing Joint Undertaking (JU) under grant agreement No 955558. The JU receives support from the European Union’s Horizon 2020 research and innovation programme and Spain, Germany, France, Italy, Poland, Switzerland and Norway. In Spain, it has received complementary funding from MCIN/AEI/10.13039/501100011033, Spain and the European Union NextGenerationEU/PRTR (contracts PCI2021-121957, PCI2021-121931, PCI2021-121944, and PCI2021- 121927). In Italy, it has been preliminary approved for complimentary funding by Ministero dello Sviluppo Economico (MiSE) (ref. project prop. 2659). The authors also acknowledge financial support by MCIN/AEI /10.13039/501100011033, Spain through the "Severo Ochoa Programme for Centres of Excellence in R&D" under Grant CEX2021-001148-S, the Spanish Government (contract PID2019- 107255 GB) and by Generalitat de Catalunya (contract 2021-SGR-00412)., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

GPU cache system for COMPSs: A task-based distributed computing framework

  • Tatu, Cristian Catalin
  • Conejero Bañón, Francisco Javier
  • Vázquez Novoa, Fernando
  • Badia Sala, Rosa Maria
In this paper, we propose a novel GPU cache system for COMPSs, a task-based distributed computing framework that enables the execution of parallel applications on heterogeneous clusters. GPU COMPSs tasks can exploit the computational power of NVIDIA GPUs to process large data blocks. However, the current implementation of COMPSs requires each task to write its output data to disk and the subsequent tasks to read them from disk, which introduces significant overhead. To overcome this limitation, we design and implement a GPU cache system that allows tasks to store and retrieve data from the GPU memory, avoiding unnecessary disk operations and reducing data transfer time. We conducted extensive experiments on several benchmarks and demonstrated that our GPU cache system can achieve significant speedups compared to the baseline COMPSs implementation., This work has been partially funded by the Spanish Government (PID2019-107255GB) y MCIN/AEI /10.13039/501100011033 (CEX2021-001148S), and by the Departament de Recerca i Universitats de la Generalitat de Catalunya, research group MPiEDist (2021 SGR 00412), and by the European Commission through the program Horizon 2020 and the European High-Performance Computing Joint Undertaking (JU) with contract 955558, project eFlows4HPC and by the MCIN/AEI/10.13039/501100011033 y European Union NextGenerationEU/PRTR (PCI2021-121957)., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Block size estimation for data partitioning in HPC applications using machine learning techniques

  • Cantini, Riccardo
  • Marozzo, Fabrizio
  • Orsino, Alessio
  • Talia, Domenico
  • Trunfio, Paolo
  • Badia Sala, Rosa Maria
  • Ejarque Artigas, Jorge
  • Vázquez-Novoa, Fernando
The extensive use of HPC infrastructures and frameworks for running data-intensive applications has led to a growing interest in data partitioning techniques and strategies. In fact, application performance can be heavily affected by how data are partitioned, which in turn depends on the selected size for data blocks, i.e. the block size. Therefore, finding an effective partitioning, i.e. a suitable block size, is a key strategy to speed-up parallel data-intensive applications and increase scalability. This paper describes a methodology, namely BLEST-ML (BLock size ESTimation through Machine Learning), for block size estimation that relies on supervised machine learning techniques. The proposed methodology was evaluated by designing an implementation tailored to dislib, a distributed computing library highly focused on machine learning algorithms built on top of the PyCOMPSs framework. We assessed the effectiveness of the provided implementation through an extensive experimental evaluation considering different algorithms from dislib, datasets, and infrastructures, including the MareNostrum 4 supercomputer. The results we obtained show the ability of BLEST-ML to efficiently determine a suitable way to split a given dataset, thus providing a proof of its applicability to enable the efficient execution of data-parallel applications in high performance environments., This work has been partially supported by the European Commission through the Horizon 2020 Research and Innovation program and the EuroHPC JU under contract 955558 (eFlows4HPC project) and by MCIN/AEI/10.13039/501100011033 and the European Union NextGenerationEU/PRTR (PCI2021‑121957 and CEX2021‑ 001148‑S) and by the Spanish Government (PID2019‑107255GB), and Generalitat de Catalunya (contract 2021‑SGR‑00412). We also acknowledge financial support from “National Centre for HPC, Big Data and Quantum Computing”, CN00000013 ‑ CUP H23C22000360005, and from “FAIR ‑ Future Artificial Intelligence Research” project ‑ CUP H23C22000860006., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Enhancing iteration performance on distributed task-based workflows

  • Barceló Cuerda, Alex
  • Queralt Calafat, Anna
  • Cortés, Toni
Task-based programming models have proven to be a robust and versatile way to approach development of applications for distributed environments. They provide natural programming patterns with high performance. However, execution on this paradigm can be very sensitive to granularity –i.e., the quantity and execution length of tasks. Granularity is often linked with the block size of the data, and finding the optimal block size has several challenges, as it requires inner knowledge of the computing environment. Our proposal is to supplement the task-based programming model with a new mechanism –our SplIter proposal. At its core, the SplIter provides a transparent way to split a collection into partitions (logical groups of blocks, obtained without any transfers nor data rearrangement), which can then be iterated. Tasks are linked to those partitions, which means that SplIter breaks the dependency between block size and task granularity. The evaluation shows that the SplIter is able to achieve performance improvements of over one order of magnitude when compared to the baseline, and it is either competitive or strictly better (depending on application characteristics) to the competitor alternative. We have chosen different applications covering a wide variety of scenarios; those applications are representatives of a broader set of applications and domains. The changes required in the source code of a task-based application are minimal, preserving the high programmability of the programming model. Two different state-of-the-art task-based frameworks have been evaluated for all the applications: COMPSs and Dask, showing that the SplIter can be effectively used within different frameworks., This work has been funded by the European Commission’s Horizon 2020 Framework program and the European High-Performance Computing Joint Undertaking (JU) under grant agreement No 955558 and by MCIN/AEI/ 10.13039/501100011033 and the European Union NextGenerationEU/PRTR (PCI2021-121957). Anna Queralt is a Serra Húnter Fellow., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Extreme-scale workflows: a perspective from the JLESC international community

  • Yildiz, Orcun
  • Gueroudji, Amal
  • Bigot, Julien
  • Raffin, Bruno
  • Badia Sala, Rosa Maria
  • Peterka, Tom
The Joint Laboratory for Extreme-Scale Computing (JLESC) focuses on software challenges in high-performance computing systems to meet the needs of today’s science campaigns, which often require large resources, consist of multiple tasks, and generate vast amounts of data. In this context, extreme-scale workflows have been the key factor in enabling scientific discoveries by helping scientists automate the dependencies and data exchanges between workflow tasks, instead of managing those manually. In this paper we present representative extreme-scale workflows and feature workflow systems developed by JLESC participating institutions. We present lessons learned while developing these tools, alongside with the open challenges and future research directions in the field of extreme-scale workflows., This research is supported by the U.S. Department of Energy, Office of Science, Office of Advanced Scientific Computing Research, under contract number DE-AC02- 06CH11357, program manager Margaret Lentz. The BSC author acknowledges the support of the Spanish Government (PID2019-107255GB) and by MCIN/AEI /10.13039 /501100011033 (CEX2021-001148-S), by the Departament de Recerca i Universitats de la Generalitat de Catalunya to the Research Group MPiEDist (2021 SGR 00412). The same author acknowledges the European Commission’s Horizon 2020 Framework program and the European High-Performance Computing Joint Undertaking (JU) under grant agreement No. 955558 and by MCIN/AEI/10.13039/501100011033 and the European Union NextGenerationEU/PRTR (PCI2021-121957), project eFlows4HPC. The INRIA author acknowledges the European Union’s Horizon research and innovation programme under grant agreement No 101144014., Peer Reviewed, Postprint (author's final draft)