Pasar al contenido principal

ENABLING DYNAMIC AND INTELLIGENT WORKFLOWS IN THE FUTURE EUROHPCECOSYSTEM

PCI2021-121927

•
Nombre agencia financiadora Agencia Estatal de Investigación
Acrónimo agencia financiadora AEI
Programa Programa Estatal para Afrontar las Prioridades de Nuestro Entorno
Subprograma Subprograma Estatal de Internacionalización
Convocatoria Proyectos Programación Conjunta Internacional
Año convocatoria 2021
Plan Estatal 2017-2020 Plan Estatal de Investigación Científica y Técnica y de Innovación
Centro beneficiario UNIVERSIDAD DE MALAGA

Resultados relacionados

Resultados totales (Incluyendo duplicados): 3
Encontrada(s) 1 página(s)

DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

Enabling dynamic and intelligent workflows for HPC, data analytics, and AI convergence

  • Ejarque, Jorge
  • Badia, Rosa M.
  • Albertin, Loïc
  • Aloisio, Giovanni
  • Baglione, Enrico
  • Becerra, Yolanda
  • Boschert, Stefan
  • Berlin, Julian R.
  • D’Anca, Alessandro
  • Elia, Donatello
  • Exertier, François
  • Fiore, Sandro
  • Flich, José
  • Folch, Arnau
  • Gibbons, Steven J.
  • Koldunov, Nikolay
  • Lordan, Francesc
  • Lorito, Stefano
  • Løvholt, Finn
  • Macías, Jorge
  • Volpe, Manuela
The evolution of High-Performance Computing (HPC) platforms enables the design and execution of progressively larger and more complex workflow applications in these systems. The complexity comes not only from the number of elements that compose the workflows but also from the type of computations they perform. While traditional HPC workflows target simulations and modelling of physical phenomena, current needs require in addition data analytics (DA) and artificial intelligence (AI) tasks. However, the development of these workflows is hampered by the lack of proper programming models and environments that support the integration of HPC, DA, and AI, as well as the lack of tools to easily deploy and execute the workflows in HPC systems. To progress in this direction, this paper presents use cases where complex workflows are required and investigates the main issues to be addressed for the HPC/DA/AI convergence. Based on this study, the paper identifies the challenges of a new workflow platform to manage complex workflows. Finally, it proposes a development approach for such a workflow platform addressing these challenges in two directions: first, by defining a software stack that provides the functionalities to manage these complex workflows; and second, by proposing the HPC Workflow as a Service (HPCWaaS) paradigm, which leverages the software stack to facilitate the reusability of complex workflows in federated HPC infrastructures. Proposals presented in this work are subject to study and development as part of the EuroHPC eFlows4HPC project., This work has received funding from the European High-Performance Computing Joint Undertaking (JU) under grant agreement No 955558. The JU receives support from the European Union’s Horizon 2020 research and innovation programme and Spain, Germany, France, Italy, Poland, Switzerland and Norway. In Spain, it has received complementary funding from MCIN/AEI/10.13039/501100011033, Spain and the European Union NextGenerationEU/PRTR (contracts PCI2021-121957, PCI2021-121931, PCI2021-121944, and PCI2021-121927). In Germany, it has received complementary funding from the German Federal Ministry of Education and Research (contracts 16HPC016K, 6GPC016K, 16HPC017 and 16HPC018). In France, it has received financial support from Caisse des dépôts et consignations (CDC) under the action PIA ADEIP (project Calculateurs). In Italy, it has been preliminary approved for complimentary funding by Ministero dello Sviluppo Economico (MiSE) (ref. project prop. 2659). In Norway, it has received complementary funding from the Norwegian Research Council, Norway under project number 323825. In Switzerland, it has been preliminary approved for complimentary funding by the State Secretariat for Education, Research, and Innovation (SERI), Norway. In Poland, it is partially supported by the National Centre for Research and Development under decision DWM/EuroHPCJU/4/2021. The authors also acknowledge financial support by MCIN/AEI /10.13039/501100011033, Spain through the “Severo Ochoa Programme for Centres of Excellence in R&D” under Grant CEX2018-000797-S, the Spanish Government, Spain (contract PID2019-107255 GB) and by Generalitat de Catalunya, Spain (contract 2017-SGR-01414). Anna Queralt is a Serra Húnter Fellow., With funding from the Spanish government through the ‘Severo Ochoa Centre of Excellence’ accreditation (CEX2018-000797-S).
UPCommons. Portal del coneixement obert de la UPC
Publicación Contribución de congreso (ConferenceOutput)

End-to-End Workflows for Climate Science: Integrating HPC Simulations, Big Data Processing, and Machine Learning

  • Elia, Donatello
  • Scardigno, Sonia
  • Ejarque, Jorge|||0000-0003-4725-5097
  • D'anca, Alessandro
  • Accarino, Gabriele
  • Scoccimarro, Enrico
  • Donno, Davide
  • Peano, Daniele
  • Immorlano, Francesco
  • Aloisio, Giovanni
Current scientific workflow systems do not typically integrate simulation-centric and data-centric aspects due to their very different software/infrastructure requirements. A transparent integration of such components into a single end-to-end workflow would lead to a more efficient and automated way for generating insights from large simulation data. This work presents a complex case study related to extreme events analysis of future climate data that integrates in the same workflow numerical simulations, Big Data analytics and Machine Learning models. The case study is being implemented in the context of the eFlows4HPC project using the project’s software stack for deployment and orchestration of the workflow. The solution implemented in the project has shown to simplify the development and execution of end-to-end climate workflows with heterogeneous software requirements. Moreover, such an approach can, in the long term, increase the reuse of workflows by scientists and their portability over different HPC infrastructures., Thiswork has received funding from the European High-Performance Computing Joint Undertaking (JU) under grant agreement No 955558. The JU receives support from the European Union’s Horizon 2020 research and innovation programme and Spain, Germany, France, Italy, Poland, Switzerland and Norway. In Spain, it has received complementary funding from MCIN/AEI/10.13039/501100011033, Spain and the European Union NextGenerationEU/PRTR (contracts PCI2021-121957, PCI2021-121931, PCI2021-121944, and PCI2021- 121927). In Italy, it has been preliminary approved for complimentary funding by Ministero dello Sviluppo Economico (MiSE) (ref. project prop. 2659). The authors also acknowledge financial support by MCIN/AEI /10.13039/501100011033, Spain through the "Severo Ochoa Programme for Centres of Excellence in R&D" under Grant CEX2021-001148-S, the Spanish Government (contract PID2019- 107255 GB) and by Generalitat de Catalunya (contract 2021-SGR-00412)., Peer Reviewed
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

End-to-End Workflows for Climate Science: Integrating HPC Simulations, Big Data Processing, and Machine Learning

  • Elia, Donatello
  • Scardigno, Sonia
  • Ejarque, Jorge
  • D'anca, Alessandro
  • Accarino, Gabriele
  • Scoccimarro, Enrico
  • Donno, Davide
  • Peano, Daniele
  • Immorlano, Francesco
  • Aloisio, Giovanni
Current scientific workflow systems do not typically integrate simulation-centric and data-centric aspects due to their very different software/infrastructure requirements. A transparent integration of such components into a single end-to-end workflow would lead to a more efficient and automated way for generating insights from large simulation data. This work presents a complex case study related to extreme events analysis of future climate data that integrates in the same workflow numerical simulations, Big Data analytics and Machine Learning models. The case study is being implemented in the context of the eFlows4HPC project using the project’s software stack for deployment and orchestration of the workflow. The solution implemented in the project has shown to simplify the development and execution of end-to-end climate workflows with heterogeneous software requirements. Moreover, such an approach can, in the long term, increase the reuse of workflows by scientists and their portability over different HPC infrastructures., Thiswork has received funding from the European High-Performance Computing Joint Undertaking (JU) under grant agreement No 955558. The JU receives support from the European Union’s Horizon 2020 research and innovation programme and Spain, Germany, France, Italy, Poland, Switzerland and Norway. In Spain, it has received complementary funding from MCIN/AEI/10.13039/501100011033, Spain and the European Union NextGenerationEU/PRTR (contracts PCI2021-121957, PCI2021-121931, PCI2021-121944, and PCI2021- 121927). In Italy, it has been preliminary approved for complimentary funding by Ministero dello Sviluppo Economico (MiSE) (ref. project prop. 2659). The authors also acknowledge financial support by MCIN/AEI /10.13039/501100011033, Spain through the "Severo Ochoa Programme for Centres of Excellence in R&D" under Grant CEX2021-001148-S, the Spanish Government (contract PID2019- 107255 GB) and by Generalitat de Catalunya (contract 2021-SGR-00412)., Peer Reviewed, Postprint (author's final draft)