Skip to main content

Resultados relacionados

Resultados totales (Incluyendo duplicados): 45
Encontrada(s) 1 página(s)

RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publication Contribución de congreso (ConferenceOutput)

Towards Efficient Neural Network Model Parallelism on Multi-FPGA Platforms

  • Rodríguez-Agut, David
  • Tornero-Gavilá, Rafael
  • Flich Cardo, José
[EN] Nowadays, convolutional neural networks (CNN) are common in a wide range of applications. Their high accuracy and efficiency contrast with their computing requirements, leading to the search for efficient hardware platforms. FPGAs are suitable due to their flexibility, energy efficiency and low latency. However, the ever increasing complexity of CNNs demands higher capacity devices, forcing the need for multi-FPGA platforms. In this paper, we present a multi-FPGA platform with distributed shared memory support for the inference of CNNs. Our solution, in contrast with previous works, enables combining different model parallelism strategies applied to CNNs, thanks to the distributed shared memory support. For a four FPGA setting, the platform reduces the execution time of 2D convolutions by a factor of 3.95 when compared to single FPGA. The inference of standard CNN models is improved by factors ranging 3.63-3.87.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publication Artículo científico (JournalArticle)

Algorithm 1039: Automatic Generators for a Family of Matrix Multiplication Routines with Apache TVM

  • Alaejos-López, Guillermo
  • Igual, Francisco D.
  • Martínez, Héctor
  • Castelló, Adrián|||0000-0002-8576-8451
  • Alonso-Jordá, Pedro|||0000-0002-6882-6592
  • Quintana-Ortí, Enrique S.|||0000-0002-5454-165X
This work was supported by the research projects PID2020-113656RB-C22 (MCIN/AEI/10.13039/ 501100011033) and PID2021-126576NB-I00; and CM via Multiannual Agreement with Complutense University in the line Program to Stimulate Research for Young Doctors in the context of the V PRICIT under projects PR65/19-22445 and CM S2018/TCS-4423. A. Castello is a FJC2019-039222-I fellow supported by MCIN/AEI/10.13039/ 501100011033. H. Martinez is a postdoctoral fellow supported by the Consejeria de Transformacion Economica, Industria, Conocimiento y Universidades de la Junta de Andalucia. This project has received funding from the European High-Performance Computing Joint Undertaking (JU) under grant agreement No 955558 (eFlows4HPC project). The JU receives support from the European Union's Horizon 2020 research and innovation programme, and Spain, Germany, France, Italy, Poland, Switzerland, and Norway.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publication Artículo científico (JournalArticle)

Parallel reduced-order modeling for digital twins using high-performance computing workflows

  • Ares de Parga, S.
  • Bravo, J. R.
  • Sibuet, N.
  • Hernandez, J. A.
  • Rossi, R.
  • Boschert, Stefan
  • Tatu, Cristian Catalin
  • Vázquez-Novoa, Fernando
  • Ejarque, Jorge
  • Badia, Rosa M.
  • Quintana-Ortí, Enrique S.|||0000-0002-5454-165X
  • Tomás Domínguez, Andrés Enrique
[EN] The integration of reduced-order models with high-performance computing is critical for developing digital twins, particularly for real-time monitoring and predictive maintenance of industrial systems. This paper presents a comprehensive, high-performance computing-enabled workflow for developing and deploying projection-based reduced-order models for large-scale mechanical simulations. We use PyCOMPSs' parallel framework to efficiently execute reduced-order model training simulations, employing parallel singular value decomposition algorithms such as randomized singular value decomposition, Lanczos singular value decomposition, and full singular value decomposition based on tall-skinny QR. Moreover, we introduce a partitioned version of the hyperreduction scheme known as the Empirical Cubature Method to further enhance computational efficiency in projection-based reduced-order models for mechanical systems. Despite the widespread use of high-performance computing for projection-based reduced-order models, there is a significant lack of publications detailing comprehensive workflows for building and deploying end-to-end projection-based reduced-order models in high-performance computing environments. Our workflow is validated through a case study focusing on the thermal dynamics of a motor, a multiphysics problem involving convective heat transfer and mechanical components. The projection-based reduced-order model is designed to deliver a real-time prognosis tool that could enable rapid and safe motor restarts post-emergency shutdowns under different operating conditions, demonstrating its potential impact on the practice of simulations in engineering mechanics. To facilitate deployment, we use the High-Performance Computing Workflow as a Service strategy and Functional Mock-Up Units to ensure compatibility and ease of integration across high-performance computing, edge, and cloud environments. The outcomes illustrate the efficacy of combining projection-based reduced-order models and high-performance computing, establishing a precedent for scalable, real-time digital twin applications in computational mechanics across multiple industries., The authors acknowledge financial support from the Spanish Ministry of Economy and Competitiveness, through the Severo Ochoa Programme for Centres of Excellence in R&D (CEX2018-000797-S and CEX2021-001148-S) , as well as to the project PID2019-107255GB (BSC authors). This project has received funding from the European High-Performance Computing Joint Undertaking (JU) under grant agreement No 955558. The JU receives support from the European Union s Horizon 2020 research and innovation programme and Spain, Germany, France, Italy, Poland, Switzerland, Norway. This publication is part of the R&D project PCI2021-121944 and PCI2021-121957, financed by MCIN/AEI/10.13039/501100011033 and by the European Union NextGenerationEU/PRTR . Sebastian Ares de Parga Regalado acknowledges financial support from the Generalitat de Catalunya through the FI_SDUR-2021 grant (2021 FISDU 00142), which provided funding for his predoctoral training. BSC authors acknowledge financial support from the Departament de Recerca i Universitats de la Generalitat de Catalunya, research group MPiEDist (2021 SGR 00412). Author Fernando Vázquez is supported by PRE2022-104134 funded by MICIU/AEI /10.13039/501100011033 and by the FSE+.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publication Artículo científico (JournalArticle)

Programming parallel dense matrix factorizations and inversion for new-generation NUMA architectures

  • Catalán, Sandra
  • Igual, Francisco D.
  • Herrero, José R.
  • Rodríguez-Sánchez, Rafael
  • Quintana-Ortí, Enrique S.|||0000-0002-5454-165X
[EN] We propose a methodology to address the programmability issues derived from the emergence of newgeneration shared-memory NUMA architectures. For this purpose, we employ dense matrix factorizations and matrix inversion (DMFI) as a use case, and we target two modern architectures (AMD Rome and Huawei Kunpeng 920) that exhibit configurable NUMA topologies. Our methodology pursues performance portability across different NUMA configurations by proposing multi-domain implementations for DMFI plus a hybrid task- and loop-level parallelization that configures multi-threaded executions to fix core-todata binding, exploiting locality at the expense of minor code modifications. In addition, we introduce a generalization of the multi-domain implementations for DMFI that offers support for virtually any NUMA topology in present and future architectures. Our experimentation on the two target architectures for three representative dense linear algebra operations validates the proposal, reveals insights on the necessity of adapting both the codes and their execution to improve data access locality, and reports performance across architectures and inter- and intra-socket NUMA configurations competitive with state-of-the-art message-passing implementations, maintaining the ease of development usually associated with shared-memory programming., This research was sponsored by project PID2019-107255GB of Ministerio de Ciencia, Innovacion y Universidades; project S2018/TCS-4423 of Comunidad de Madrid; project 2017-SGR-1414 of the Generalitat de Catalunya and the Madrid Government under the Multiannual Agreement with UCM in the line Pro-gram to Stimulate Research for Young Doctors in the context of the V PRICIT, project PR65/19-22445. This project has also re-ceived funding from the European High-Performance Computing Joint Undertaking (JU) under grant agreement No 955558. The JU receives support from the European Union's Horizon 2020 research and innovation programme, and Spain, Germany, France, Italy, Poland, Switzerland, Norway. The work is also supported by grants PID2020-113656RB-C22 and PID2021-126576NB-I00 of MCIN/AEI/10.13039/501100011033 and by ERDF A way of making Europe.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publication Artículo científico (JournalArticle)

A BLIS-like matrix multiplication for machine learning in the RISC-V ISA-based GAP8 processor

  • Ramírez-Betancourth, Cristian
  • Castelló, Adrián|||0000-0002-8576-8451
  • Quintana-Ortí, Enrique S.|||0000-0002-5454-165X
[EN] We address the efficient realization of matrix multiplication (gemm), with application in the convolution operator for machine learning, for the RISC-V core present in the GreenWaves GAP8 processor. Our approach leverages BLIS (Basic Linear Algebra Instantiation Software) to develop an implementation that (1) re-organizes the gemm algorithm adapting its micro-kernel to exploit the hardware-supported dot product kernel in the GAP8; (2) explicitly orchestrates the data transfers across the hierarchy of scratchpad memories via DMA (direct memory access); and (3) operates with integer arithmetic., This work was supported by the research project PID2020-113656RB-C22 of MCIN/AEI/10.13039/501100011033. C. Ramirez is a "Santiago Grisolia" fellow supported by Generalitat Valenciana. Adrian Castello is a FJC2019-039222-I fellow supported by MCIN/AEI/10.13039/501100011033. This project has received funding from the European High-Performance Computing Joint Undertaking (JU) under Grant Agreement No. 955558. The JU receives support from the European Union's Horizon 2020 research and innovation program, and Spain, Germany, France, Italy, Poland, Switzerland, Norway.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publication Artículo científico (JournalArticle)

A machine learning estimator trained on synthetic data for real-time earthquake ground-shaking predictions in Southern California

  • Monterrubio-Velasco, Marisol
  • Callaghan, Scott
  • Modesto, David
  • Carrasco, Jose Carlos
  • Badía, Rosa M.
  • Vázquez-Novoa, Fernando
  • Pienkowska, Marta
  • de la Puente, Josep
  • Pallarés-Font de Mora, Pablo|||0000-0002-0120-3251
  • Quintana-Ortí, Enrique S.|||0000-0002-5454-165X
[EN] After large-magnitude earthquakes, a crucial task for impact assessment is to rapidly and accurately estimate the ground shaking in the affected region. To satisfy real-time constraints, intensity measures are traditionally evaluated with empirical Ground Motion Models that can drastically limit the accuracy of the estimated values. As an alternative, here we present Machine Learning strategies trained on physics-based simulations that require similar evaluation times. We trained and validated the proposed Machine Learning-based Estimator for ground shaking maps with one of the largest existing datasets (<100M simulated seismograms) from CyberShake developed by the Southern California Earthquake Center covering the Los Angeles basin. For a well-tailored synthetic database, our predictions outperform empirical Ground Motion Models provided that the events considered are compatible with the training data. Using the proposed strategy we show significant error reductions not only for synthetic, but also for five real historical earthquakes, relative to empirical Ground Motion Models., This work has been funded by the European Commission's Horizon 2020 Framework program and the European High-Performance Computing Joint Undertaking (JU) under grant agreement No 955558 and by MCIN/AEI/10.13039/501100011033 and the European Union NextGenerationEU/PRTR (PCI2021-121957), project eFlows4HPC. This research has been supported by the European High-Performance Computing Joint Undertaking (JU) as well as Spain, Italy, Iceland, Germany, Norway, France, Finland, and Croatia under grant agreement no. 101093038, (ChEESE-CoE) The authors acknowledge the Center for Advanced Research Computing (CARC) at the University of Southern California for providing computing resources that have contributed to the research results reported within this publication. URL: https://carc.usc.edu. This research used resources of the Oak Ridge Leadership Computing Facility, which is a DOE Office of Science User Facility supported under Contract DE-AC05-00OR22725, and the National Center for Supercomputer Applications under the Blue Waters Sustained-Petascale Computing Project. SCEC is funded by USGS Cooperative Agreement G17AC00047 and NSF Cooperative Agreement EAR-1600087. The authors also thank Dr. Kevin Milner, who provided the ASK-14 estimations needed for the comparisons presented in this paper. The authors thank Dr. Arnau Folch who reviewed the draft of the paper, improving it with his comments and suggestions.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publication Artículo científico (JournalArticle)

Sparse matrix-vector and matrix-multivector products for the truncated SVD on graphics processors

  • Aliaga, José I.
  • Anzt, Hartwig
  • Quintana-Ortí, Enrique S.|||0000-0002-5454-165X
  • Tomás Domínguez, Andrés Enrique
[EN] Many practical algorithms for numerical rank computations implement an iterative procedure that involves repeated multiplications of a vector, or a collection of vectors, with both a sparse matrix A$$ A $$ and its transpose. Unfortunately, the realization of these sparse products on current high performance libraries often deliver much lower arithmetic throughput when the matrix involved in the product is transposed. In this work, we propose a hybrid sparse matrix layout, named CSRC, that combines the flexibility of some well-known sparse formats to offer a number of appealing properties: (1) CSRC can be obtained at low cost from the popular CSR (compressed sparse row) format; (2) CSRC has similar storage requirements as CSR; and especially, (3) the implementation of the sparse product kernels delivers high performance for both the direct product and its transposed variant on modern graphics accelerators thanks to a significant reduction of atomic operations compared to a conventional implementation based on CSR. This solution thus renders considerably higher performance when integrated into an iterative algorithm for the truncated singular value decomposition (SVD), such as the randomized SVD or, as demonstrated in the experimental results, the block Golub-Kahan-Lanczos algorithm., This work received funding from the US Exascale Computing Project (17-SC-20-SC), with the collaboration of US Department of Energy Office of Science; Spanish National Plan for Scientific and Technical Research and Innovation: PID2020-113656RB of MCIN/AEI/10.13039/501100011033; Universitat Jaume I: UJI-B2021-58; and the European High-Performance Computing Joint Undertaking (JU) under grant agreement number 955558 (eFlows4HPC project). The JU receives support from the European Union s Horizon 2020 Research and Innovation Programme, and Spain, Germany, France, Italy, Poland, Switzerland, Norway.
UPCommons. Portal del coneixement obert de la UPC
Publication Artículo científico (JournalArticle)

Enhancing iteration performance on distributed task-based workflows

  • Barceló Cuerda, Alex|||0000-0002-0877-5738
  • Queralt Calafat, Anna|||0000-0003-2782-2955
  • Cortés, Toni|||0000-0002-2537-8937
Task-based programming models have proven to be a robust and versatile way to approach development of applications for distributed environments. They provide natural programming patterns with high performance. However, execution on this paradigm can be very sensitive to granularity –i.e., the quantity and execution length of tasks. Granularity is often linked with the block size of the data, and finding the optimal block size has several challenges, as it requires inner knowledge of the computing environment. Our proposal is to supplement the task-based programming model with a new mechanism –our SplIter proposal. At its core, the SplIter provides a transparent way to split a collection into partitions (logical groups of blocks, obtained without any transfers nor data rearrangement), which can then be iterated. Tasks are linked to those partitions, which means that SplIter breaks the dependency between block size and task granularity. The evaluation shows that the SplIter is able to achieve performance improvements of over one order of magnitude when compared to the baseline, and it is either competitive or strictly better (depending on application characteristics) to the competitor alternative. We have chosen different applications covering a wide variety of scenarios; those applications are representatives of a broader set of applications and domains. The changes required in the source code of a task-based application are minimal, preserving the high programmability of the programming model. Two different state-of-the-art task-based frameworks have been evaluated for all the applications: COMPSs and Dask, showing that the SplIter can be effectively used within different frameworks., This work has been funded by the European Commission’s Horizon 2020 Framework program and the European High-Performance Computing Joint Undertaking (JU) under grant agreement No 955558 and by MCIN/AEI/ 10.13039/501100011033 and the European Union NextGenerationEU/PRTR (PCI2021-121957). Anna Queralt is a Serra Húnter Fellow., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Artículo científico (JournalArticle)

Boosting HPC data analysis performance with the ParSoDA-Py library

  • Belcastro, Loris
  • Giampà, Salvatore
  • Marozzo, Fabrizio
  • Talia, Domenico
  • Trunfio, Paolo
  • Badia Sala, Rosa Maria|||0000-0003-2941-5499
  • Ejarque, Jorge|||0000-0003-4725-5097
  • Mammadli, Nihad
Developing and executing large-scale data analysis applications in parallel and distributed environments can be a complex and time-consuming task. Developers often find themselves diverted from their application logic to handle technical details about the underlying runtime and related issues. To simplify this process, ParSoDA, a Java library, has been proposed to facilitate the development of parallel data mining applications executed on HPC systems. It simplifies the process by providing built-in scalability mechanisms relying on the Hadoop and Spark frameworks. This paper presents ParSoDA-Py, the Python version of the ParSoDA library, which allows for further support of commonly used runtimes and libraries for big data analysis. After a complete library redesign, ParSoDA can be now easily integrated with other Python-based distributed runtimes for HPC systems, such as COMPSs and Apache Spark, and with the large ecosystem of Python-based data processing libraries. The paper discusses the adaptation process, which takes into consideration the new technical requirements, and evaluates both usability and scalability through some case study applications., This work has been partially funded by the European Commission’s Horizon 2020 Framework program and the European High-Performance Computing Joint Undertaking (JU) under Grant agreement No 955558 and by MCIN/AEI/10.13039/501100011033 and the European Union NextGenerationEU/PRTR (PCI2021-121957), project eFlows4HPC. It has also been supported by the Spanish Government (PID2019-107255GB) and by the Departament de Recerca i Universitats de la Generalitat de Catalunya to the Research Group MPiEDist (2021 SGR 00412) We also acknowledge financial support from “National Centre for HPC, Big Data and Quantum Computing," CN00000013 - CUP H23C22000360005, and from “FAIR - Future Artificial Intelligence Research" Project - CUP H23C22000860006., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Artículo científico (JournalArticle)

A kinematically stabilized linear tetrahedral finite element for compressible and nearly incompressible finite elasticity

  • Scovazzi, Guglielmo
  • Zorrilla Martínez, Rubén|||0000-0001-8270-7170
  • Rossi, Riccardo|||0000-0003-0528-7074
We propose a stabilized linear tetrahedral finite element method for static, finite elasticity problems involving compressible and nearly incompressible materials. Our approach relies on a mixed formulation, in which the nodal displacement unknown filed is complemented by a nodal Jacobian determinant unknown field. This approach is simple to implement in practical applications (e.g., in commercial software), since it only requires information already available when computing the Newton–Raphson tangent matrix associated with irreducible (i.e., displacement-based) finite element formulations. By nature, the proposed method is easily extensible to nonlinear models involving visco-plastic flow. An extensive suite of numerical tests in two and three dimensions is presented, to demonstrate the performance of the method., This research is partly supported by the European High-Performance Computing Joint Undertaking (JU) through the project eFlows4HPC (grant agreement number 955558). The JU receives support from the European Union Horizon 2020 research and innovation program and Spain, Germany, France, Italy, Poland, Switzerland, Norway. This publication is also part of the R&D project PCI2021-121944, financed by MCIN/AEI/10.13039/501100011033 and by the “European Union NextGenerationEU/PRTR”. The authors also acknowledge financial support from the Spanish Ministry of Economy and Competitiveness , through the “Severo Ochoa Programme for Centres of Excellence in R&D” (CEX2018-000797-S)., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Artículo científico (JournalArticle)

A machine learning estimator trained on synthetic data for real-time earthquake ground-shaking predictions in Southern California

  • Monterrubio Velasco, Marisol|||0000-0003-0790-1832
  • Callaghan, Scott
  • Modesto Galende, David|||0000-0001-9540-8815
  • Carrasco Jiménez, José Carlos
  • Badia Sala, Rosa Maria|||0000-0003-2941-5499
  • Pallarés Font de Mora, Pablo
  • Vázquez Novoa, Fernando
  • Quintana Ortí, Enrique Salvador
  • Pienkowska, Marta
  • de la Puente, Josep
After large-magnitude earthquakes, a crucial task for impact assessment is to rapidly and accurately estimate the ground shaking in the affected region. To satisfy real-time constraints, intensity measures are traditionally evaluated with empirical Ground Motion Models that can drastically limit the accuracy of the estimated values. As an alternative, here we present Machine Learning strategies trained on physics-based simulations that require similar evaluation times. We trained and validated the proposed Machine Learning-based Estimator for ground shaking maps with one of the largest existing datasets (<100M simulated seismograms) from CyberShake developed by the Southern California Earthquake Center covering the Los Angeles basin. For a well-tailored synthetic database, our predictions outperform empirical Ground Motion Models provided that the events considered are compatible with the training data. Using the proposed strategy we show significant error reductions not only for synthetic, but also for five real historical earthquakes, relative to empirical Ground Motion Models., This work has been funded by the European Commission’s Horizon 2020 Framework program and the European High-Performance Computing Joint Undertaking (JU) under grant agreement No 955558 and by MCIN/AEI/ 10.13039/501100011033 and the European Union NextGeneration EU/PRTR(PCI2021-121957), project eFlows4HPC. This research has been supported by the European High-Performance Computing Joint Undertaking (JU) as well as Spain, Italy, Iceland, Germany, Norway, France, Finland, and Croatia under grant agreement no. 101093038, (ChEESE-CoE). The authors acknowledge the Center for Advanced Research Computing (CARC) at the University of Southern California for providing computing resources that have contributed to the research results reported within this publication. URL: https://carc.usc.edu. This research used resources of the Oak Ridge Leadership Computing Facility, which is a DOE Office of Science User Facility supported under Contract DE-AC05-00OR22725, and the National Center for Supercomputer Applications under the Blue Waters Sustained-Petascale Computing Project. SCEC is funded by USGS Cooperative Agreement G17AC00047 and NSF Cooperative Agreement EAR1600087., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Artículo científico (JournalArticle)

Dynamic resource allocation for efficient parallel CFD simulations

  • Houzeaux, Guillaume|||0000-0002-2592-1426
  • Badia Sala, Rosa Maria|||0000-0003-2941-5499
  • Borrell Pol, Ricard
  • Dosimont, Damien
  • Ejarque Artigas, Jorge
  • Garcia Gasulla, Marta|||0000-0003-3682-9905
  • López Herrero, Víctor
CFD users of supercomputers usually resort to rule-of-thumb methods to select the number of subdomains (partitions) when relying on MPI-based parallelization. One common approach is to set a minimum number of elements or cells per subdomain, under which the parallel efficiency of the code is “known” to fall below a subjective level, say 80%. The situation is even worse when the user is not aware of the “good” practices for the given code and a huge amount of resources can thus be wasted. This work presents an elastic computing methodology that adapts at runtime the resources allocated to a simulation automatically. The criterion to control the required resources is based on a runtime measure of the communication efficiency of the execution. According to some analytical estimates, the resources are then expanded or reduced to fulfill this criterion and eventually execute an efficient simulation., This work has been supported by the Spanish Government (PID2019- 107255GB); by Generalitat de Catalunya (contract 2014-SGR-1051); by the European Commission H2020 project POP CoE (GA n. 824080); by the European Commission H2020 project CompBioMed CoE (GA n. 823712) and by the European Commission and the EuroHPC JU under contract 955558 (eFlows4HPC project)., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Managing workflow malleability in urgent computing for earthquake alerts

  • Ejarque Artigas, Jorge
  • Monterrubio Velasco, Marisol|||0000-0003-0790-1832
  • Bhihe, Cedric
  • Pienkowska, Marta
  • de la Puente, Josep
  • Badia Sala, Rosa Maria|||0000-0003-2941-5499
When large earthquakes happen, first responders need fast and accurate information regarding their impact. UCIS4EQ is an urgent computing platform that estimates ground shaking based on high-performance parallel 3D simulations. In this work, we present the PyCOMPSs implementation of UCIS4EQ towards urgent high-performance computing with a particular focus on providing a malleability mechanism. This allows UCIS4EQ to scan for data updates during runtime and dynamically incorporate said data into its execution. This involves, in particular, leveraging several HPC jobs. The implementation has been validated with data from the Samos Izmir earthquake. The malleable version of the workflow can reduce the use of computational resources by 40% while reducing the variability of the results and keeping the same time-to-solution as a non-malleable run., This work has been supported by the Spanish Government (PID2019-107255GB) and by MCIN/AEI/ 10.13039/501100011033 (CEX2021-001148-S), by Generalitat de Catalunya (contract 2021-SGR-00412), and by the European Commission’s Horizon 2020 Framework program and the European High-Performance Computing Joint Undertaking (JU) under grant agreement No 955558 and by MCIN/AEI/ 10.13039/501100011033 and the European Union NextGenerationEU/ PRTR (project eFlows4HPC, PCI2021-121957)., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Artículo científico (JournalArticle)

Revisiting active object stores: Bringing data locality to the limit with NVM

  • Barceló Cuerda, Alex|||0000-0002-0877-5738
  • Queralt Calafat, Anna|||0000-0003-2782-2955
  • Cortés, Toni|||0000-0002-2537-8937
Object stores are widely used software stacks that achieve excellent scale-out with a well-defined interface and robust performance. However, their traditional get/put interface is unable to exploit data locality at its fullest, and limits reaching its peak performance. In particular, there is one way to improve data locality that has not yet achieved mainstream adoption: the active object store. Although there are some projects that have implemented the main idea of the active object store such as Swift’s Storlets or Ceph Object Classes, the scope of these implementations is limited. We believe that there is a huge potential for active object stores in the current status quo. Hyper-converged nodes are bringing more computing capabilities to storage nodes –and viceversa. The proliferation of non-volatile memory (NVM) technology is blurring the line between system memory (fast and scarce) and block devices (slow and abundant). More and more applications need to manage a sheer amount of data (data analytics, Big Data, Machine Learning & AI, etc.), demanding bigger clusters and more complex computations. All these elements are potential game changers that need to be evaluated in the scope of active object stores. More specifically, having NVM devices presents additional opportunities, such as in-place execution. Being able to use the NVM from within the storage system while taking advantage of in-place execution (thanks to the byte-addressable nature of the NVM), in conjunction with the computing capabilities of hyper-converged nodes, can lead to active object stores that greatly outperform their non-active counterparts. In this article we propose an active object store software stack and evaluate it on an NVM-populated node. We will show how this setup is able to reduce execution times from 10% up to more than 90% in a variety of representative application scenarios. Our discussion will focus on the active aspect of the system as well as on the implications of the memory configuration., This work has been partially supported by the Spanish Government (PID2019-107255 GB), by Generalitat de Catalunya (contract 2014-SGR-1051), and by the European Commission through the Horizon 2020 Research and Innovation program and the EuroHPC JU under contract 955558 (eFlows4HPC project). Anna Queralt is a Serra Húnter Fellow. This work has been partially supported by Intel® under the BSC-Intel collaboration on 3D XPoint™ Technology., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Artículo científico (JournalArticle)

Extreme-scale workflows: a perspective from the JLESC international community

  • Yildiz, Orcun
  • Gueroudji, Amal
  • Bigot, Julien
  • Raffin, Bruno
  • Badia Sala, Rosa Maria|||0000-0003-2941-5499
  • Peterka, Tom
The Joint Laboratory for Extreme-Scale Computing (JLESC) focuses on software challenges in high-performance computing systems to meet the needs of today’s science campaigns, which often require large resources, consist of multiple tasks, and generate vast amounts of data. In this context, extreme-scale workflows have been the key factor in enabling scientific discoveries by helping scientists automate the dependencies and data exchanges between workflow tasks, instead of managing those manually. In this paper we present representative extreme-scale workflows and feature workflow systems developed by JLESC participating institutions. We present lessons learned while developing these tools, alongside with the open challenges and future research directions in the field of extreme-scale workflows., This research is supported by the U.S. Department of Energy, Office of Science, Office of Advanced Scientific Computing Research, under contract number DE-AC02- 06CH11357, program manager Margaret Lentz. The BSC author acknowledges the support of the Spanish Government (PID2019-107255GB) and by MCIN/AEI /10.13039 /501100011033 (CEX2021-001148-S), by the Departament de Recerca i Universitats de la Generalitat de Catalunya to the Research Group MPiEDist (2021 SGR 00412). The same author acknowledges the European Commission’s Horizon 2020 Framework program and the European High-Performance Computing Joint Undertaking (JU) under grant agreement No. 955558 and by MCIN/AEI/10.13039/501100011033 and the European Union NextGenerationEU/PRTR (PCI2021-121957), project eFlows4HPC. The INRIA author acknowledges the European Union’s Horizon research and innovation programme under grant agreement No 101144014., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Artículo científico (JournalArticle)

Enabling dynamic and intelligent workflows for HPC, data analytics, and AI convergence

  • Ejarque Artigas, Jorge
  • Badia Sala, Rosa Maria|||0000-0003-2941-5499
  • Becerra Fontal, Yolanda|||0000-0003-2357-7796
  • Rodrigo Berlín, Julián
  • Folch Duran, Arnau
  • Lordan Gomis, Francesc|||0000-0002-9845-8890
  • Monterrubio Velasco, Marisol|||0000-0003-0790-1832
  • de la Puente, Josep
  • Queralt Calafat, Anna|||0000-0003-2782-2955
  • Rodríguez Rodríguez, Juan E.
  • Rossi, Riccardo|||0000-0003-0528-7074
The evolution of High-Performance Computing (HPC) platforms enables the design and execution of progressively larger and more complex workflow applications in these systems. The complexity comes not only from the number of elements that compose the workflows but also from the type of computations they perform. While traditional HPC workflows target simulations and modelling of physical phenomena, current needs require in addition data analytics (DA) and artificial intelligence (AI) tasks. However, the development of these workflows is hampered by the lack of proper programming models and environments that support the integration of HPC, DA, and AI, as well as the lack of tools to easily deploy and execute the workflows in HPC systems. To progress in this direction, this paper presents use cases where complex workflows are required and investigates the main issues to be addressed for the HPC/DA/AI convergence. Based on this study, the paper identifies the challenges of a new workflow platform to manage complex workflows. Finally, it proposes a development approach for such a workflow platform addressing these challenges in two directions: first, by defining a software stack that provides the functionalities to manage these complex workflows; and second, by proposing the HPC Workflow as a Service (HPCWaaS) paradigm, which leverages the software stack to facilitate the reusability of complex workflows in federated HPC infrastructures. Proposals presented in this work are subject to study and development as part of the EuroHPC eFlows4HPC project., This work has received funding from the European HighPerformance Computing Joint Undertaking (JU) under grant agreement No 955558. The JU receives support from the European Union’s Horizon 2020 research and innovation programme and Spain, Germany, France, Italy, Poland, Switzerland and Norway. In Spain, it has received complementary funding from MCIN/AEI/10.13039/501100011033, Spain and the European Union NextGenerationEU/PRTR (contracts PCI2021-121957, PCI2021-121931, PCI2021-121944, and PCI2021-121927). In Germany, it has received complementary funding from the German Federal Ministry of Education and Research (contracts 16HPC016K, 6GPC016K, 16HPC017 and 16HPC018). In France, it has received financial support from Caisse des dépôts et consignations (CDC) under the action PIA ADEIP (project Calculateurs). In Italy, it has been preliminary approved for complimentary funding by Ministero dello Sviluppo Economico (MiSE) (ref. project prop. 2659). In Norway, it has received complementary funding from the Norwegian Research Council, Norway under project number 323825. In Switzerland, it has been preliminary approved for complimentary funding by the State Secretariat for Education, Research, and Innovation (SERI), Norway. In Poland, it is partially supported by the National Centre for Research and Development under decision DWM/EuroHPCJU/4/2021. The authors also acknowledge financial support by MCIN/AEI /10.13039/501100011033, Spain through the ‘‘Severo Ochoa Programme for Centres of Excellence in R&D’’ under Grant CEX2018-000797-S, the Spanish Government, Spain (contract PID2019-107255 GB) and by Generalitat de Catalunya, Spain (contract 2017-SGR-01414). Anna Queralt is a Serra Húnter Fellow., Peer Reviewed, Article signat per 37 autors/es: Jorge Ejarque (a), Rosa M. Badia (a), Loïc Albertin (f), Giovanni Aloisio (h), Enrico Baglione (k), Yolanda Becerra (a,c), Stefan Boschert (o) , Julian R. Berlin (a), Alessandro D’Anca (h), Donatello Elia (h), François Exertier (f), Sandro Fiore (i), José Flich (e), Arnau Folch (m,a), Steven J. Gibbons (p), Nikolay Koldunov (l), Francesc Lordan (a), Stefano Lorito (k), Finn Løvholt (p), Jorge Macías (j), Fabrizio Marozzo (g), Alberto Michelini (k), Marisol Monterrubio-Velasco (a), Marta Pienkowska (n), Josep de la Puente (a), Anna Queralt (c,a), Enrique S. Quintana-Ortí (e), Juan E. Rodríguez (a), Fabrizio Romano (k), Riccardo Rossi (b,c), Jedrzej Rybicki (d), Miroslaw Kupczyk (q), Jacopo Selva (k), Domenico Talia (g), Roberto Tonini (k), Paolo Trunfio (g), Manuela Volpe (k) // (a) Barcelona Supercomputing Center (BSC), (b) Centre Internacional de Mètodes Numèrics a l’Enginyeria (CIMNE), (c) Universitat Politècnica de Catalunya (UPC), (d) Jülich Supercomputing Centre (JSC), (e) Universitat Politécnica de València (UPV), (f) Atos BDS R&D HPC & AI Software, (g) DtoK Lab Srl, (h) Centro Euro-Mediterraneo sui Cambiamenti Climatici (CMCC), (i) Department of Information Engineering and Computer Science, University of Trento, (j) Universidad de Málaga (UMA), (k) Istituto Nazionale di Geofisica e Vulcanologia (INGV), (l) Alfred-Wegener-Institut Helmholtz-Zentrum für Polar- und Meeresforschung, (m) Consejo Superior Investigaciones Cientificas (CSIC), (n) Eidgenössische Technische Hochschule (ETH) Zürich, (o) Siemens AG, (p) Norwegian Geotechnical Institute (NGI), (q) Poznan Supercomputing and Networking Center (PSNC)
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

GPU cache system for COMPSs: A task-based distributed computing framework

  • Tatu, Cristian Catalin
  • Conejero Bañón, Francisco Javier
  • Vázquez Novoa, Fernando
  • Badia Sala, Rosa Maria|||0000-0003-2941-5499
In this paper, we propose a novel GPU cache system for COMPSs, a task-based distributed computing framework that enables the execution of parallel applications on heterogeneous clusters. GPU COMPSs tasks can exploit the computational power of NVIDIA GPUs to process large data blocks. However, the current implementation of COMPSs requires each task to write its output data to disk and the subsequent tasks to read them from disk, which introduces significant overhead. To overcome this limitation, we design and implement a GPU cache system that allows tasks to store and retrieve data from the GPU memory, avoiding unnecessary disk operations and reducing data transfer time. We conducted extensive experiments on several benchmarks and demonstrated that our GPU cache system can achieve significant speedups compared to the baseline COMPSs implementation., This work has been partially funded by the Spanish Government (PID2019-107255GB) y MCIN/AEI /10.13039/501100011033 (CEX2021-001148S), and by the Departament de Recerca i Universitats de la Generalitat de Catalunya, research group MPiEDist (2021 SGR 00412), and by the European Commission through the program Horizon 2020 and the European High-Performance Computing Joint Undertaking (JU) with contract 955558, project eFlows4HPC and by the MCIN/AEI/10.13039/501100011033 y European Union NextGenerationEU/PRTR (PCI2021-121957)., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Artículo científico (JournalArticle)

Automatizing the creation of specialized high-performance computing containers

  • Ejarque Artigas, Jorge
  • Badia Sala, Rosa Maria|||0000-0003-2941-5499
With Exascale computing already here, supercomputers are systems every time larger, more complex, and heterogeneous. While expert system administrators can install and deploy applications in the systems correctly, this is something that general users can not usually do. The eFlows4HPC project aims to provide methodologies and tools to enable the use and reuse of application workflows. One of the aspects that the project focuses on is simplifying the application deployment in large and complex systems. The approach uses containers, not generic ones, but containers tailored for each target High-Performance Computing (HPC) system. This paper presents the Container Image Creation service developed in the framework of the project and experimentation based on project applications. We compare the performance of the specialized containers against generic containers and against a native installation. The results show that in almost all cases, the specialized containers outperform the generic ones (up to 2× faster), and in all cases, the performance is the same as with the native installation., This work has been supported by the Spanish Government (PID2019-107255GB) and by MCIN/AEI /10.13039/501100011033 (CEX2021-001148-S), by Generalitat de Catalunya (contract 2021-SGR-00412), and by the European Commission’s Horizon 2020 Framework program and the European High-Performance Computing Joint Undertaking (JU) under grant agreement No 955558 and by MCIN/AEI/10.13039/501100011033 and the European Union NextGenerationEU/PRTR (project eFlows4HPC)., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Artículo científico (JournalArticle)

Block size estimation for data partitioning in HPC applications using machine learning techniques

  • Cantini, Riccardo
  • Marozzo, Fabrizio
  • Orsino, Alessio
  • Talia, Domenico
  • Trunfio, Paolo
  • Badia Sala, Rosa Maria|||0000-0003-2941-5499
  • Ejarque Artigas, Jorge
  • Vázquez-Novoa, Fernando
The extensive use of HPC infrastructures and frameworks for running data-intensive applications has led to a growing interest in data partitioning techniques and strategies. In fact, application performance can be heavily affected by how data are partitioned, which in turn depends on the selected size for data blocks, i.e. the block size. Therefore, finding an effective partitioning, i.e. a suitable block size, is a key strategy to speed-up parallel data-intensive applications and increase scalability. This paper describes a methodology, namely BLEST-ML (BLock size ESTimation through Machine Learning), for block size estimation that relies on supervised machine learning techniques. The proposed methodology was evaluated by designing an implementation tailored to dislib, a distributed computing library highly focused on machine learning algorithms built on top of the PyCOMPSs framework. We assessed the effectiveness of the provided implementation through an extensive experimental evaluation considering different algorithms from dislib, datasets, and infrastructures, including the MareNostrum 4 supercomputer. The results we obtained show the ability of BLEST-ML to efficiently determine a suitable way to split a given dataset, thus providing a proof of its applicability to enable the efficient execution of data-parallel applications in high performance environments., This work has been partially supported by the European Commission through the Horizon 2020 Research and Innovation program and the EuroHPC JU under contract 955558 (eFlows4HPC project) and by MCIN/AEI/10.13039/501100011033 and the European Union NextGenerationEU/PRTR (PCI2021‑121957 and CEX2021‑ 001148‑S) and by the Spanish Government (PID2019‑107255GB), and Generalitat de Catalunya (contract 2021‑SGR‑00412). We also acknowledge financial support from “National Centre for HPC, Big Data and Quantum Computing”, CN00000013 ‑ CUP H23C22000360005, and from “FAIR ‑ Future Artificial Intelligence Research” project ‑ CUP H23C22000860006., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Artículo científico (JournalArticle)

Parallel reduced-order modeling for digital twins using high-performance computing workflows

  • Ares de Parga Regalado, Sebastian|||0000-0001-5709-4683
  • Bravo Martínez, José Raúl|||0000-0002-4465-7536
  • Sibuet Ruiz, Nicolás|||0000-0003-2968-3653
  • Hernández Ortega, Joaquín Alberto|||0000-0001-9334-4002
  • Rossi, Riccardo|||0000-0003-0528-7074
  • Boschert, Stefan
  • Quintana Ortí, Enrique Salvador
  • Tomás Domínguez, Andrés Enrique
  • Tatu, Cristian Catalin
  • Vázquez Novoa, Fernando
  • Ejarque Artigas, Jorge
  • Badia Sala, Rosa Maria|||0000-0003-2941-5499
The integration of reduced-order models with high-performance computing is critical for developing digital twins, particularly for real-time monitoring and predictive maintenance of industrial systems. This paper presents a comprehensive, high-performance computing-enabled workflow for developing and deploying projection-based reduced-order models for large-scale mechanical simulations. We use PyCOMPSs’ parallel framework to efficiently execute reduced-order model training simulations, employing parallel singular value decomposition algorithms such as randomized singular value decomposition, Lanczos singular value decomposition, and full singular value decomposition based on tall-skinny QR. Moreover, we introduce a partitioned version of the hyperreduction scheme known as the Empirical Cubature Method to further enhance computational efficiency in projection-based reduced-order models for mechanical systems. Despite the widespread use of high-performance computing for projection-based reduced-order models, there is a significant lack of publications detailing comprehensive workflows for building and deploying end-to-end projection-based reduced-order models in high-performance computing environments. Our workflow is validated through a case study focusing on the thermal dynamics of a motor, a multiphysics problem involving convective heat transfer and mechanical components. The projection-based reduced-order model is designed to deliver a real-time prognosis tool that could enable rapid and safe motor restarts post-emergency shutdowns under different operating conditions, demonstrating its potential impact on the practice of simulations in engineering mechanics. To facilitate deployment, we use the High-Performance Computing Workflow as a Service strategy and Functional Mock-Up Units to ensure compatibility and ease of integration across high-performance computing, edge, and cloud environments. The outcomes illustrate the efficacy of combining projection-based reduced-order models and high-performance computing, establishing a precedent for scalable, real-time digital twin applications in computational mechanics across multiple industries., The authors acknowledge financial support from the Spanish Ministry of Economy and Competitiveness, through the “Severo Ochoa Programme for Centres of Excellence in R&D” (CEX2018-000797-S and CEX2021-001148-S)”, as well as to the project PID2019-107255GB (BSC authors). This project has received funding from the European High-Performance Computing Joint Undertaking (JU) under grant agreement No 955558. The JU receives support from the European Union’s Horizon 2020 research and innovation programme and Spain, Germany, France, Italy, Poland, Switzerland, Norway. This publication is part of the R&D project PCI2021-121944 and PCI2021-121957, financed by MCIN/AEI/10.13039/501100011033 and by the “European Union NextGenerationEU/PRTR”. Sebastian Ares de Parga Regalado acknowledges financial support from the Generalitat de Catalunya through the FI_SDUR-2021 grant (2021 FISDU 00142), which provided funding for his predoctoral training. BSC authors acknowledge financial support from the Departament de Recerca i Universitats de la Generalitat de Catalunya, research group MPiEDist (2021 SGR 00412). Author Fernando Vázquez is supported by PRE2022-104134 funded by MICIU/AEI /10.13039/501100011033 and by the FSE+., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Artículo científico (JournalArticle)

Portability and scalability evaluation of large-scale statistical modeling and prediction software through HPC-ready containers

  • Abdulah, Sameh
  • Ejarque Artigas, Jorge
  • Marzouk, Omar
  • Ltaief, Hatem
  • Sun, Ying
  • Genton, Marc G.
  • Badia Sala, Rosa Maria|||0000-0003-2941-5499
  • Keyes, David E.
HPC-based applications often have complex workflows with many software dependencies that hinder their portability on contemporary HPC architectures. In addition, these applications often require extraordinary efforts to deploy and execute at performance potential on new HPC systems, while the users expert in these applications generally have less expertise in HPC and related technologies. This paper provides a dynamic solution that facilitates containerization for transferring HPC software onto diverse parallel systems. The study relies on the HPC Workflow as a Service (HPCWaaS) paradigm proposed by the EuroHPC eFlows4HPC project. It offers to deploy workflows through containers tailored for any of a number of specific HPC systems. Traditional container image creation tools rely on OS system packages compiled for generic architecture families (x86_64, amd64, ppc64, …) and specific MPI or GPU runtime library versions. The containerization solution proposed in this paper leverages HPC Builders such as Spack or Easybuild and multi-platform builders such as buildx to create a service for automating the creation of container images for the software specific to each hardware architecture, aiming to sustain the overall performance of the software. We assess the efficiency of our proposed solution for porting the geostatistics ExaGeoStat software on various parallel systems while preserving the computational performance. The results show that the performance of the generated images is comparable with the native execution of the software on the same architectures. On the distributed-memory system, the containerized version can scale up to 256 nodes without impacting performance., This work has been supported by King Abdullah University of Science and Technology (KAUST), Saudi Arabia, the Spanish Government (contract PID2019-107255GB), by the Generalitat de Catalunya (contract 2017-SGR-01414), and by the European Commission’s Horizon 2020 Framework program and the European High-Performance Computing Joint Undertaking (JU) under grant agreement No 955558 and by MCIN/AEI/10.13039/501100011033 and the European Union NextGenerationEU/PRTR (project eFlows4HPC)., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Artículo científico (JournalArticle)

Model generation architecture for digital twins and deployment system

  • Vázquez Novoa, Fernando
  • Conejero Bañón, Francisco Javier
  • Badia Sala, Rosa Maria|||0000-0003-2941-5499
A Digital Twin is a virtual copy or representation of a physical object. Given the same input, both should produce the same output. A bidirectional data exchange between both of them should keep the objects in the same state by automatically reflecting changes from one of the objects to the other. Despite the popularity of the topic, there is a lack of standardization and no widespread adoption of techniques or architectures, resulting in ad-hoc implementations for each use case. This paper proposes a methodology for the generation of digital twins based on the definition of their life cycle, with phases that include training, operation and retraining. To allow efficient management of resources, each phase of the Digital Twin is implemented as a PyCOMPSs workflow. The proposed methodologies and architectures aims to simplify and strengthen the design and development of digital twins. It also allows for the extension of existing ones, the replacement of their components with more advanced ones, and the support of their reuse. We also consider workflow deployment, which is often a difficult task requiring specialized personnel and time. This article focuses on presenting the training phase of a Digital Twin life-cycle and demonstrates its operation by applying it to two real-world use case scenarios in ongoing projects. The use of the proposal led to a simplification of the developed code, allowed a distribution of the computational work among the available resources in a user-agnostic manner and the digital models created faithfully represent the physical object. Additionally, the paper presents the design of the operating life-cycle phase of a Digital Twin., Author Fernando Vázquez is supported by PRE2022-104134 funded by MICIU/AEI /10.13039/501100011033 and by the FSE+. This work has been partially funded by the the projects PID2019- 107255GB, CEX2021-001148-S, and PID2023-147979NB-C21 from the MCIN/AEI and MICIU/AEI /10.13039/501100011033 and by FEDER, UE, and by the Departament de Recerca i Universitats de la Generalitat de Catalunya, research group MPiEDist (2021 SGR 00412), and by the European Commission through the program Horizon 2020 and the European High-Performance Computing Joint Undertaking (JU) with contract 955558 and by the MCIN/AEI/10.13039/501100011033 y European Union NextGenerationEU/PRTR (PCI2021-121957), project eFlows4HPC; as well as by the European Commission under program Horizon Action under contract 101056886 (CAELESTIS). This work has also received funding from the European High Performance Computing Joint Undertaking (JU) and Spain, Italy, Iceland, Germany, Norway, France, Finland and Croatia under grant agreement No 101093038., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Artículo científico (JournalArticle)

A memory-efficient MultiVector Quasi-Newton method for black-box Fluid-Structure Interaction coupling

  • Zorrilla Martínez, Rubén|||0000-0001-8270-7170
  • Rossi, Riccardo|||0000-0003-0528-7074
In this work we present a novel Quasi-Newton technique for the black-box partitioned coupling of interface coupled problems. The new RandomiZed Multi-Vector Quasi-Newton method stems from the combination of the original Multi-Vector Quasi-Newton technique with the randomized Singular Value Decomposition algorithm, avoiding thus any dense DOFs-sized square matrix operation. This results in a reduction from quadratic to linear complexity in terms of the number of DOFs. Besides this, the need of storing the old inverse Jacobian is also avoided. Instead, only two very “thin” matrices are required to be saved, thus implying a much smaller memory footprint. Furthermore, our proposal can be used free of any user-defined parameter. The article describes the application of the method to the FSI interface residual equations in both Interface Quasi-Newton and Interface Block Quasi-Newton forms. For the latter, we also derive a closed form expression for the update, thus avoiding any linear system of equations resolution, by applying the Woodbury matrix identity to the inverse Jacobian decomposition matrices., This research is partly supported by the European High-Performance Computing Joint Undertaking (JU) through the project eFlows4HPC (grant agreement No 955558). The JU receives support from the European Union’s Horizon 2020 research and innovation program and Spain, Germany, France, Italy, Poland, Switzerland, Norway. The authors also acknowledge financial support from the Spanish Ministry of Economy and Competitiveness, through the “Severo Ochoa Programme for Centres of Excellence in R&D” (CEX2018-000797-S)., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

End-to-End Workflows for Climate Science: Integrating HPC Simulations, Big Data Processing, and Machine Learning

  • Elia, Donatello
  • Scardigno, Sonia
  • Ejarque, Jorge|||0000-0003-4725-5097
  • D'anca, Alessandro
  • Accarino, Gabriele
  • Scoccimarro, Enrico
  • Donno, Davide
  • Peano, Daniele
  • Immorlano, Francesco
  • Aloisio, Giovanni
Current scientific workflow systems do not typically integrate simulation-centric and data-centric aspects due to their very different software/infrastructure requirements. A transparent integration of such components into a single end-to-end workflow would lead to a more efficient and automated way for generating insights from large simulation data. This work presents a complex case study related to extreme events analysis of future climate data that integrates in the same workflow numerical simulations, Big Data analytics and Machine Learning models. The case study is being implemented in the context of the eFlows4HPC project using the project’s software stack for deployment and orchestration of the workflow. The solution implemented in the project has shown to simplify the development and execution of end-to-end climate workflows with heterogeneous software requirements. Moreover, such an approach can, in the long term, increase the reuse of workflows by scientists and their portability over different HPC infrastructures., Thiswork has received funding from the European High-Performance Computing Joint Undertaking (JU) under grant agreement No 955558. The JU receives support from the European Union’s Horizon 2020 research and innovation programme and Spain, Germany, France, Italy, Poland, Switzerland and Norway. In Spain, it has received complementary funding from MCIN/AEI/10.13039/501100011033, Spain and the European Union NextGenerationEU/PRTR (contracts PCI2021-121957, PCI2021-121931, PCI2021-121944, and PCI2021- 121927). In Italy, it has been preliminary approved for complimentary funding by Ministero dello Sviluppo Economico (MiSE) (ref. project prop. 2659). The authors also acknowledge financial support by MCIN/AEI /10.13039/501100011033, Spain through the "Severo Ochoa Programme for Centres of Excellence in R&D" under Grant CEX2021-001148-S, the Spanish Government (contract PID2019- 107255 GB) and by Generalitat de Catalunya (contract 2021-SGR-00412)., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Artículo científico (JournalArticle)

PyCOMPSs as an instrument for translational computer science

  • Badia Sala, Rosa Maria|||0000-0003-2941-5499
  • Conejero Bañón, Francisco Javier
  • Ejarque Artigas, Jorge
  • Lezzi, Daniele|||0000-0001-5081-7244
  • Lordan Gomis, Francesc|||0000-0002-9845-8890
With the advent of distributed computing, the need for frameworks that facilitate its programming and management has also appeared. These tools have typically been used to support the research on application areas that require them. This poses good initial conditions for translational computer science (TCS), although this does not always occur. This article describes our experience with the PyCOMPSs project, a programming model for distributed computing. While it is a research instrument for our team, it has also been applied in multiple real use cases under the umbrella of European Funded projects, or as part of internal projects between various departments at the Barcelona Supercomputing Center. This article illustrates how the authors have engaged in TCS as an underlying research methodology, collecting experiences from three European projects., This work was supported in part by Spanish Government under Contract TIN2015-65316-P, in part by the Generalitat de Catalunya under Contract 2014-SGR-1051, and in part by the European Commission’s Horizon 2020 Framework program through BioExcel Center of Excellence under Contract 823830 and Contract 675728, in part by the ExaQUte Project under Contract 800898, in part by the European High-Performance Computing Joint Undertaking (JU) under Grant 955558, in part by the MCIN/AEI/10.13039/501100011033, and in part by the European Union NextGenerationEU/PRTR., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Artículo científico (JournalArticle)

An enriched finite element/level-set model for two-phase electrohydrodynamic simulations

  • Narváez Muñoz, Christian|||0000-0002-3472-4037
  • Hashemi, Mohammad Reza|||0000-0002-6421-0993
  • Ryzhakov, Pavel|||0000-0002-4672-9038
  • Pons Prats, Jordi|||0000-0002-4930-9135
In this work, a numerical model for the simulation of two-phase electrohydrodynamic (EHD) problems is proposed. It is characterized by a physically consistent treatment of surface tension as well as a jump in the electric material properties. The formulation is based on a finite element method enriched with special shape functions, capable of accurate capturing discontinuities both in the fluid pressure and the gradient of the electric potential. Phase interface is, thus, represented as a zero-thickness boundary. The proposed methodology allows modeling the electric force as an interfacial one, strictly abiding with the physics. The approach is tested using the droplet deformation benchmarks. Moreover, application of the method to study a three-dimensional (3D) case, not characterized by symmetry of revolution, is shown. The proposed methodology defines a basis for an enriched finite element method for a wide range of EHD problems., The authors acknowledge the financial support of the Ministerio de Ciencia, Innovaci on e Universidades of Spain via the “Severo Ochoa Programme” for Centres of Excellence in R&D (Referece No. CEX2018-000797-S) given to the International Centre for Numerical Methods in Engineering (CIMNE). The work of C. Narvaez-Mu~noz was supported by the “Severo Ochoa Ph.D. Scholarship” Reference No. PRE2020-096632. Parts of this work were done in the framework of DIDRO project (Toward establishing a Digital twin for manufacturing via drop-on-demand inkjet printing. Proyectos Estrat egicos Orientados a la Transici on Ecol ogica y a la Transici on Digital. Reference No. TED2921-130471B-I00) supported by the Ministerio de Ciencia, Innovaci on e Universidades of Spain. M. Hashemi acknowledges the funding received from European Union’s Horizon 2020 Research and Innovation Programme (European High-Performance Computing Joint Undertaking Grant Agreement No. 955558) as part of EFLOWS4HPC project. P. Ryzhakov and J. Pons-Prats are Serra Hunter fellows., Peer Reviewed
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

PyCOMPSs as an instrument for translational computer science

  • Badia Sala, Rosa Maria
  • Conejero Bañón, Francisco Javier
  • Ejarque Artigas, Jorge
  • Lezzi, Daniele
  • Lordan Gomis, Francesc
With the advent of distributed computing, the need for frameworks that facilitate its programming and management has also appeared. These tools have typically been used to support the research on application areas that require them. This poses good initial conditions for translational computer science (TCS), although this does not always occur. This article describes our experience with the PyCOMPSs project, a programming model for distributed computing. While it is a research instrument for our team, it has also been applied in multiple real use cases under the umbrella of European Funded projects, or as part of internal projects between various departments at the Barcelona Supercomputing Center. This article illustrates how the authors have engaged in TCS as an underlying research methodology, collecting experiences from three European projects., This work was supported in part by Spanish Government under Contract TIN2015-65316-P, in part by the Generalitat de Catalunya under Contract 2014-SGR-1051, and in part by the European Commission’s Horizon 2020 Framework program through BioExcel Center of Excellence under Contract 823830 and Contract 675728, in part by the ExaQUte Project under Contract 800898, in part by the European High-Performance Computing Joint Undertaking (JU) under Grant 955558, in part by the MCIN/AEI/10.13039/501100011033, and in part by the European Union NextGenerationEU/PRTR., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Boosting HPC data analysis performance with the ParSoDA-Py library

  • Belcastro, Loris
  • Giampà, Salvatore
  • Marozzo, Fabrizio
  • Talia, Domenico
  • Trunfio, Paolo
  • Badia Sala, Rosa Maria
  • Ejarque, Jorge
  • Mammadli, Nihad
Developing and executing large-scale data analysis applications in parallel and distributed environments can be a complex and time-consuming task. Developers often find themselves diverted from their application logic to handle technical details about the underlying runtime and related issues. To simplify this process, ParSoDA, a Java library, has been proposed to facilitate the development of parallel data mining applications executed on HPC systems. It simplifies the process by providing built-in scalability mechanisms relying on the Hadoop and Spark frameworks. This paper presents ParSoDA-Py, the Python version of the ParSoDA library, which allows for further support of commonly used runtimes and libraries for big data analysis. After a complete library redesign, ParSoDA can be now easily integrated with other Python-based distributed runtimes for HPC systems, such as COMPSs and Apache Spark, and with the large ecosystem of Python-based data processing libraries. The paper discusses the adaptation process, which takes into consideration the new technical requirements, and evaluates both usability and scalability through some case study applications., This work has been partially funded by the European Commission’s Horizon 2020 Framework program and the European High-Performance Computing Joint Undertaking (JU) under Grant agreement No 955558 and by MCIN/AEI/10.13039/501100011033 and the European Union NextGenerationEU/PRTR (PCI2021-121957), project eFlows4HPC. It has also been supported by the Spanish Government (PID2019-107255GB) and by the Departament de Recerca i Universitats de la Generalitat de Catalunya to the Research Group MPiEDist (2021 SGR 00412) We also acknowledge financial support from “National Centre for HPC, Big Data and Quantum Computing," CN00000013 - CUP H23C22000360005, and from “FAIR - Future Artificial Intelligence Research" Project - CUP H23C22000860006., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Portability and scalability evaluation of large-scale statistical modeling and prediction software through HPC-ready containers

  • Abdulah, Sameh
  • Ejarque Artigas, Jorge
  • Marzouk, Omar
  • Ltaief, Hatem
  • Sun, Ying
  • Genton, Marc G.
  • Badia Sala, Rosa Maria
  • Keyes, David E.
HPC-based applications often have complex workflows with many software dependencies that hinder their portability on contemporary HPC architectures. In addition, these applications often require extraordinary efforts to deploy and execute at performance potential on new HPC systems, while the users expert in these applications generally have less expertise in HPC and related technologies. This paper provides a dynamic solution that facilitates containerization for transferring HPC software onto diverse parallel systems. The study relies on the HPC Workflow as a Service (HPCWaaS) paradigm proposed by the EuroHPC eFlows4HPC project. It offers to deploy workflows through containers tailored for any of a number of specific HPC systems. Traditional container image creation tools rely on OS system packages compiled for generic architecture families (x86_64, amd64, ppc64, …) and specific MPI or GPU runtime library versions. The containerization solution proposed in this paper leverages HPC Builders such as Spack or Easybuild and multi-platform builders such as buildx to create a service for automating the creation of container images for the software specific to each hardware architecture, aiming to sustain the overall performance of the software. We assess the efficiency of our proposed solution for porting the geostatistics ExaGeoStat software on various parallel systems while preserving the computational performance. The results show that the performance of the generated images is comparable with the native execution of the software on the same architectures. On the distributed-memory system, the containerized version can scale up to 256 nodes without impacting performance., This work has been supported by King Abdullah University of Science and Technology (KAUST), Saudi Arabia, the Spanish Government (contract PID2019-107255GB), by the Generalitat de Catalunya (contract 2017-SGR-01414), and by the European Commission’s Horizon 2020 Framework program and the European High-Performance Computing Joint Undertaking (JU) under grant agreement No 955558 and by MCIN/AEI/10.13039/501100011033 and the European Union NextGenerationEU/PRTR (project eFlows4HPC)., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

A memory-efficient MultiVector Quasi-Newton method for black-box Fluid-Structure Interaction coupling

  • Zorrilla Martínez, Rubén
  • Rossi, Riccardo
In this work we present a novel Quasi-Newton technique for the black-box partitioned coupling of interface coupled problems. The new RandomiZed Multi-Vector Quasi-Newton method stems from the combination of the original Multi-Vector Quasi-Newton technique with the randomized Singular Value Decomposition algorithm, avoiding thus any dense DOFs-sized square matrix operation. This results in a reduction from quadratic to linear complexity in terms of the number of DOFs. Besides this, the need of storing the old inverse Jacobian is also avoided. Instead, only two very “thin” matrices are required to be saved, thus implying a much smaller memory footprint. Furthermore, our proposal can be used free of any user-defined parameter. The article describes the application of the method to the FSI interface residual equations in both Interface Quasi-Newton and Interface Block Quasi-Newton forms. For the latter, we also derive a closed form expression for the update, thus avoiding any linear system of equations resolution, by applying the Woodbury matrix identity to the inverse Jacobian decomposition matrices., This research is partly supported by the European High-Performance Computing Joint Undertaking (JU) through the project eFlows4HPC (grant agreement No 955558). The JU receives support from the European Union’s Horizon 2020 research and innovation program and Spain, Germany, France, Italy, Poland, Switzerland, Norway. The authors also acknowledge financial support from the Spanish Ministry of Economy and Competitiveness, through the “Severo Ochoa Programme for Centres of Excellence in R&D” (CEX2018-000797-S)., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Model generation architecture for digital twins and deployment system

  • Vázquez Novoa, Fernando
  • Conejero Bañón, Francisco Javier
  • Badia Sala, Rosa Maria
A Digital Twin is a virtual copy or representation of a physical object. Given the same input, both should produce the same output. A bidirectional data exchange between both of them should keep the objects in the same state by automatically reflecting changes from one of the objects to the other. Despite the popularity of the topic, there is a lack of standardization and no widespread adoption of techniques or architectures, resulting in ad-hoc implementations for each use case. This paper proposes a methodology for the generation of digital twins based on the definition of their life cycle, with phases that include training, operation and retraining. To allow efficient management of resources, each phase of the Digital Twin is implemented as a PyCOMPSs workflow. The proposed methodologies and architectures aims to simplify and strengthen the design and development of digital twins. It also allows for the extension of existing ones, the replacement of their components with more advanced ones, and the support of their reuse. We also consider workflow deployment, which is often a difficult task requiring specialized personnel and time. This article focuses on presenting the training phase of a Digital Twin life-cycle and demonstrates its operation by applying it to two real-world use case scenarios in ongoing projects. The use of the proposal led to a simplification of the developed code, allowed a distribution of the computational work among the available resources in a user-agnostic manner and the digital models created faithfully represent the physical object. Additionally, the paper presents the design of the operating life-cycle phase of a Digital Twin., Author Fernando Vázquez is supported by PRE2022-104134 funded by MICIU/AEI /10.13039/501100011033 and by the FSE+. This work has been partially funded by the the projects PID2019- 107255GB, CEX2021-001148-S, and PID2023-147979NB-C21 from the MCIN/AEI and MICIU/AEI /10.13039/501100011033 and by FEDER, UE, and by the Departament de Recerca i Universitats de la Generalitat de Catalunya, research group MPiEDist (2021 SGR 00412), and by the European Commission through the program Horizon 2020 and the European High-Performance Computing Joint Undertaking (JU) with contract 955558 and by the MCIN/AEI/10.13039/501100011033 y European Union NextGenerationEU/PRTR (PCI2021-121957), project eFlows4HPC; as well as by the European Commission under program Horizon Action under contract 101056886 (CAELESTIS). This work has also received funding from the European High Performance Computing Joint Undertaking (JU) and Spain, Italy, Iceland, Germany, Norway, France, Finland and Croatia under grant agreement No 101093038., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

A kinematically stabilized linear tetrahedral finite element for compressible and nearly incompressible finite elasticity

  • Scovazzi, Guglielmo
  • Zorrilla Martínez, Rubén
  • Rossi, Riccardo
We propose a stabilized linear tetrahedral finite element method for static, finite elasticity problems involving compressible and nearly incompressible materials. Our approach relies on a mixed formulation, in which the nodal displacement unknown filed is complemented by a nodal Jacobian determinant unknown field. This approach is simple to implement in practical applications (e.g., in commercial software), since it only requires information already available when computing the Newton–Raphson tangent matrix associated with irreducible (i.e., displacement-based) finite element formulations. By nature, the proposed method is easily extensible to nonlinear models involving visco-plastic flow. An extensive suite of numerical tests in two and three dimensions is presented, to demonstrate the performance of the method., This research is partly supported by the European High-Performance Computing Joint Undertaking (JU) through the project eFlows4HPC (grant agreement number 955558). The JU receives support from the European Union Horizon 2020 research and innovation program and Spain, Germany, France, Italy, Poland, Switzerland, Norway. This publication is also part of the R&D project PCI2021-121944, financed by MCIN/AEI/10.13039/501100011033 and by the “European Union NextGenerationEU/PRTR”. The authors also acknowledge financial support from the Spanish Ministry of Economy and Competitiveness , through the “Severo Ochoa Programme for Centres of Excellence in R&D” (CEX2018-000797-S)., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Parallel reduced-order modeling for digital twins using high-performance computing workflows

  • Ares de Parga Regalado, Sebastian
  • Bravo Martínez, José Raúl
  • Sibuet Ruiz, Nicolás
  • Hernández Ortega, Joaquín Alberto
  • Rossi, Riccardo
  • Boschert, Stefan
  • Quintana Ortí, Enrique Salvador
  • Tomás Domínguez, Andrés Enrique
  • Tatu, Cristian Catalin
  • Vázquez Novoa, Fernando
  • Ejarque Artigas, Jorge
  • Badia Sala, Rosa Maria
The integration of reduced-order models with high-performance computing is critical for developing digital twins, particularly for real-time monitoring and predictive maintenance of industrial systems. This paper presents a comprehensive, high-performance computing-enabled workflow for developing and deploying projection-based reduced-order models for large-scale mechanical simulations. We use PyCOMPSs’ parallel framework to efficiently execute reduced-order model training simulations, employing parallel singular value decomposition algorithms such as randomized singular value decomposition, Lanczos singular value decomposition, and full singular value decomposition based on tall-skinny QR. Moreover, we introduce a partitioned version of the hyperreduction scheme known as the Empirical Cubature Method to further enhance computational efficiency in projection-based reduced-order models for mechanical systems. Despite the widespread use of high-performance computing for projection-based reduced-order models, there is a significant lack of publications detailing comprehensive workflows for building and deploying end-to-end projection-based reduced-order models in high-performance computing environments. Our workflow is validated through a case study focusing on the thermal dynamics of a motor, a multiphysics problem involving convective heat transfer and mechanical components. The projection-based reduced-order model is designed to deliver a real-time prognosis tool that could enable rapid and safe motor restarts post-emergency shutdowns under different operating conditions, demonstrating its potential impact on the practice of simulations in engineering mechanics. To facilitate deployment, we use the High-Performance Computing Workflow as a Service strategy and Functional Mock-Up Units to ensure compatibility and ease of integration across high-performance computing, edge, and cloud environments. The outcomes illustrate the efficacy of combining projection-based reduced-order models and high-performance computing, establishing a precedent for scalable, real-time digital twin applications in computational mechanics across multiple industries., The authors acknowledge financial support from the Spanish Ministry of Economy and Competitiveness, through the “Severo Ochoa Programme for Centres of Excellence in R&D” (CEX2018-000797-S and CEX2021-001148-S)”, as well as to the project PID2019-107255GB (BSC authors). This project has received funding from the European High-Performance Computing Joint Undertaking (JU) under grant agreement No 955558. The JU receives support from the European Union’s Horizon 2020 research and innovation programme and Spain, Germany, France, Italy, Poland, Switzerland, Norway. This publication is part of the R&D project PCI2021-121944 and PCI2021-121957, financed by MCIN/AEI/10.13039/501100011033 and by the “European Union NextGenerationEU/PRTR”. Sebastian Ares de Parga Regalado acknowledges financial support from the Generalitat de Catalunya through the FI_SDUR-2021 grant (2021 FISDU 00142), which provided funding for his predoctoral training. BSC authors acknowledge financial support from the Departament de Recerca i Universitats de la Generalitat de Catalunya, research group MPiEDist (2021 SGR 00412). Author Fernando Vázquez is supported by PRE2022-104134 funded by MICIU/AEI /10.13039/501100011033 and by the FSE+., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Revisiting active object stores: Bringing data locality to the limit with NVM

  • Barceló Cuerda, Alex
  • Queralt Calafat, Anna
  • Cortés, Toni
Object stores are widely used software stacks that achieve excellent scale-out with a well-defined interface and robust performance. However, their traditional get/put interface is unable to exploit data locality at its fullest, and limits reaching its peak performance. In particular, there is one way to improve data locality that has not yet achieved mainstream adoption: the active object store. Although there are some projects that have implemented the main idea of the active object store such as Swift’s Storlets or Ceph Object Classes, the scope of these implementations is limited. We believe that there is a huge potential for active object stores in the current status quo. Hyper-converged nodes are bringing more computing capabilities to storage nodes –and viceversa. The proliferation of non-volatile memory (NVM) technology is blurring the line between system memory (fast and scarce) and block devices (slow and abundant). More and more applications need to manage a sheer amount of data (data analytics, Big Data, Machine Learning & AI, etc.), demanding bigger clusters and more complex computations. All these elements are potential game changers that need to be evaluated in the scope of active object stores. More specifically, having NVM devices presents additional opportunities, such as in-place execution. Being able to use the NVM from within the storage system while taking advantage of in-place execution (thanks to the byte-addressable nature of the NVM), in conjunction with the computing capabilities of hyper-converged nodes, can lead to active object stores that greatly outperform their non-active counterparts. In this article we propose an active object store software stack and evaluate it on an NVM-populated node. We will show how this setup is able to reduce execution times from 10% up to more than 90% in a variety of representative application scenarios. Our discussion will focus on the active aspect of the system as well as on the implications of the memory configuration., This work has been partially supported by the Spanish Government (PID2019-107255 GB), by Generalitat de Catalunya (contract 2014-SGR-1051), and by the European Commission through the Horizon 2020 Research and Innovation program and the EuroHPC JU under contract 955558 (eFlows4HPC project). Anna Queralt is a Serra Húnter Fellow. This work has been partially supported by Intel® under the BSC-Intel collaboration on 3D XPoint™ Technology., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Managing workflow malleability in urgent computing for earthquake alerts

  • Ejarque Artigas, Jorge
  • Monterrubio Velasco, Marisol
  • Bhihe, Cedric
  • Pienkowska, Marta
  • de la Puente, Josep
  • Badia Sala, Rosa Maria
When large earthquakes happen, first responders need fast and accurate information regarding their impact. UCIS4EQ is an urgent computing platform that estimates ground shaking based on high-performance parallel 3D simulations. In this work, we present the PyCOMPSs implementation of UCIS4EQ towards urgent high-performance computing with a particular focus on providing a malleability mechanism. This allows UCIS4EQ to scan for data updates during runtime and dynamically incorporate said data into its execution. This involves, in particular, leveraging several HPC jobs. The implementation has been validated with data from the Samos Izmir earthquake. The malleable version of the workflow can reduce the use of computational resources by 40% while reducing the variability of the results and keeping the same time-to-solution as a non-malleable run., This work has been supported by the Spanish Government (PID2019-107255GB) and by MCIN/AEI/ 10.13039/501100011033 (CEX2021-001148-S), by Generalitat de Catalunya (contract 2021-SGR-00412), and by the European Commission’s Horizon 2020 Framework program and the European High-Performance Computing Joint Undertaking (JU) under grant agreement No 955558 and by MCIN/AEI/ 10.13039/501100011033 and the European Union NextGenerationEU/ PRTR (project eFlows4HPC, PCI2021-121957)., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

A machine learning estimator trained on synthetic data for real-time earthquake ground-shaking predictions in Southern California

  • Monterrubio Velasco, Marisol
  • Callaghan, Scott
  • Modesto Galende, David
  • Carrasco Jiménez, José Carlos
  • Badia Sala, Rosa Maria
  • Pallarés Font de Mora, Pablo
  • Vázquez Novoa, Fernando
  • Quintana Ortí, Enrique Salvador
  • Pienkowska, Marta
  • de la Puente, Josep
After large-magnitude earthquakes, a crucial task for impact assessment is to rapidly and accurately estimate the ground shaking in the affected region. To satisfy real-time constraints, intensity measures are traditionally evaluated with empirical Ground Motion Models that can drastically limit the accuracy of the estimated values. As an alternative, here we present Machine Learning strategies trained on physics-based simulations that require similar evaluation times. We trained and validated the proposed Machine Learning-based Estimator for ground shaking maps with one of the largest existing datasets (<100M simulated seismograms) from CyberShake developed by the Southern California Earthquake Center covering the Los Angeles basin. For a well-tailored synthetic database, our predictions outperform empirical Ground Motion Models provided that the events considered are compatible with the training data. Using the proposed strategy we show significant error reductions not only for synthetic, but also for five real historical earthquakes, relative to empirical Ground Motion Models., This work has been funded by the European Commission’s Horizon 2020 Framework program and the European High-Performance Computing Joint Undertaking (JU) under grant agreement No 955558 and by MCIN/AEI/ 10.13039/501100011033 and the European Union NextGeneration EU/PRTR(PCI2021-121957), project eFlows4HPC. This research has been supported by the European High-Performance Computing Joint Undertaking (JU) as well as Spain, Italy, Iceland, Germany, Norway, France, Finland, and Croatia under grant agreement no. 101093038, (ChEESE-CoE). The authors acknowledge the Center for Advanced Research Computing (CARC) at the University of Southern California for providing computing resources that have contributed to the research results reported within this publication. URL: https://carc.usc.edu. This research used resources of the Oak Ridge Leadership Computing Facility, which is a DOE Office of Science User Facility supported under Contract DE-AC05-00OR22725, and the National Center for Supercomputer Applications under the Blue Waters Sustained-Petascale Computing Project. SCEC is funded by USGS Cooperative Agreement G17AC00047 and NSF Cooperative Agreement EAR1600087., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

End-to-End Workflows for Climate Science: Integrating HPC Simulations, Big Data Processing, and Machine Learning

  • Elia, Donatello
  • Scardigno, Sonia
  • Ejarque, Jorge
  • D'anca, Alessandro
  • Accarino, Gabriele
  • Scoccimarro, Enrico
  • Donno, Davide
  • Peano, Daniele
  • Immorlano, Francesco
  • Aloisio, Giovanni
Current scientific workflow systems do not typically integrate simulation-centric and data-centric aspects due to their very different software/infrastructure requirements. A transparent integration of such components into a single end-to-end workflow would lead to a more efficient and automated way for generating insights from large simulation data. This work presents a complex case study related to extreme events analysis of future climate data that integrates in the same workflow numerical simulations, Big Data analytics and Machine Learning models. The case study is being implemented in the context of the eFlows4HPC project using the project’s software stack for deployment and orchestration of the workflow. The solution implemented in the project has shown to simplify the development and execution of end-to-end climate workflows with heterogeneous software requirements. Moreover, such an approach can, in the long term, increase the reuse of workflows by scientists and their portability over different HPC infrastructures., Thiswork has received funding from the European High-Performance Computing Joint Undertaking (JU) under grant agreement No 955558. The JU receives support from the European Union’s Horizon 2020 research and innovation programme and Spain, Germany, France, Italy, Poland, Switzerland and Norway. In Spain, it has received complementary funding from MCIN/AEI/10.13039/501100011033, Spain and the European Union NextGenerationEU/PRTR (contracts PCI2021-121957, PCI2021-121931, PCI2021-121944, and PCI2021- 121927). In Italy, it has been preliminary approved for complimentary funding by Ministero dello Sviluppo Economico (MiSE) (ref. project prop. 2659). The authors also acknowledge financial support by MCIN/AEI /10.13039/501100011033, Spain through the "Severo Ochoa Programme for Centres of Excellence in R&D" under Grant CEX2021-001148-S, the Spanish Government (contract PID2019- 107255 GB) and by Generalitat de Catalunya (contract 2021-SGR-00412)., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

GPU cache system for COMPSs: A task-based distributed computing framework

  • Tatu, Cristian Catalin
  • Conejero Bañón, Francisco Javier
  • Vázquez Novoa, Fernando
  • Badia Sala, Rosa Maria
In this paper, we propose a novel GPU cache system for COMPSs, a task-based distributed computing framework that enables the execution of parallel applications on heterogeneous clusters. GPU COMPSs tasks can exploit the computational power of NVIDIA GPUs to process large data blocks. However, the current implementation of COMPSs requires each task to write its output data to disk and the subsequent tasks to read them from disk, which introduces significant overhead. To overcome this limitation, we design and implement a GPU cache system that allows tasks to store and retrieve data from the GPU memory, avoiding unnecessary disk operations and reducing data transfer time. We conducted extensive experiments on several benchmarks and demonstrated that our GPU cache system can achieve significant speedups compared to the baseline COMPSs implementation., This work has been partially funded by the Spanish Government (PID2019-107255GB) y MCIN/AEI /10.13039/501100011033 (CEX2021-001148S), and by the Departament de Recerca i Universitats de la Generalitat de Catalunya, research group MPiEDist (2021 SGR 00412), and by the European Commission through the program Horizon 2020 and the European High-Performance Computing Joint Undertaking (JU) with contract 955558, project eFlows4HPC and by the MCIN/AEI/10.13039/501100011033 y European Union NextGenerationEU/PRTR (PCI2021-121957)., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Block size estimation for data partitioning in HPC applications using machine learning techniques

  • Cantini, Riccardo
  • Marozzo, Fabrizio
  • Orsino, Alessio
  • Talia, Domenico
  • Trunfio, Paolo
  • Badia Sala, Rosa Maria
  • Ejarque Artigas, Jorge
  • Vázquez-Novoa, Fernando
The extensive use of HPC infrastructures and frameworks for running data-intensive applications has led to a growing interest in data partitioning techniques and strategies. In fact, application performance can be heavily affected by how data are partitioned, which in turn depends on the selected size for data blocks, i.e. the block size. Therefore, finding an effective partitioning, i.e. a suitable block size, is a key strategy to speed-up parallel data-intensive applications and increase scalability. This paper describes a methodology, namely BLEST-ML (BLock size ESTimation through Machine Learning), for block size estimation that relies on supervised machine learning techniques. The proposed methodology was evaluated by designing an implementation tailored to dislib, a distributed computing library highly focused on machine learning algorithms built on top of the PyCOMPSs framework. We assessed the effectiveness of the provided implementation through an extensive experimental evaluation considering different algorithms from dislib, datasets, and infrastructures, including the MareNostrum 4 supercomputer. The results we obtained show the ability of BLEST-ML to efficiently determine a suitable way to split a given dataset, thus providing a proof of its applicability to enable the efficient execution of data-parallel applications in high performance environments., This work has been partially supported by the European Commission through the Horizon 2020 Research and Innovation program and the EuroHPC JU under contract 955558 (eFlows4HPC project) and by MCIN/AEI/10.13039/501100011033 and the European Union NextGenerationEU/PRTR (PCI2021‑121957 and CEX2021‑ 001148‑S) and by the Spanish Government (PID2019‑107255GB), and Generalitat de Catalunya (contract 2021‑SGR‑00412). We also acknowledge financial support from “National Centre for HPC, Big Data and Quantum Computing”, CN00000013 ‑ CUP H23C22000360005, and from “FAIR ‑ Future Artificial Intelligence Research” project ‑ CUP H23C22000860006., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Automatizing the creation of specialized high-performance computing containers

  • Ejarque Artigas, Jorge
  • Badia Sala, Rosa Maria
With Exascale computing already here, supercomputers are systems every time larger, more complex, and heterogeneous. While expert system administrators can install and deploy applications in the systems correctly, this is something that general users can not usually do. The eFlows4HPC project aims to provide methodologies and tools to enable the use and reuse of application workflows. One of the aspects that the project focuses on is simplifying the application deployment in large and complex systems. The approach uses containers, not generic ones, but containers tailored for each target High-Performance Computing (HPC) system. This paper presents the Container Image Creation service developed in the framework of the project and experimentation based on project applications. We compare the performance of the specialized containers against generic containers and against a native installation. The results show that in almost all cases, the specialized containers outperform the generic ones (up to 2× faster), and in all cases, the performance is the same as with the native installation., This work has been supported by the Spanish Government (PID2019-107255GB) and by MCIN/AEI /10.13039/501100011033 (CEX2021-001148-S), by Generalitat de Catalunya (contract 2021-SGR-00412), and by the European Commission’s Horizon 2020 Framework program and the European High-Performance Computing Joint Undertaking (JU) under grant agreement No 955558 and by MCIN/AEI/10.13039/501100011033 and the European Union NextGenerationEU/PRTR (project eFlows4HPC)., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Enhancing iteration performance on distributed task-based workflows

  • Barceló Cuerda, Alex
  • Queralt Calafat, Anna
  • Cortés, Toni
Task-based programming models have proven to be a robust and versatile way to approach development of applications for distributed environments. They provide natural programming patterns with high performance. However, execution on this paradigm can be very sensitive to granularity –i.e., the quantity and execution length of tasks. Granularity is often linked with the block size of the data, and finding the optimal block size has several challenges, as it requires inner knowledge of the computing environment. Our proposal is to supplement the task-based programming model with a new mechanism –our SplIter proposal. At its core, the SplIter provides a transparent way to split a collection into partitions (logical groups of blocks, obtained without any transfers nor data rearrangement), which can then be iterated. Tasks are linked to those partitions, which means that SplIter breaks the dependency between block size and task granularity. The evaluation shows that the SplIter is able to achieve performance improvements of over one order of magnitude when compared to the baseline, and it is either competitive or strictly better (depending on application characteristics) to the competitor alternative. We have chosen different applications covering a wide variety of scenarios; those applications are representatives of a broader set of applications and domains. The changes required in the source code of a task-based application are minimal, preserving the high programmability of the programming model. Two different state-of-the-art task-based frameworks have been evaluated for all the applications: COMPSs and Dask, showing that the SplIter can be effectively used within different frameworks., This work has been funded by the European Commission’s Horizon 2020 Framework program and the European High-Performance Computing Joint Undertaking (JU) under grant agreement No 955558 and by MCIN/AEI/ 10.13039/501100011033 and the European Union NextGenerationEU/PRTR (PCI2021-121957). Anna Queralt is a Serra Húnter Fellow., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Dynamic resource allocation for efficient parallel CFD simulations

  • Houzeaux, Guillaume
  • Badia Sala, Rosa Maria
  • Borrell Pol, Ricard
  • Dosimont, Damien
  • Ejarque Artigas, Jorge
  • Garcia Gasulla, Marta
  • López Herrero, Víctor
CFD users of supercomputers usually resort to rule-of-thumb methods to select the number of subdomains (partitions) when relying on MPI-based parallelization. One common approach is to set a minimum number of elements or cells per subdomain, under which the parallel efficiency of the code is “known” to fall below a subjective level, say 80%. The situation is even worse when the user is not aware of the “good” practices for the given code and a huge amount of resources can thus be wasted. This work presents an elastic computing methodology that adapts at runtime the resources allocated to a simulation automatically. The criterion to control the required resources is based on a runtime measure of the communication efficiency of the execution. According to some analytical estimates, the resources are then expanded or reduced to fulfill this criterion and eventually execute an efficient simulation., This work has been supported by the Spanish Government (PID2019- 107255GB); by Generalitat de Catalunya (contract 2014-SGR-1051); by the European Commission H2020 project POP CoE (GA n. 824080); by the European Commission H2020 project CompBioMed CoE (GA n. 823712) and by the European Commission and the EuroHPC JU under contract 955558 (eFlows4HPC project)., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

An enriched finite element/level-set model for two-phase electrohydrodynamic simulations

  • Narváez Muñoz, Christian
  • Hashemi, Mohammad Reza
  • Ryzhakov, Pavel
  • Pons Prats, Jordi
In this work, a numerical model for the simulation of two-phase electrohydrodynamic (EHD) problems is proposed. It is characterized by a physically consistent treatment of surface tension as well as a jump in the electric material properties. The formulation is based on a finite element method enriched with special shape functions, capable of accurate capturing discontinuities both in the fluid pressure and the gradient of the electric potential. Phase interface is, thus, represented as a zero-thickness boundary. The proposed methodology allows modeling the electric force as an interfacial one, strictly abiding with the physics. The approach is tested using the droplet deformation benchmarks. Moreover, application of the method to study a three-dimensional (3D) case, not characterized by symmetry of revolution, is shown. The proposed methodology defines a basis for an enriched finite element method for a wide range of EHD problems., The authors acknowledge the financial support of the Ministerio de Ciencia, Innovaci on e Universidades of Spain via the “Severo Ochoa Programme” for Centres of Excellence in R&D (Referece No. CEX2018-000797-S) given to the International Centre for Numerical Methods in Engineering (CIMNE). The work of C. Narvaez-Mu~noz was supported by the “Severo Ochoa Ph.D. Scholarship” Reference No. PRE2020-096632. Parts of this work were done in the framework of DIDRO project (Toward establishing a Digital twin for manufacturing via drop-on-demand inkjet printing. Proyectos Estrat egicos Orientados a la Transici on Ecol ogica y a la Transici on Digital. Reference No. TED2921-130471B-I00) supported by the Ministerio de Ciencia, Innovaci on e Universidades of Spain. M. Hashemi acknowledges the funding received from European Union’s Horizon 2020 Research and Innovation Programme (European High-Performance Computing Joint Undertaking Grant Agreement No. 955558) as part of EFLOWS4HPC project. P. Ryzhakov and J. Pons-Prats are Serra Hunter fellows., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Enabling dynamic and intelligent workflows for HPC, data analytics, and AI convergence

  • Ejarque Artigas, Jorge
  • Badia Sala, Rosa Maria
  • Becerra Fontal, Yolanda
  • Rodrigo Berlín, Julián
  • Folch Duran, Arnau
  • Lordan Gomis, Francesc
  • Monterrubio Velasco, Marisol
  • de la Puente, Josep
  • Queralt Calafat, Anna
  • Rodríguez Rodríguez, Juan E.
  • Rossi, Riccardo
The evolution of High-Performance Computing (HPC) platforms enables the design and execution of progressively larger and more complex workflow applications in these systems. The complexity comes not only from the number of elements that compose the workflows but also from the type of computations they perform. While traditional HPC workflows target simulations and modelling of physical phenomena, current needs require in addition data analytics (DA) and artificial intelligence (AI) tasks. However, the development of these workflows is hampered by the lack of proper programming models and environments that support the integration of HPC, DA, and AI, as well as the lack of tools to easily deploy and execute the workflows in HPC systems. To progress in this direction, this paper presents use cases where complex workflows are required and investigates the main issues to be addressed for the HPC/DA/AI convergence. Based on this study, the paper identifies the challenges of a new workflow platform to manage complex workflows. Finally, it proposes a development approach for such a workflow platform addressing these challenges in two directions: first, by defining a software stack that provides the functionalities to manage these complex workflows; and second, by proposing the HPC Workflow as a Service (HPCWaaS) paradigm, which leverages the software stack to facilitate the reusability of complex workflows in federated HPC infrastructures. Proposals presented in this work are subject to study and development as part of the EuroHPC eFlows4HPC project., This work has received funding from the European HighPerformance Computing Joint Undertaking (JU) under grant agreement No 955558. The JU receives support from the European Union’s Horizon 2020 research and innovation programme and Spain, Germany, France, Italy, Poland, Switzerland and Norway. In Spain, it has received complementary funding from MCIN/AEI/10.13039/501100011033, Spain and the European Union NextGenerationEU/PRTR (contracts PCI2021-121957, PCI2021-121931, PCI2021-121944, and PCI2021-121927). In Germany, it has received complementary funding from the German Federal Ministry of Education and Research (contracts 16HPC016K, 6GPC016K, 16HPC017 and 16HPC018). In France, it has received financial support from Caisse des dépôts et consignations (CDC) under the action PIA ADEIP (project Calculateurs). In Italy, it has been preliminary approved for complimentary funding by Ministero dello Sviluppo Economico (MiSE) (ref. project prop. 2659). In Norway, it has received complementary funding from the Norwegian Research Council, Norway under project number 323825. In Switzerland, it has been preliminary approved for complimentary funding by the State Secretariat for Education, Research, and Innovation (SERI), Norway. In Poland, it is partially supported by the National Centre for Research and Development under decision DWM/EuroHPCJU/4/2021. The authors also acknowledge financial support by MCIN/AEI /10.13039/501100011033, Spain through the ‘‘Severo Ochoa Programme for Centres of Excellence in R&D’’ under Grant CEX2018-000797-S, the Spanish Government, Spain (contract PID2019-107255 GB) and by Generalitat de Catalunya, Spain (contract 2017-SGR-01414). Anna Queralt is a Serra Húnter Fellow., Peer Reviewed, Article signat per 37 autors/es: Jorge Ejarque (a), Rosa M. Badia (a), Loïc Albertin (f), Giovanni Aloisio (h), Enrico Baglione (k), Yolanda Becerra (a,c), Stefan Boschert (o) , Julian R. Berlin (a), Alessandro D’Anca (h), Donatello Elia (h), François Exertier (f), Sandro Fiore (i), José Flich (e), Arnau Folch (m,a), Steven J. Gibbons (p), Nikolay Koldunov (l), Francesc Lordan (a), Stefano Lorito (k), Finn Løvholt (p), Jorge Macías (j), Fabrizio Marozzo (g), Alberto Michelini (k), Marisol Monterrubio-Velasco (a), Marta Pienkowska (n), Josep de la Puente (a), Anna Queralt (c,a), Enrique S. Quintana-Ortí (e), Juan E. Rodríguez (a), Fabrizio Romano (k), Riccardo Rossi (b,c), Jedrzej Rybicki (d), Miroslaw Kupczyk (q), Jacopo Selva (k), Domenico Talia (g), Roberto Tonini (k), Paolo Trunfio (g), Manuela Volpe (k) // (a) Barcelona Supercomputing Center (BSC), (b) Centre Internacional de Mètodes Numèrics a l’Enginyeria (CIMNE), (c) Universitat Politècnica de Catalunya (UPC), (d) Jülich Supercomputing Centre (JSC), (e) Universitat Politécnica de València (UPV), (f) Atos BDS R&D HPC & AI Software, (g) DtoK Lab Srl, (h) Centro Euro-Mediterraneo sui Cambiamenti Climatici (CMCC), (i) Department of Information Engineering and Computer Science, University of Trento, (j) Universidad de Málaga (UMA), (k) Istituto Nazionale di Geofisica e Vulcanologia (INGV), (l) Alfred-Wegener-Institut Helmholtz-Zentrum für Polar- und Meeresforschung, (m) Consejo Superior Investigaciones Cientificas (CSIC), (n) Eidgenössische Technische Hochschule (ETH) Zürich, (o) Siemens AG, (p) Norwegian Geotechnical Institute (NGI), (q) Poznan Supercomputing and Networking Center (PSNC), Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

Extreme-scale workflows: a perspective from the JLESC international community

  • Yildiz, Orcun
  • Gueroudji, Amal
  • Bigot, Julien
  • Raffin, Bruno
  • Badia Sala, Rosa Maria
  • Peterka, Tom
The Joint Laboratory for Extreme-Scale Computing (JLESC) focuses on software challenges in high-performance computing systems to meet the needs of today’s science campaigns, which often require large resources, consist of multiple tasks, and generate vast amounts of data. In this context, extreme-scale workflows have been the key factor in enabling scientific discoveries by helping scientists automate the dependencies and data exchanges between workflow tasks, instead of managing those manually. In this paper we present representative extreme-scale workflows and feature workflow systems developed by JLESC participating institutions. We present lessons learned while developing these tools, alongside with the open challenges and future research directions in the field of extreme-scale workflows., This research is supported by the U.S. Department of Energy, Office of Science, Office of Advanced Scientific Computing Research, under contract number DE-AC02- 06CH11357, program manager Margaret Lentz. The BSC author acknowledges the support of the Spanish Government (PID2019-107255GB) and by MCIN/AEI /10.13039 /501100011033 (CEX2021-001148-S), by the Departament de Recerca i Universitats de la Generalitat de Catalunya to the Research Group MPiEDist (2021 SGR 00412). The same author acknowledges the European Commission’s Horizon 2020 Framework program and the European High-Performance Computing Joint Undertaking (JU) under grant agreement No. 955558 and by MCIN/AEI/10.13039/501100011033 and the European Union NextGenerationEU/PRTR (PCI2021-121957), project eFlows4HPC. The INRIA author acknowledges the European Union’s Horizon research and innovation programme under grant agreement No 101144014., Peer Reviewed, Postprint (author's final draft)