Pasar al contenido principal

COMPUTACION CIENTIFICA SERVERLESS A TRAVES DEL HIBRIDO CONTINUO CLOUD

PID2020-113126RB-I00

•
Nombre agencia financiadora Agencia Estatal de Investigación
Acrónimo agencia financiadora AEI
Programa Programa Estatal de I+D+i Orientado a los Retos de la Sociedad
Convocatoria Proyectos de I+D+i (Generación de Conocimiento y Retos Investigación)
Año convocatoria 2020
Plan Estatal 2017-2020 Plan Estatal de Investigación Científica y Técnica y de Innovación
Centro beneficiario UNIVERSITAT POLITÈCNICA DE VALÈNCIA

Resultados relacionados

Resultados totales (Incluyendo duplicados): 14
Encontrada(s) 1 página(s)

RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publicación Artículo científico (JournalArticle)

Rescheduling serverless workloads across the cloud-to-edge continuum

  • Risco, Sebastián
  • Alarcón-Marín, Caterina
  • Langarita-Benítez, Sergio
  • Caballer Fernández, Miguel|||0000-0001-9393-3077
  • Moltó, Germán|||0000-0002-8049-253X
[EN] Serverless computing was a breakthrough in Cloud computing due to its high elasticity capabilities and fine-grained pay-per-use model offered by the main public Cloud providers. Meanwhile, open-source serverless platforms supporting the FaaS (Function as a Service) model allow users to take advantage of many of their benefits while operating on the on-premises platforms of organizations. This opens the possibility to deploy and exploit them on the different layers of the cloud-to-edge continuum, either on IoT (Internet of Things) devices located at the Edge (i.e. next to data acquisition devices), in on-premises clusters closer to the data sources (i.e. Fog computing) or directly on the Cloud. This paper presents two strategies to mitigate the overload that disparate data ingestion rates may cause in low-powered devices at the Edge or Fog layers. To this end, it is proposed to delegate and reschedule serverless jobs between the different layers of the cloud-to-edge continuum using an open-source platform for event-driven file processing. To demonstrate the performance of these strategies, a use case for fire detection is proposed that includes processing in the Fog via minified Kubernetes clusters located near the Edge, in the private Cloud via on-premises elastic clusters and, finally, in the public Cloud by using the AWS (Amazon Web Services) Lambda FaaS service. The results indicate that these strategies can mitigate overloads in use cases involving processing across the cloud-to-edge continuum by coordinating several layers of computing resources., Grant PID2020-113126RB-I00 funded by MCIN/AEI/10.13039/501100011033. Project PDC2021-120844-I00 funded by MCIN/AEI/10.13039/501100011033 and by the European Union NextGenerationEU/PRTR. This work was supported by the project AI-SPRINT "AI in Secure Privacy-Preserving Computing Continuum" that has received funding from the European Union's Horizon 2020 Research and Innovation Programme under Grant 101016577. This work was also supported by the project AI4EOSC "Artificial Intelligence for the European Open Science Cloud" that has received funding from the European Union's Horizon Europe Research and Innovation Programme under Grant 101058593.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publicación Artículo científico (JournalArticle)

Benchmark of the PenRed Monte Carlo framework for HDR brachytherapy

  • Oliver-Gil, Sandra|||0000-0001-8258-3972
  • Giménez-Alventosa, Vicent|||0000-0003-1646-6094
  • Berumen, F.
  • Giménez, V.
  • Beaulieu, L.
  • Ballester, F.
  • Vijande, J.
[EN] Purpose: The purpose of this study is to validate the PenRed Monte Carlo framework for clinical applications in brachytherapy. PenRed is a C++ version of Penelope Monte Carlo code with additional tallies and utilities. Methods and materials: Six benchmarking scenarios are explored to validate the use of PenRed and its improved bachytherapy-oriented capabilities for HDR brachytherapy. A new tally allowing the evaluation of collisional kerma for any material using the track length kerma estimator and the possibility to obtain the seed positions, weights and directions processing directly the DICOM file are now implemented in the PenRed distribution. The four non-clinical test cases developed by the Joint AAPM-ESTRO-ABG-ABS WG-DCAB were evaluated by comparing local and global absorbed dose differences with respect to established reference datasets. A prostate and a palliative lung cases, were also studied. For them, absorbed dose ratios, global absorbed dose differences, and cumulative dose-volume histograms were obtained and discussed. Results: The air-kerma strength and the dose rate constant corresponding to the two sources agree with the reference datatests within 0.3% (Sk) and 0.1% (¿). With respect to the first three WG-DCAB test cases, more than 99.8% of the voxels present local (global) differences within ±1%(±0.1%) of the reference datasets. For test Case 4 reference dataset, more than 94.9%(97.5%) of voxels show an agreement within ±1%(±0.1%), better than similar benchmarking calculations in the literature. The track length kerma estimator scorer implemented increases the numerical efficiency of brachytherapy calculations two orders of magnitude, while the specific brachytherapy source allows the user to avoid the use of error-prone intermediate steps to translate the DICOM information into the simulation. In both clinical cases, only minor absorbed dose differences arise in the low-dose isodoses. 99.8% and 100% of the voxels have a global absorbed dose difference ratio within ±0.2% for the prostate and lung cases, respectively. The role played by the different segmentation and composition material in the bone structures was discussed, obtaining negligible absorbed dose differences. Dose-volume histograms were in agreement with the reference data. Conclusions: PenRed incorporates new tallies and utilities and has been validated for its use for detailed and precise high-dose-rate brachytherapy simulations., This work is partially supported by the Ministerio de Ciencia e Innovación of Spain (MCIN) grants PID2020- 113126RB-I00 and PID2021-125096NB-I00 funded by MCIN/AEI/10.13039/501100011033. V. G. acknowledges partial support from AEI-MICINN under grant PID2020-1 13334GB-I00/AEI/10.13039/501100011033 and by Generalitat Valenciana through the project PROMETEO/2019/087. The work has also been supported by the Natural Sciences and Engineering Research Council of Canada (NSERC) grant RGPIN-2019-05038. Francisco Berumen acknowledges support by the Fonds de Recherche du Québec ¿ Nature et Technologies (FRQNT). J.V. and F.B. would like to acknowledge funding by MCIN/ AEI/10.13039 and the Generalitat Valenciana (GVA) grant PROMETEO/2021/064. Our colleague Prof. Vicent Giménez Gómez passed away on November 16, 2022, during the proofs of this study. We dedicate it to his memory. The Universitat de València lost a dedicated and accomplished physicist.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publicación Artículo científico (JournalArticle)

A serverless gateway for event-driven machine learning inference in multiple clouds

  • Naranjo-Delgado, Diana María
  • Risco, Sebastián
  • Moltó, Germán|||0000-0002-8049-253X
  • Blanquer Espert, Ignacio|||0000-0003-1692-8922
[EN] Serverless computing and, in particular, the functions as a service model has become a convincing paradigm for the development and implementation of highly scalable applications in the cloud. This is due to the transparent management of three key functionalities: triggering of functions due to events, automatic provisioning and scalability of resources, and fine-grained pay-per-use. This article presents a serverless web-based scientific gateway to execute the inference phase of previously trained machine learning and artificial intelligence models. The execution of the models is performed both in Amazon Web Services and in on-premises clouds with the OSCAR framework for serverless scientific computing. In both cases, the computing infrastructure grows elastically according to the demand adopting scale-to-zero approaches to minimize costs. The web interface provides an improved user experience by simplifying the use of the models. The usage of machine learning in a computing platform that can use both on-premises clouds and public clouds constitutes a step forward in the adoption of serverless computing for scientific applications., AI-SPRINT "Artificial Intelligence in Secure PRIvacy-preserving computing coN-Tinuum", Grant/Award Number: 101016577; DEEP-Hybrid-DataCloud "Designing and Enabling E-infrastructures for intensive Processing in a Hybrid DataCloud", Grant/Award Number: 777435; SERCLOCO, Grant/Award Numbers: MCIN/AEI/10.13039/501100011033, PID2020-113126RB-I00
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publicación Artículo científico (JournalArticle)

OSCAR-P and aMLLibrary: Profiling and predicting the performance of FaaS-based applications in computing continua

  • Sala, Roberto
  • Guindani, Bruno
  • Galimberti, Enrico
  • Filippini, Federica
  • Sedghani, Hamta
  • Ardagna, Danilo
  • Risco, Sebastián
  • Moltó, Germán|||0000-0002-8049-253X
  • Caballer Fernández, Miguel|||0000-0001-9393-3077
[EN] This paper proposes an automated framework for efficient application profiling and training of Machine Learning (ML) performance models, composed of two parts: OSCAR-P and aMLLibrary. OSCAR-P is an auto-profiling tool designed to automatically test serverless application workflows running on multiple hardware and node combinations in cloud and edge environments. OSCAR-P obtains relevant profiling information on the execution time of the individual application components. These data are later used by aMLLibrary to train ML-based performance models. This makes it possible to predict the performance of applications on unseen configurations. We test our framework on clusters with different architectures (x86 and arm64) and workloads, considering multi-component use-case applications. This extensive experimental campaign proves the efficiency of OSCAR-P and aMLLibrary, significantly reducing the time needed for the application profiling, data collection, and data processing. The preliminary results obtained on the ML performance models accuracy show a Mean Absolute Percentage Error lower than 30% in all the considered scenarios., The European Commission funded this work under the H2020 project AI-SPRINT (grant n. 101016577). OSCAR was developed under project PDC2021-120844-I00 funded by MCIN/AEI/-10.13039/5011000-11033 and by the European Union NextGenerationEU/PRTR and under grant PID2020-113126RB-I00 funded by MCIN/AEI/10.13039/501100011033. aMLLibrary was developed under the H2020 HPC JU LIGATE (grant n. 956137).
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publicación Artículo científico (JournalArticle)

Leveraging an open source serverless framework for high energy physics computing

  • Padulano, Vincenzo Eduardo
  • Oliver Cortés, Pablo
  • Tejedor Saavedra, Enric
  • Risco, Sebastián
  • Alonso-Jordá, Pedro|||0000-0002-6882-6592
  • Moltó, Germán|||0000-0002-8049-253X
[EN] CERN (Centre Europeen pour la Recherce Nucleaire) is the largest research centre for high energy physics (HEP). It ofers unique computational challenges as a result of the large amount of data generated by the large hadron collider. CERN has developed and supports a software called ROOT, which is the de facto standard for HEP data analysis. This framework ofers a high-level and easy-to-use interface called RDataFrame, which allows managing and processing large data sets. In recent years, its functionality has been extended to take advantage of distributed computing capabilities. Thanks to its declarative programming model, the user-facing API can be decoupled from the actual execution backend. This decoupling allows physical analysis to scale automatically to thousands of computational cores over various types of distributed resources. In fact, the distributed RDataFrame module already supports the use of established general industry engines such as Apache Spark or Dask. Notwithstanding the foregoing, these current solutions will not be sufcient to meet future requirements in terms of the amount of data that the new projected accelerators will generate. It is of interest, for this reason, to investigate a diferent approach, the one ofered by serverless computing. Based on a frst prototype using AWS Lambda, this work presents the creation of a new backend for RDataFrame distributed over the OSCAR tool, an open source framework that supports serverless computing. The implementation introduces new ways, relative to the AWS Lambdabased prototype, to synchronize the work of functions., Open Access funding provided thanks to the CRUE-CSIC agreement with Springer Nature. This work was supported by the research projects PID2020-113656RB-C22 (MCIN/AEI/10.13039/ 501100011033). Also, Grant PID2020-113126RB-I00 funded by MCIN/AEI/10.13039/501100011033 and Project PDC2021-120844-I00 funded by MCIN/AEI/10.13039/501100011033 funded by the European Union NextGenerationEU/PRTR all support the research on the OSCAR software tool.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publicación Artículo científico (JournalArticle)

Efficient and scalable covariate drift detection in machine learning systems with serverless computing

  • Céspedes Sisniega, Jaime
  • López García, Álvaro
  • Rodríguez-Benítez, Vicente|||0000-0002-7335-3849
  • Moltó, Germán|||0000-0002-8049-253X
[EN] As machine learning models are increasingly deployed in production, robust monitoring and detection of concept and covariate drift become critical. This paper addresses the gap in the widespread adoption of drift detection techniques by proposing a serverless-based approach for batch covariate drift detection in ML systems. Leveraging the open-source OSCAR framework and the open-source Frouros drift detection library, we develop a set of services that enable parallel execution of two key components: the ML inference pipeline and the batch covariate drift detection pipeline. To this end, our proposal takes advantage of the elasticity and efficiency of serverless computing for ML pipelines, including scalability, cost-effectiveness, and seamless integration with existing infrastructure. We evaluate this approach through an edge ML use case, showcasing its operation on a simulated batch covariate drift scenario. Our research highlights the importance of integrating drift detection as a fundamental requirement in developing robust and trustworthy AI systems and encourages the adoption of these techniques in ML deployment pipelines. In this way, organizations can proactively identify and mitigate the adverse effects of covariate drift while capitalizing on the benefits offered by serverless computing., This work was supported by the project AI4EOSC "Artificial Intelligence for the European Open Science Cloud" that has received funding from the European Union's Horizon Europe Research and Innovation Programme under Grant 101058593. JCS and ALG acknowledge the funding from the Agencia Estatal de Investigacion, Unidad de Excelencia Maria de Maeztu, ref. MDM-2017-0765. GM and VR would like to acknowledge Grant PID2020-113126RB-I00 funded by MICIU/AEI/10.13039/501100011033 and also project PDC2021-120844-I00 funded by MICIU/AEI/10.13039/501100011033 and by the European Union NextGenerationEU/PRTR.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publicación Artículo científico (JournalArticle)

CMK: Enhancing Resource Usage Monitoring across Diverse Bioinformatics Workflow Management Systems

  • Nica, Bogdan Robert
  • Götz, Stefan
  • Moltó, Germán|||0000-0002-8049-253X
[EN] The increasing use of multiple Workflow Management Systems (WMS) employing various workflow languages and shared workflow repositories enhances the open-source bioinformatics ecosystem. Efficient resource utilization in these systems is crucial for keeping costs low and improving processing times, especially for large-scale bioinformatics workflows running in cloud environments. Recognizing this, our study introduces a novel reference architecture, Cloud Monitoring Kit (CMK), for a multi-platform monitoring system. Our solution is designed to generate uniform, aggregated metrics from containerized workflow tasks scheduled by different WMS. Central to the proposed solution is the use of task labeling methods, which enable convenient grouping and aggregating of metrics independent of the WMS employed. This approach builds upon existing technology, providing additional benefits of modularity and capacity to seamlessly integrate with other data processing or collection systems. We have developed and released an open-source implementation of our system, which we evaluated on Amazon Web Services (AWS) using a transcriptomics data analysis workflow executed on two scientific WMS. The findings of this study indicate that CMK provides valuable insights into resource utilization. In doing so, it paves the way for more efficient management of resources in containerized scientific workflows running in public cloud environments, and it provides a foundation for optimizing task configurations, reducing costs, and enhancing scheduling decisions. Overall, our solution addresses the immediate needs of bioinformatics workflows and offers a scalable and adaptable framework for future advancements in cloud-based scientific computing., Open Access funding provided thanks to the CRUE-CSIC agreement with Springer Nature. This work has received funding from the Valencian Innovation Agency (AVI) file INNTA3/2021/5 (INNODOCTO). GM and RN would like to thank Grant PID2020-113126RB-I00 funded by MICIU/AEI/10.13039/501100011033. This work was supported by the project "An interdisciplinary Digital Twin Engine for science" (interTwin) that has received funding from the European Union's Horizon Europe Programme under Grant 101058386.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publicación Artículo científico (JournalArticle)

Infrastructure Manager: A TOSCA-Based Orchestrator for the Computing Continuum

  • Caballer Fernández, Miguel|||0000-0001-9393-3077
  • Moltó, Germán|||0000-0002-8049-253X
  • Calatrava Arroyo, Amanda|||0000-0002-9018-9171
  • Blanquer Espert, Ignacio|||0000-0003-1692-8922
[EN] The edge-to-cloud continuum involves heterogeneous computing resources, including low-power physical devices, Virtual Machines (VMs) in cloud management platforms and serverless computing services based on the FaaS (Functions as a Service) model. This requires novel strategies to describe and efficiently deploy complex applications that execute across the computing continuum. To this end, this paper introduces the developments in the Infrastructure Manager (IM), an open-source TOSCA-based orchestrator to provision and configure virtualized computing resources from a wide range of cloud platforms. By supplementing TOSCA with additional types, the IM can also provision from FaaS platforms across the computing continuum by leveraging public cloud services such as AWS Lambda and on-premises serverless platforms, such as OSCAR. This allows event-driven data-processing applications across multiple computing platforms and architectures. The evolution of the Infrastructure Manager is described to accommodate the definition in TOSCA of complex applications that span across the computing continuum and their automated provisioning and configuration using Infrastructure as Code (IaC) approaches. Its effectiveness is assessed through a real use case involving a machine-learning classifier application for assisting in the early diagnosis of Rheumatic Heart Disease (RHD). The results show that the new developments enable the IM to efficiently deploy complete application architectures described in TOSCA across the computing continuum, from VMs to FaaS services., Grant PID2020-113126RB-I00 funded by MCIN/AEI/10.13039/501100011033. Also, Project PDC2021- 120844-I00 funded by MCIN/AEI/10.13039/501100011033 and by the European Union NextGenerationEU/PRTR. This work was partially supported by the project AI-SPRINT AI in Secure Privacy-Preserving Computing Continuum which has received funding from the European Union s Horizon 2020 Research and Innovation Programme under Grant 101016577 and by the AI4EOSC Artificial Intelligence for the European Open Science Cloud that has received funding from the European Union s Horizon Europe Research and Innovation Programme under Grant 101058593. Finally, Amanda wants to thank the support received by the Universitat Politècnica de València for the project Extensión de modelos de computaci¿n en cloud en entornos confiables y altamente distribuidos , from the PAID-10-20 postdoctoral program.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publicación Artículo científico (JournalArticle)

TaScaaS: A Multi-Tenant Serverless Task Scheduler and Load Balancer as a Service

  • Giménez-Alventosa, Vicent|||0000-0003-1646-6094
  • Moltó, Germán|||0000-0002-8049-253X
  • Segrelles Quilis, José Damián|||0000-0001-5698-7965
[EN] A combination of distributed multi-tenant infrastructures, such as public Clouds and on-premises installations belonging to different organisations, are frequently used for scientific research because of the high computational requirements involved. Although resource sharing maximises their usage, it typically causes undesirable effects such as the noisy neighbour, producing unpredictable variations of the infrastructure computing capabilities. These fluctuations affect execution efficiency, even of loosely coupled applications, such as many Monte Carlo based simulation programs. This highlights the need of a service capable to handle workload distribution across multiple infrastructures to mitigate these unpredictable performance fluctuations. With this aim, this work introduces TaScaaS, a highly scalable and completely serverless service deployed on AWS to distribute loosely coupled jobs among several computing infrastructures, and load balance them using a completely asynchronous approach to cope with the performance fluctuations with minimum impact in the execution time. We demonstrate how TaScaaS is not only capable of handling these fluctuations efficiently, achieving reduction in execution times up to 45% in our experiments, but also split the jobs to be computed to meet the user-defined execution time., This work was supported in part by the Spanish "Ministerio de Ciencia e Innovación" for the project Serverless Scientific Computing Across the Hybrid Cloud Continuum (SERCLOCO) under Grant PID2020-113126RB-I00, in part by the program "Ayudas para lacontratación de personal investigador en formación de carácter predoctoral, programa VALi+d" from the Conselleria d'Educació of the Generalitat Valenciana, Spain, under Grant ACIF/2018/148, in part by the Fondo Social Europeo (FSE), in part by the project "AI in Secure Privacy-Preserving Computing Continuum (AI-SPRINT)" through the European Union's Horizon 2020 Research and Innovation Programme under Grant 101016577, in part by the European Regional Development Fund (ERDF) of the Comunitat Valenciana 2014¿2020, the regional government of the Comunitat Valenciana, Spain, (High-Performance Algorithms for the Modeling, Simulation and early Detection of diseases in Personalized Medicine), under Project IDIFEDER/2018/032, in part by the European Open Science Cloud - Hub (EOSC-Hub) Project under Grant 777536, and in part by the Helix Nebula Science Cloud (HNSciCloud) Project is also sponsoring the service, allowing users to access the HNSciCloud services pilot for limited scale usage using the voucher schemes provided by the two contractors: T-Systems and Exoscale, under Grant 687614
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

Benchmark of the PenRed Monte Carlo framework for HDR brachytherapy

  • Oliver, Sandra
  • Giménez Alventosa, Vicent
  • Berumen, Francisco
  • Giménez Gómez, Vicente
  • Beaulieu, Luc
  • Ballester, Facundo
  • Vijande, Javier
Purpose: The purpose of this study is to validate the PenRed Monte Carlo framework for clinical applications in brachytherapy. PenRed is a C++ version of Penelope Monte Carlo code with additional tallies and utilities. Methods and materials: Six benchmarking scenarios are explored to validate the use of PenRed and its improved bachytherapy-oriented capabilities for HDR brachytherapy. A new tally allowing the evaluation of collisional kerma for any material using the track length kerma estimator and the possibility to obtain the seed positions, weights and directions processing directly the DICOM file are now implemented in the PenRed distribution. The four non-clinical test cases developed by the Joint AAPM-ESTRO-ABG-ABS WG-DCAB were evaluated by comparing local and global absorbed dose differences with respect to established reference datasets. A prostate and a palliative lung cases, were also studied. For them, absorbed dose ratios, global absorbed dose differences, and cumulative dose-volume histograms were obtained and discussed. Results: The air-kerma strength and the dose rate constant corresponding to the two sources agree with the reference datatests within 0.3% (S) and 0.1% (Λ). With respect to the first three WG-DCAB test cases, more than 99.8% of the voxels present local (global) differences within ±1%(±0.1%) of the reference datasets. For test Case 4 reference dataset, more than 94.9%(97.5%) of voxels show an agreement within ±1%(±0.1%), better than similar benchmarking calculations in the literature. The track length kerma estimator scorer implemented increases the numerical efficiency of brachytherapy calculations two orders of magnitude, while the specific brachytherapy source allows the user to avoid the use of error-prone intermediate steps to translate the DICOM information into the simulation. In both clinical cases, only minor absorbed dose differences arise in the low-dose isodoses. 99.8% and 100% of the voxels have a global absorbed dose difference ratio within ±0.2% for the prostate and lung cases, respectively. The role played by the different segmentation and composition material in the bone structures was discussed, obtaining negligible absorbed dose differences. Dose-volume histograms were in agreement with the reference data. Conclusions: PenRed incorporates new tallies and utilities and has been validated for its use for detailed and precise high-dose-rate brachytherapy simulations., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

Efficient and scalable covariate drift detection in machine learning systems with serverless computing

  • Céspedes Sisniega, Jaime
  • Rodríguez Rodríguez, Vicente
  • Moltó, Germán
  • López-García, Álvaro
As machine learning models are increasingly deployed in production, robust monitoring and detection of concept and covariate drift become critical. This paper addresses the gap in the widespread adoption of drift detection techniques by proposing a serverless-based approach for batch covariate drift detection in ML systems. Leveraging the open-source OSCAR framework and the open-source Frouros drift detection library, we develop a set of services that enable parallel execution of two key components: the ML inference pipeline and the batch covariate drift detection pipeline. To this end, our proposal takes advantage of the elasticity and efficiency of serverless computing for ML pipelines, including scalability, cost-effectiveness, and seamless integration with existing infrastructure. We evaluate this approach through an edge ML use case, showcasing its operation on a simulated batch covariate drift scenario. Our research highlights the importance of integrating drift detection as a fundamental requirement in developing robust and trustworthy AI systems and encourages the adoption of these techniques in ML deployment pipelines. In this way, organizations can proactively identify and mitigate the adverse effects of covariate drift while capitalizing on the benefits offered by serverless computing., This work was supported by the project AI4EOSC “Artificial Intelligence for the European Open Science Cloud” that has received funding from the European Union’s Horizon Europe Research and Innovation Programme under Grant 101058593. JCS and ALG acknowledge the funding from the Agencia Estatal de Investigación, Unidad de Excelencia María de Maeztu, ref. MDM-2017-0765. GM and VR would like to acknowledge Grant PID2020-113126RB-I00 funded by MICIU/AEI/10.13039/501100011033 and also project PDC2021-120844-I00 funded by MICIU/AEI/10.13039/501100011033 and by the European Union NextGenerationEU/PRTR ., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

CMK: Enhancing Resource Usage Monitoring across Diverse Bioinformatics Workflow Management Systems

  • Nica, Robert
  • Götz, Stefan
  • Moltó, Germán
The increasing use of multiple Workflow Management Systems (WMS) employing various workflow languages and shared workflow repositories enhances the open-source bioinformatics ecosystem. Efficient resource utilization in these systems is crucial for keeping costs low and improving processing times, especially for large-scale bioinformatics workflows running in cloud environments. Recognizing this, our study introduces a novel reference architecture, Cloud Monitoring Kit (CMK), for a multi-platform monitoring system. Our solution is designed to generate uniform, aggregated metrics from containerized workflow tasks scheduled by different WMS. Central to the proposed solution is the use of task labeling methods, which enable convenient grouping and aggregating of metrics independent of the WMS employed. This approach builds upon existing technology, providing additional benefits of modularity and capacity to seamlessly integrate with other data processing or collection systems. We have developed and released an open-source implementation of our system, which we evaluated on Amazon Web Services (AWS) using a transcriptomics data analysis workflow executed on two scientific WMS. The findings of this study indicate that CMK provides valuable insights into resource utilization. In doing so, it paves the way for more efficient management of resources in containerized scientific workflows running in public cloud environments, and it provides a foundation for optimizing task configurations, reducing costs, and enhancing scheduling decisions. Overall, our solution addresses the immediate needs of bioinformatics workflows and offers a scalable and adaptable framework for future advancements in cloud-based scientific computing., Open Access funding provided thanks to the CRUE-CSIC agreement with Springer Nature. This work has received funding from the Valencian Innovation Agency (AVI) file INNTA3/2021/5 (INNODOCTO). GM and RN would like to thank Grant PID2020-113126RB-I00 funded by MICIU/AEI/10.13039/501100011033. This work was supported by the project “An interdisciplinary Digital Twin Engine for science” (interTwin) that has received funding from the European Union’s Horizon Europe Programme under Grant 101058386., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

Rescheduling serverless workloads across the cloud-to-edge continuum

  • Risco, Sebastián
  • Alarcón, Caterina
  • Langarita, Sergio
  • Caballer, Miguel
  • Moltó, Germán
Serverless computing was a breakthrough in Cloud computing due to its high elasticity capabilities and fine-grained pay-per-use model offered by the main public Cloud providers. Meanwhile, open-source serverless platforms supporting the FaaS (Function as a Service) model allow users to take advantage of many of their benefits while operating on the on-premises platforms of organizations. This opens the possibility to deploy and exploit them on the different layers of the cloud-to-edge continuum, either on IoT (Internet of Things) devices located at the Edge (i.e. next to data acquisition devices), in on-premises clusters closer to the data sources (i.e. Fog computing) or directly on the Cloud. This paper presents two strategies to mitigate the overload that disparate data ingestion rates may cause in low-powered devices at the Edge or Fog layers. To this end, it is proposed to delegate and reschedule serverless jobs between the different layers of the cloud-to-edge continuum using an open-source platform for event-driven file processing. To demonstrate the performance of these strategies, a use case for fire detection is proposed that includes processing in the Fog via minified Kubernetes clusters located near the Edge, in the private Cloud via on-premises elastic clusters and, finally, in the public Cloud by using the AWS (Amazon Web Services) Lambda FaaS service. The results indicate that these strategies can mitigate overloads in use cases involving processing across the cloud-to-edge continuum by coordinating several layers of computing resources., Grant PID2020-113126RB-I00 funded by MCIN/AEI/10.13039/501100011033. Project PDC2021-120844-I00 funded by MCIN/AEI/10.13039/501100011033 and by the European Union NextGenerationEU/PRTR. This work was supported by the project AI-SPRINT “AI in Secure Privacy-Preserving Computing Continuum” that has received funding from the European Union’s Horizon 2020 Research and Innovation Programme under Grant 101016577. This work was also supported by the project AI4EOSC “Artificial Intelligence for the European Open Science Cloud” that has received funding from the European Union’s Horizon Europe Research and Innovation Programme under Grant 101058593., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

OSCAR-P and aMLLibrary: Profiling and Predicting the Performance of FaaS-based Applications in Computing Continua

  • Sala, Roberto
  • Guindani, Bruno
  • Galimberti, Enrico
  • Filippini, Federica
  • Sedghani, Hamta
  • Ardagna, Danilo
  • Risco, Sebastián
  • Moltó, Germán
  • Caballer, Miguel
This paper proposes an automated framework for efficient application profiling and training of Machine Learning (ML) performance models, composed of two parts: OSCAR-P and aMLLibrary. OSCAR-P is an auto-profiling tool designed to automatically test serverless application workflows running on multiple hardware and node combinations in cloud and edge environments. OSCAR-P obtains relevant profiling information on the execution time of the individual application components. These data are later used by aMLLibrary to train ML-based performance models. This makes it possible to predict the performance of applications on unseen configurations. We test our framework on clusters with different architectures (x86 and arm64) and workloads, considering multi-component use-case applications. This extensive experimental campaign proves the efficiency of OSCAR-P and aMLLibrary, significantly reducing the time needed for the application profiling, data collection, and data processing. The preliminary results obtained on the ML performance models accuracy show a Mean Absolute Percentage Error lower than 30% in all the considered scenarios.The European Commission funded this work under the H2020 project AI-SPRINT (grant n. 101016577). OSCAR was developed under project PDC2021-120844-I00 funded by MCIN/AEI/-10.13039/5011000-11033 and by the European Union NextGenerationEU/PRTR and under grant PID2020-113126RB-I00 funded by MCIN/AEI /10.13039/501100011033. aMLLibrary was developed under the H2020 HPC JU LIGATE (grant n. 956137)., The European Commission funded this work under the H2020 project AI-SPRINT (grant n. 101016577). OSCAR was developed under project PDC2021-120844-I00 funded by MCIN/AEI/-10.13039/5011000-11033 and by the European Union NextGenerationEU/PRTR and under grant PID2020-113126RB-I00 funded by MCIN/AEI /10.13039/501100011033. aMLLibrary was developed under the H2020 HPC JU LIGATE (grant n. 956137)., Peer reviewed