Pasar al contenido principal

Resultados relacionados

Resultados totales (Incluyendo duplicados): 30
Encontrada(s) 1 página(s)

DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

The EGI Federated Cloud e-infrastructure

  • Fernández-del-Castillo, Enol
  • Scardaci, Diego
  • López-García, Álvaro
Under a Creative Commons license.-- Trabajo presentado a: 1st International Conference on Cloud Forward: From Distributed to Complete Computing., The EGI Federated Cloud is a standards-based open cloud system that offers a scalable and flexible e-infrastructure to the European research community, and extends the EGI computational offer beyond the traditional High Throughput Computing of the grid plat- form with new models like long-lived services and on demand computation. The EGI Cloud technology enables the federation of institutional clouds to run scientists workloads, simulations and services spanned across multiple administrative locations, allowing the users to access and exploit its resources as a unique system. The architecture of the federation was defined after a two year period of development based on a set of user stories describing operations on a cloud infrascturues and was officially launched in May 2014. Since then the EGI Federated Cloud operates as a federation of heterogeneous Infrastructure as a Service type clouds with every participating provider implementing the same set of interfaces towards users and system administrators. Enforcing cloud technology agnosticism and of supporting service mobility by means of open standards has also allowed for the inclusion of commercial cloud providers into an infrastructure previously supported only by academic institutions. This contributes to a wider goal of the funders to create economic and social impact from supported research activities. This paper details the state of the art, the design and development processes, and the organisational effort that have lead to the creation and deployment of the EGI Cloud Platform., The authors acknowledge the support of the EGI-Engage project [RI-654142] and the use of resources provided by the European Grid Infrastructure., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Otros (Other)

The EGI Federated Cloud e-Infrastructure

  • Fernández-del-Castillo, Enol
  • Scardaci, Diego
  • López-García, Álvaro
Trabajo presentado al EGI Community Forum, celebrado en Bari (Italia) del 10 al 13 de noviembre de 2015., This work by Parties of the EGI-Engage Consortium is licensed under a Creative Commons Attribution 4.0 International License., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Otros (Other)

Access to open research data for environmental science: the LifeWatch experience in biodiversity, Advancing data-driven research through the presented by Advancing data-driven research through the Data Commons Session at RDA P6 PARIS

  • Marco, Jesús
Trabajo presentado al 6th RDA (Research Data Alliance) Plenary meeting, celebrado en Paris del 23 al 25 de septiembre de 2015., Three solutions being developed in the EGI LifeWatch Competence Center are presented: TRUFA (an HPC NGS application), the modelling of a water reservoir, and Assisted Image Recognition to support Citizen Science., EGI-Engage is co-funded by the Horizon 2020 Framework Programme of the European Union under grant number 654142., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Otros (Other)

An integrated IaaS and PaaS architecture for scientific computing

  • Donvito, Giacinto
  • Blanquer, Ignacio
Trabajo presentado al EGI Community Forum, celebrado en Bari (Italia) del 10 al 13 de noviembre de 2015., Scientific applications often require multiple computing resources deployed on a coordinated way. The deployment of multiple resources require installing and configuring special software applications which should be updated when changes in the virtual infrastructure take place. When working on hybrid and federated cloud environments, restrictions on the hypervisor or cloud management platform must be minimised to facilitate geographic-wide brokering and cross-site deployments. Moreover, preserving the individual operation at the site-level in federated clouds is also important for scalability and interoperability. In that sense, the INDIGO-DataCloud project [1] has been designed with the objective of building up a PaaS-level cloud solution for research. One of the key multi-level components is the PaaS computing core. This part constitutes the kernel for the deployment of services and computing virtual infrastructures for the users. It is complemented with the virtualized storage, federated AAI and networking. The INDIGO-DataCloud PaaS core will be based on a microservice architecture [2]. Microservices consist of a set of narrowly focused, independently deployable services, typically implemented using container-embedded applications, exposed by RESTful interface. Microservices are designed to be highly scalable, highly available and targeted for the use in cloud environments. INDIGO's microservices will be deployed, dynamically scheduled and managed using tools such as kubernetes [3]. In cases where multi-tenancy is not yet intrinsically supported by the particular microservice, like the container manager, INDIGO-DataCloud may decide to offer multiple instances to bridge that gap. INDIGO PaaS will offer an upper layer orchestration service for distributed applications using the TOSCA language standard [4]. It will deal with the requested service instantiation and application execution, managing the needed microservices in order, for example, to select the right end-point for the deployment. Cross-site deployments will also be possible. This PaaS, aimed at providing a more efficient platform for scientific computing, will require additional characteristics from the underlying layers. The INDIGO PaaS will leverage an enhanced IaaS that will provide a richer set of features currently missing. The usage of TOSCA permits IaaS providers to offer infrastructure orchestration, making possible to manage the deployment and configuration of the resources that are being provided. The life-cycle of the resources is therefore managed through the APIs exposed by the IaaS end-points. The TOSCA templates will be translated into their native deployment schemas using IM [5] for OpenNebula and Heat-Translator [6] for OpenStack HEAT. Both OpenNebula and OpenStack will incorporate drivers to support the deployment of containers as first-class resources on the IaaS. This will provide high efficiency when building up complex configurations from a repository of container images. The scheduling algorithms for both cloud management frameworks will be improved, in order to provide a better experience for the end-users and a more efficient utilization of the computational resources. The usage of two-level orchestrator (at the level of PaaS and within each IaaS instances) will enhance the capabilities of providing a dynamic and on-demand increase in cloud resources, Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Otros (Other)

Digital knowledge platforms: a framework for Open Science

  • Marco, Jesús
  • Calderon, Alicia
  • Rodríguez Marrero, Ana Y.
  • Castro Manzano, P. de
  • Aguilar, Fernando
  • Cabrillo, I. J.
  • Fernández-del-Castillo, Enol
  • Cañas, Guadalupe
Trabajo presentado al EGI Community Forum, celebrado en Bari (Italia) del 10 al 13 de noviembre de 2015., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Otros (Other)

Opportunistic use of supercomputers: linking with Cloud and Grid platforms

  • Cabellos, Luis
  • Aguilar, Fernando
  • Marco, Jesús
  • Palacio, Aida
Trabajo presentado al EGI Community Forum, celebrado en Bari (Italia) del 10 al 13 de noviembre de 2015., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Otros (Other)

R computing services as SaaS in the cloud

  • Aguilar, Fernando
Trabajo presentado al EGI Community Forum, celebrado en Bari (Italia) del 10 al 13 de noviembre de 2015., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Otros (Other)

Scalable agent platforms with friendly interaction for modeling practical problems

  • Cabellos, Luis
  • Marco, Jesús
  • Rodríguez, Héctor
Trabajo presentado al EGI Community Forum, celebrado en Bari (Italia) del 10 al 13 de noviembre de 2015., Many problems can be addressed in a realistic way with the help of Agent Based Model tools. However, these tools are sometimes not easy to use for a final user, or are not able to scale up to use the computing resources required by the problem. We propose to develop a general platform supporting different ABM solutions, and deployed as a service in HPC Cloud resources. We analyze a first possible pilot, through the discussion of a simple but real use case: the anthropogenic impact in the water quality in a lake., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Otros (Other)

Setting up a new FedCloud site in collaboration with the industry

  • Aguilar, Fernando
Trabajo presentado al EGI Community Forum, celebrado en Bari (Italia) del 10 al 13 de noviembre de 2015., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Otros (Other)

Challenges in the EGI-LifeWatch competence center

  • Marco, Jesús
Trabajo presentado a la EGI Conference: "Engaging the Research Community towards an Open Science Commons", celebrada en Lisboa (Portugal) del 18 al 22 de mayo de 2015., EGI-Engage is co-funded by the Horizon 2020 Framework Programme of the European Union under grant number 654142., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Otros (Other)

EGI LifeWatch competence center support to LW community

  • Marco, Jesús
Trabajo presentado a la EGI Conference: "Engaging the Research Community towards an Open Science Commons", celebrada en Lisboa (Portugal) del 18 al 22 de mayo de 2015., Presentation of the EGI LW CC, including the following actions: -Big Data and Ecological Observatories: LW Marine; -Supporting Workflows & Virtual Labs in FedCloud; -Advanced Support to Citizen Science in Biodiversity., EGI-Engage is co-funded by the Horizon 2020 Framework Programme of the European Union under grant number 654142., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Otros (Other)

IFCA activities on business model

  • Marco, Jesús
Trabajo presentado al Spanish JRU EGI-ENGAGE meeting celebrado en Madrid el 23 de febrero de 2015., EGI-Engage is co-funded by the Horizon 2020 Framework Programme of the European Union under grant number 654142., No
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Otros (Other)

Presentación de EGI-ENGAGE, Engaging the EGI Community towards an Open Science Commons

  • Campos, Isabel
Trabajo presentado al Spanish JRU EGI-ENGAGE meeting celebrado en Madrid el 23 de febrero de 2015., EGI-InSPIRE RI-261323. EGI-Engage is co-funded by the Horizon 2020 Framework Programme of the European Union under grant number 654142., No
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Otros (Other)

FedCloud in EGI-ENGAGE

  • Fernández-del-Castillo, Enol
Trabajo presentado al Spanish JRU EGI-ENGAGE meeting celebrado en Madrid el 23 de febrero de 2015., EGI-InSPIRE RI-261323. EGI-Engage is co-funded by the Horizon 2020 Framework Programme of the European Union under grant number 654142., No
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Otros (Other)

IFCA-CSIC EGI-Engage JRA activities

  • López-García, Álvaro
Trabajo presentado al Spanish JRU EGI-ENGAGE meeting celebrado en Madrid el 23 de febrero de 2015., EGI-InSPIRE RI-261323. EGI-Engage is co-funded by the Horizon 2020 Framework Programme of the European Union under grant number 654142., No
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Otros (Other)

Lifewatch activity (IFCA-UPVLC-BIFI)

  • Marco, Jesús
Trabajo presentado al Spanish JRU EGI-ENGAGE meeting celebrado en Madrid el 23 de febrero de 2015., EGI-Engage is co-funded by the Horizon 2020 Framework Programme of the European Union under grant number 654142., No
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Otros (Other)

Towards a competence center supporting the LIFEWATCH user community: LIFEWATCH CC in EGI-ENGAGE

  • Marco, Jesús
Trabajo presentado al JRU-LW-ES meeting, celebrado en Málaga (España) el 25 de febreo de 2015., EGI-Engage is co-funded by the Horizon 2020 Framework Programme of the European Union under grant number 654142., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Otros (Other)

VLabs for biodiversity and ecosystem research

  • Marco, Jesús
Trabajo presentado a la EGI Conference: "Engaging the Research Community towards an Open Science Commons", celebrada en Lisboa (Portugal) del 18 al 22 de mayo de 2015., This presentation describes the scheme to be used in the LifeWatch EGI Competence Center to support Virtual Labs on Cloud IaaS and PaaS., EGI-Engage is co-funded by the Horizon 2020 Framework Programme of the European Union under grant number 654142., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

Standards for enabling heterogeneous IaaS cloud federations

  • López-García, Álvaro
  • Fernández-del-Castillo, Enol
  • Orviz, Pablo
arXiv:1711.08045v1, Technology market is continuing a rapid growth phase where different resource providers and Cloud Management Frameworks are positioning to provide ad-hoc solutions-as management interfaces, information discovery or billing-trying to differentiate from competitors resulting in incompatibilites between them when addressing more complex scenarios like federated clouds. Therefore, grasping interoperability problems present in current infrastructures by studying how existing and emerging standards could enhance the cloud user experience. In this paper we will review the current open challenges in Infrastructure as a Service cloud interoperability and federation, as well as point to the potential standards that should alleviate these problems., The authors want to acknowledge the support, in no particular order, of the EGI-Engage (grant number 654142) and INDIGO-Datacloud (grant number 653549) projects, funded by the European Commission's Horizon 2020 Framework Programme., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

Resource provisioning in Science Clouds: Requirements and challenges

  • López-García, Álvaro
  • Fernández-del-Castillo, Enol
  • Orviz, Pablo
  • Campos, Isabel
  • Marco, Jesús
arXiv:1709.08526v1, Cloud computing has permeated into the information technology industry in the last few years, and it is emerging nowadays in scientific environments. Science user communities are demanding a broad range of computing power to satisfy the needs of high-performance applications, such as local clusters, high-performance computing systems, and computing grids. Different workloads are needed from different computational models, and the cloud is already considered as a promising paradigm. The scheduling and allocation of resources is always a challenging matter in any form of computation and clouds are not an exception. Science applications have unique features that differentiate their workloads; hence, their requirements have to be taken into consideration to be fulfilled when building a Science Cloud. This paper will discuss what are the main scheduling and resource allocation challenges for any Infrastructure as a Service provider supporting scientific applications., The authors want to acknowledge the support of the EGI-Engage (grant 654142) and INDIGO-DataCloud (grant 653549)projects, funded by the European Commission's Horizon 2020 Framework Programme., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

umd-verification: Automation of Software Validation for the EGI Federated e-Infrastructure

  • Orviz, Pablo
  • Pina, Joâo
  • López-García, Álvaro
  • Campos, Isabel
  • David, Mario
  • Gomes, Jorge
Orviz, Pablo; Fernández-del-Castillo, Enol; Pina, Joâo; 2018; umd-verification; DIGITAL.CSIC; http://doi.org/, Supporting e-Science in the EGI e-Infrastructure requires extensive and reliable software, for advanced computing use, deployed across over approximately 300 European and worldwide data centers. The Unified Middleware Distribution (UMD) and Cloud Middleware Distribution (CMD) are the channels to deliver the software for the EGI e-Infrastructure consumption. The software is compiled, validated and distributed following the Software Provisioning Process (SWPP), where the Quality Criteria (QC) definition sets the minimum quality requirements for EGI acceptance. The growing number of software components currently existing within UMD and CMD distributions hinders the application of the traditional, manual-based validation mechanisms, thus driving the adoption of automated solutions. This paper presents umd-verification, an open-source tool that enforces the fulfillment of the QC requirements in an automated way for the continuous validation of the software products for scientific disposal. The umd-verification tool has been successfully integrated within the SWPP pipeline and is progressively supporting the full validation of the products in the UMD and CMD repositories. While the cost of supporting new products is dependant on the availability of Infrastructure as Code solutions to take over the deployment and high test coverage, the results obtained for the already integrated products are promising, as the time invested in the validation of products has been drastically reduced. Furthermore, automation adoption has brought along benefits for the reliability of the process, such as the removal of human-associated errors or the risk of regression of previously tested functionalities., This work has been partially funded by the EGI-Engage project (Engaging the Research Community towards an Open Science Commons) under grant agreement No. 654182., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

Using Scipion for stream image processing at Cryo-EM facilities

  • Gómez-Blanco, Josué
  • Rosa-Trevín, J. M. de la
  • Marabini, Roberto
  • Caño, Laura del
  • Jiménez, A.
  • Martínez, Marta
  • Melero, Roberto
  • Majtner, T.
  • Maluenda, D.
  • Mota, J.
  • Rancel, Y.
  • Ramírez-Aportela, Erney
  • Vilas, J. L.
  • Carroni, M.
  • Fleischmann, S.
  • Lindahl, E.
  • Ashton, A. W.
  • Basham, M.
  • Clare, D. K.
  • Savage, K.
  • Siebert, C. A.
  • Sharov, G. G.
  • Sorzano, Carlos Óscar S.
  • Conesa Mingo, Pablo
  • Carazo, José M.
Three dimensional electron microscopy is becoming a very data-intensive field in which vast amounts of experimental images are acquired at high speed. To manage such large-scale projects, we had previously developed a modular workflow system called Scipion (de la Rosa-Trevín et al., 2016). We present here a major extension of Scipion that allows processing of EM images while the data is being acquired. This approach helps to detect problems at early stages, saves computing time and provides users with a detailed evaluation of the data quality before the acquisition is finished. At present, Scipion has been deployed and is in production mode in seven Cryo-EM facilities throughout the world., Spanish Ministry of Economy and Competitiveness through Grants BIO2016-76400R(AEI/FEDER, UE) and AEI/FEDER BFU 2016 74868P, the Comunidad Autónoma de Madrid through Grant: S2017/BMD-3817, European Union (EU) and Horizon 2020 through grant Corbel (INFRADEV-1-2014-1, Proposal: 654248). The “Knut & Alice Wallenberg Foundation”, and “A Pilot Facility development grant from Science for Life Laboratory”. European Union (EU) and Horizon 2020 through grant EOSCpilot (INFRADEV-04-2016, Proposal: 739563).This work used the EGI Infrastructure and is co-funded by the EGIEngage project (Horizon 2020) under Grant No. 654142. European Union (EU) and Horizon 2020 through grant West-Life (EINFRA-2015-1, Proposal: 675858) European Union (EU) and Horizon 2020 through grant Elixir - EXCELERATE (INFRADEV-3-2015, Proposal: 676559) European Union (EU) and Horizon 2020 through grant iNEXT (INFRAIA-1-2014-2015, Proposal: 653706)., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Otros (Other)

umd-verification

  • Orviz, Pablo
  • Fernández-del-Castillo, Enol
  • Pina, Joâo
Apache License Version 2.0, January 2004., A tool based on Python's Fabric library that aims to automate the validation of all the EGI UMD and CMD software products, according to EGI's Quality Criteria specification., EGI-Engage GA 654142. EOSC-hub, GA 777536., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Otros (Other)

Openstack Keystone VOMS authentication module

  • López-García, Álvaro
  • Fernández-del-Castillo, Enol
  • Pansanel, Jerome
  • Huffman, Adam
  • Livenson, Ilja
  • Hagemeier, Björn
Apache License 2.0, This module is intended to provide VOMS authentication to a Mitaka OpenStack Keystone. It is designed to be integrated as an external authentication plugin, so that Keystone will preserve its original features and users will still be able to authenticate using any of the Keystone native mechanisms., EGI-Engage, GA 654142., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

ooi: OpenStack OCCI interface

  • López-García, Álvaro
  • Fernández-del-Castillo, Enol
  • Orviz, Pablo
In this document we present an implementation of the Open Grid Forum’s Open Cloud Computing Interface (OCCI) for OpenStack, namely ooi (Openstack occi interface, 2015). OCCI is an open standard for management tasks over cloud resources, focused on interoperability, portability and integration. ooi aims to implement this open interface for the OpenStack cloud middleware, promoting interoperability with other OCCI-enabled cloud management frameworks and infrastructures. ooi focuses on being non-invasive with a vanilla OpenStack installation, not tied to a particular OpenStack release version, The authors want to acknowledge the support of the EGI-Engage (grant number 654142) and INDIGO-Datacloud (grant number 653549) projects, funded by the European Commission’s Horizon 2020 Framework Programme., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Otros (Other)

ooi: OpenStack OCCI Interface

  • López-García, Álvaro
  • Fernández-del-Castillo, Enol
  • Orviz, Pablo
  • David, Mario
  • Sevilla Cedillo, Jorge
  • Pansanel, Jerome
  • Alves, Luis
Apache License 2.0, January 2004.-- OOI is an implementation the Open Cloud Computing Interface (OCCI) for OpenStack. http://openstack.org, ooi is an implementation the Open Grid Forum's Open Cloud Computing Interface (OCCI) for OpenStack that currently implements the version 1.2 (OCCI 1.2) of the standard., EGI-Engage, GA 654142. INDIGO-DataCloud GA 653549., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Otros (Other)

cASO: OpenStack Accounting extractor

  • López-García, Álvaro
  • Fernández-del-Castillo, Enol
Apache License 2.0, cASO is a pluggable extractor of Cloud Accounting Usage Records from an OpenStack installation. cASO gets usageinformation from nova or ceilometer APIs and can generate valid output for Apel SSM., EGI-Engage, GA 654142. EOSC-Hub, GA 777536., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

An efficient cloud scheduler design supporting preemptible instances

  • López-García, Álvaro
  • Fernández-del-Castillo, Enol
  • Campos, Isabel
Maximizing resource utilization by performing an efficient resource provisioning is a key factor for any cloud provider: commercial actors can maximize their revenues, whereas scientific and non-commercial providers can maximize their infrastructure utilization. Traditionally, batch systems have allowed data centers to fill their resources as much as possible by using backfilling and similar techniques. However, in an IaaS cloud, where virtual machines are supposed to live indefinitely, or at least as long as the user is able to pay for them, these policies are not easily implementable. In this work we present a new scheduling algorithm for IaaS providers that is able to support preemptible instances, that can be stopped by higher priority requests without introducing large modifications in the current cloud schedulers. This scheduler enables the implementation of new cloud usage and payment models that allow more efficient usage of the resources and potential new revenue sources for commercial providers. We also study the correctness and the performance overhead of the proposed scheduler against existing solutions., The authors acknowledge the financial support from the European Commission Horizon 2020 via INDIGO-DataCloud project (grant number 653549) and EGI-ENGAGE (grant number 654142) and the Ministry of Economy and Competitiveness for the support through the National Plan under contract number FPA2013-40715-P., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

Software quality assurance in INDIGO-DataCloud project: a converging evolution of software engineering practices to support European research e-infrastructures

  • Orviz, Pablo
  • David, Mario
  • Duma, Doina Cristina
  • Ronchieri, Elisabetta
  • Gomes, Jorge
  • Salomoni, Davide
From the advent of Grid technology - as the new paradigm of distributed computing - to the current days of Cloud computing models, the continuous need of new tools and services to match the scientific community requirements has been addressed in Europe through dedicated software development projects for e-Infrastructure creation, operation and management. This work presents the most signi cant software quality breakthroughs obtained in one of such projects, INDIGO-DataCloud, the main challenges and barriers confronted throughout the lifespan of the project, and how they were partially or totally overcome. The knowledge base established throughout the last 15 years of diverse software development initiatives in Europe for sustaining distributed research e-Infrastructures, supported by the advances in the area of software engineering, de nitely contributed to improve the quality and reliability of the software delivered, and consequently, the operational stability of the European e-Infrastructures. INDIGO-DataCloud project is a good evidence of such insights, where, unlike the preceding trend found in past projects, the enforcement of Software Quality Assurance practices has been present since the very early stages of the software lifecycle., DataGrid - Research and Technological Development for an International Data Grid project has received funding from the European Union's Fifth Framework Programme under grant agreement IST-2000-25182E. EGEE - Enabling grids For E-science project has received funding from the European Union's Sixth Framework Programme under grant agreement INFSO-RI-508833. EGEE - Enabling grids For E-science-II project has received funding from the European Union's Sixth Framework Programme under grant agreement INFSORI031688. EGEE - Enabling grids For E-science-III project has received funding from the European Union's Seventh Framework Programme under grant agreement INFSO-RI-222667. ETICS - E-Infrastructure for Testing, Integration and Configuration of Software project has received funding from the European Union's Sixth Framework Programme under grant agreement INFSO-RI-026753. ETICS - E-Infrastructure for Testing, Integration and Configuration of Software - Phase 2 project has received funding from the European Union's Seventh Framework Programme under grant agreement INFSO-RI-223782. EGI-InSPIRE - European Grid Initiative: Integrated Sustainable Pan-European Infrastructure for Researchers in Europe project has received funding from the European Union's Seventh Framework Programme under grant agreement INFSO-RI-261323. EGI-Engage project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement RIA-654142. INDIGO-DataCloud project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement RIA-653549., Peer reviewed
Dipòsit Digital de Documents de la UAB
Publicación Artículo científico (JournalArticle)

A Critical Path File Location (CPFL) algorithm for data-aware multiworkflow scheduling on HPC clusters

  • Acevedo Giménez, César Esteban
  • Hernández Budé, Porfidio|||0000-0002-8592-934X
  • Espinosa, Antonio|||0000-0002-6460-3789
  • Méndez Muñoz, Víctor|||0000-0002-9044-1189
A representative set of workflows found in bioinformatics pipelines must deal with large data sets. Most scientific workflows are defined as Direct Acyclic Graphs (DAGs). Despite DAGs are useful to understand dependence relationships, they do not provide any information about input, output and temporal data files. This information about the location of files of data intensive applications helps to avoid performance issues. This paper presents a multiworkflow store-aware scheduler in a cluster environment called Critical Path File Location (CPFL) policy where the access time to disk is more relevant than network, as an extension of the classical list scheduling policies. Our purpose is to find the best location of data files in a hierarchical storage system. The resulting algorithm is tested in an HPC cluster and in a simulated cluster scenario with bioinformatics synthetic workflows, and largely used benchmarks like Montage and Epigenomics. The resulting simulator is tuned and validated with the first test results from the real infrastructure. The evaluation of our proposal shows promising results up to 70% on benchmarks in real HPC clusters using 128 cores and up to 69% of makespan improvement on simulated 512 cores clusters with a deviation between 0.9% and 3% regarding the real HPC cluster.