Skip to main content

Resultados relacionados

Resultados totales (Incluyendo duplicados): 9
Encontrada(s) 1 página(s)

DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Artículo científico (JournalArticle)

The EGI Federated Cloud e-infrastructure

  • Fernández-del-Castillo, Enol
  • Scardaci, Diego
  • López-García, Álvaro
Under a Creative Commons license.-- Trabajo presentado a: 1st International Conference on Cloud Forward: From Distributed to Complete Computing., The EGI Federated Cloud is a standards-based open cloud system that offers a scalable and flexible e-infrastructure to the European research community, and extends the EGI computational offer beyond the traditional High Throughput Computing of the grid plat- form with new models like long-lived services and on demand computation. The EGI Cloud technology enables the federation of institutional clouds to run scientists workloads, simulations and services spanned across multiple administrative locations, allowing the users to access and exploit its resources as a unique system. The architecture of the federation was defined after a two year period of development based on a set of user stories describing operations on a cloud infrascturues and was officially launched in May 2014. Since then the EGI Federated Cloud operates as a federation of heterogeneous Infrastructure as a Service type clouds with every participating provider implementing the same set of interfaces towards users and system administrators. Enforcing cloud technology agnosticism and of supporting service mobility by means of open standards has also allowed for the inclusion of commercial cloud providers into an infrastructure previously supported only by academic institutions. This contributes to a wider goal of the funders to create economic and social impact from supported research activities. This paper details the state of the art, the design and development processes, and the organisational effort that have lead to the creation and deployment of the EGI Cloud Platform., The authors acknowledge the support of the EGI-Engage project [RI-654142] and the use of resources provided by the European Grid Infrastructure., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Artículo científico (JournalArticle)

Standards for enabling heterogeneous IaaS cloud federations

  • López-García, Álvaro
  • Fernández-del-Castillo, Enol
  • Orviz, Pablo
arXiv:1711.08045v1, Technology market is continuing a rapid growth phase where different resource providers and Cloud Management Frameworks are positioning to provide ad-hoc solutions-as management interfaces, information discovery or billing-trying to differentiate from competitors resulting in incompatibilites between them when addressing more complex scenarios like federated clouds. Therefore, grasping interoperability problems present in current infrastructures by studying how existing and emerging standards could enhance the cloud user experience. In this paper we will review the current open challenges in Infrastructure as a Service cloud interoperability and federation, as well as point to the potential standards that should alleviate these problems., The authors want to acknowledge the support, in no particular order, of the EGI-Engage (grant number 654142) and INDIGO-Datacloud (grant number 653549) projects, funded by the European Commission's Horizon 2020 Framework Programme., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Artículo científico (JournalArticle)

Resource provisioning in Science Clouds: Requirements and challenges

  • López-García, Álvaro
  • Fernández-del-Castillo, Enol
  • Orviz, Pablo
  • Campos, Isabel
  • Marco, Jesús
arXiv:1709.08526v1, Cloud computing has permeated into the information technology industry in the last few years, and it is emerging nowadays in scientific environments. Science user communities are demanding a broad range of computing power to satisfy the needs of high-performance applications, such as local clusters, high-performance computing systems, and computing grids. Different workloads are needed from different computational models, and the cloud is already considered as a promising paradigm. The scheduling and allocation of resources is always a challenging matter in any form of computation and clouds are not an exception. Science applications have unique features that differentiate their workloads; hence, their requirements have to be taken into consideration to be fulfilled when building a Science Cloud. This paper will discuss what are the main scheduling and resource allocation challenges for any Infrastructure as a Service provider supporting scientific applications., The authors want to acknowledge the support of the EGI-Engage (grant 654142) and INDIGO-DataCloud (grant 653549)projects, funded by the European Commission's Horizon 2020 Framework Programme., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Artículo científico (JournalArticle)

umd-verification: Automation of Software Validation for the EGI Federated e-Infrastructure

  • Orviz, Pablo
  • Pina, Joâo
  • López-García, Álvaro
  • Campos, Isabel
  • David, Mario
  • Gomes, Jorge
Orviz, Pablo; Fernández-del-Castillo, Enol; Pina, Joâo; 2018; umd-verification; DIGITAL.CSIC; http://doi.org/, Supporting e-Science in the EGI e-Infrastructure requires extensive and reliable software, for advanced computing use, deployed across over approximately 300 European and worldwide data centers. The Unified Middleware Distribution (UMD) and Cloud Middleware Distribution (CMD) are the channels to deliver the software for the EGI e-Infrastructure consumption. The software is compiled, validated and distributed following the Software Provisioning Process (SWPP), where the Quality Criteria (QC) definition sets the minimum quality requirements for EGI acceptance. The growing number of software components currently existing within UMD and CMD distributions hinders the application of the traditional, manual-based validation mechanisms, thus driving the adoption of automated solutions. This paper presents umd-verification, an open-source tool that enforces the fulfillment of the QC requirements in an automated way for the continuous validation of the software products for scientific disposal. The umd-verification tool has been successfully integrated within the SWPP pipeline and is progressively supporting the full validation of the products in the UMD and CMD repositories. While the cost of supporting new products is dependant on the availability of Infrastructure as Code solutions to take over the deployment and high test coverage, the results obtained for the already integrated products are promising, as the time invested in the validation of products has been drastically reduced. Furthermore, automation adoption has brought along benefits for the reliability of the process, such as the removal of human-associated errors or the risk of regression of previously tested functionalities., This work has been partially funded by the EGI-Engage project (Engaging the Research Community towards an Open Science Commons) under grant agreement No. 654182., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Artículo científico (JournalArticle)

Using Scipion for stream image processing at Cryo-EM facilities

  • Gómez-Blanco, Josué
  • Rosa-Trevín, J. M. de la
  • Marabini, Roberto
  • Caño, Laura del
  • Jiménez, A.
  • Martínez, Marta
  • Melero, Roberto
  • Majtner, T.
  • Maluenda, D.
  • Mota, J.
  • Rancel, Y.
  • Ramírez-Aportela, Erney
  • Vilas, J. L.
  • Carroni, M.
  • Fleischmann, S.
  • Lindahl, E.
  • Ashton, A. W.
  • Basham, M.
  • Clare, D. K.
  • Savage, K.
  • Siebert, C. A.
  • Sharov, G. G.
  • Sorzano, Carlos Óscar S.
  • Conesa Mingo, Pablo
  • Carazo, José M.
Three dimensional electron microscopy is becoming a very data-intensive field in which vast amounts of experimental images are acquired at high speed. To manage such large-scale projects, we had previously developed a modular workflow system called Scipion (de la Rosa-Trevín et al., 2016). We present here a major extension of Scipion that allows processing of EM images while the data is being acquired. This approach helps to detect problems at early stages, saves computing time and provides users with a detailed evaluation of the data quality before the acquisition is finished. At present, Scipion has been deployed and is in production mode in seven Cryo-EM facilities throughout the world., Spanish Ministry of Economy and Competitiveness through Grants BIO2016-76400R(AEI/FEDER, UE) and AEI/FEDER BFU 2016 74868P, the Comunidad Autónoma de Madrid through Grant: S2017/BMD-3817, European Union (EU) and Horizon 2020 through grant Corbel (INFRADEV-1-2014-1, Proposal: 654248). The “Knut & Alice Wallenberg Foundation”, and “A Pilot Facility development grant from Science for Life Laboratory”. European Union (EU) and Horizon 2020 through grant EOSCpilot (INFRADEV-04-2016, Proposal: 739563).This work used the EGI Infrastructure and is co-funded by the EGIEngage project (Horizon 2020) under Grant No. 654142. European Union (EU) and Horizon 2020 through grant West-Life (EINFRA-2015-1, Proposal: 675858) European Union (EU) and Horizon 2020 through grant Elixir - EXCELERATE (INFRADEV-3-2015, Proposal: 676559) European Union (EU) and Horizon 2020 through grant iNEXT (INFRAIA-1-2014-2015, Proposal: 653706)., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Artículo científico (JournalArticle)

ooi: OpenStack OCCI interface

  • López-García, Álvaro
  • Fernández-del-Castillo, Enol
  • Orviz, Pablo
In this document we present an implementation of the Open Grid Forum’s Open Cloud Computing Interface (OCCI) for OpenStack, namely ooi (Openstack occi interface, 2015). OCCI is an open standard for management tasks over cloud resources, focused on interoperability, portability and integration. ooi aims to implement this open interface for the OpenStack cloud middleware, promoting interoperability with other OCCI-enabled cloud management frameworks and infrastructures. ooi focuses on being non-invasive with a vanilla OpenStack installation, not tied to a particular OpenStack release version, The authors want to acknowledge the support of the EGI-Engage (grant number 654142) and INDIGO-Datacloud (grant number 653549) projects, funded by the European Commission’s Horizon 2020 Framework Programme., Peer Reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Artículo científico (JournalArticle)

An efficient cloud scheduler design supporting preemptible instances

  • López-García, Álvaro
  • Fernández-del-Castillo, Enol
  • Campos, Isabel
Maximizing resource utilization by performing an efficient resource provisioning is a key factor for any cloud provider: commercial actors can maximize their revenues, whereas scientific and non-commercial providers can maximize their infrastructure utilization. Traditionally, batch systems have allowed data centers to fill their resources as much as possible by using backfilling and similar techniques. However, in an IaaS cloud, where virtual machines are supposed to live indefinitely, or at least as long as the user is able to pay for them, these policies are not easily implementable. In this work we present a new scheduling algorithm for IaaS providers that is able to support preemptible instances, that can be stopped by higher priority requests without introducing large modifications in the current cloud schedulers. This scheduler enables the implementation of new cloud usage and payment models that allow more efficient usage of the resources and potential new revenue sources for commercial providers. We also study the correctness and the performance overhead of the proposed scheduler against existing solutions., The authors acknowledge the financial support from the European Commission Horizon 2020 via INDIGO-DataCloud project (grant number 653549) and EGI-ENGAGE (grant number 654142) and the Ministry of Economy and Competitiveness for the support through the National Plan under contract number FPA2013-40715-P., Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication Artículo científico (JournalArticle)

Software quality assurance in INDIGO-DataCloud project: a converging evolution of software engineering practices to support European research e-infrastructures

  • Orviz, Pablo
  • David, Mario
  • Duma, Doina Cristina
  • Ronchieri, Elisabetta
  • Gomes, Jorge
  • Salomoni, Davide
From the advent of Grid technology - as the new paradigm of distributed computing - to the current days of Cloud computing models, the continuous need of new tools and services to match the scientific community requirements has been addressed in Europe through dedicated software development projects for e-Infrastructure creation, operation and management. This work presents the most signi cant software quality breakthroughs obtained in one of such projects, INDIGO-DataCloud, the main challenges and barriers confronted throughout the lifespan of the project, and how they were partially or totally overcome. The knowledge base established throughout the last 15 years of diverse software development initiatives in Europe for sustaining distributed research e-Infrastructures, supported by the advances in the area of software engineering, de nitely contributed to improve the quality and reliability of the software delivered, and consequently, the operational stability of the European e-Infrastructures. INDIGO-DataCloud project is a good evidence of such insights, where, unlike the preceding trend found in past projects, the enforcement of Software Quality Assurance practices has been present since the very early stages of the software lifecycle., DataGrid - Research and Technological Development for an International Data Grid project has received funding from the European Union's Fifth Framework Programme under grant agreement IST-2000-25182E. EGEE - Enabling grids For E-science project has received funding from the European Union's Sixth Framework Programme under grant agreement INFSO-RI-508833. EGEE - Enabling grids For E-science-II project has received funding from the European Union's Sixth Framework Programme under grant agreement INFSORI031688. EGEE - Enabling grids For E-science-III project has received funding from the European Union's Seventh Framework Programme under grant agreement INFSO-RI-222667. ETICS - E-Infrastructure for Testing, Integration and Configuration of Software project has received funding from the European Union's Sixth Framework Programme under grant agreement INFSO-RI-026753. ETICS - E-Infrastructure for Testing, Integration and Configuration of Software - Phase 2 project has received funding from the European Union's Seventh Framework Programme under grant agreement INFSO-RI-223782. EGI-InSPIRE - European Grid Initiative: Integrated Sustainable Pan-European Infrastructure for Researchers in Europe project has received funding from the European Union's Seventh Framework Programme under grant agreement INFSO-RI-261323. EGI-Engage project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement RIA-654142. INDIGO-DataCloud project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement RIA-653549., Peer reviewed
Dipòsit Digital de Documents de la UAB
Publication Artículo científico (JournalArticle)

A Critical Path File Location (CPFL) algorithm for data-aware multiworkflow scheduling on HPC clusters

  • Acevedo Giménez, César Esteban
  • Hernández Budé, Porfidio|||0000-0002-8592-934X
  • Espinosa, Antonio|||0000-0002-6460-3789
  • Méndez Muñoz, Víctor|||0000-0002-9044-1189
A representative set of workflows found in bioinformatics pipelines must deal with large data sets. Most scientific workflows are defined as Direct Acyclic Graphs (DAGs). Despite DAGs are useful to understand dependence relationships, they do not provide any information about input, output and temporal data files. This information about the location of files of data intensive applications helps to avoid performance issues. This paper presents a multiworkflow store-aware scheduler in a cluster environment called Critical Path File Location (CPFL) policy where the access time to disk is more relevant than network, as an extension of the classical list scheduling policies. Our purpose is to find the best location of data files in a hierarchical storage system. The resulting algorithm is tested in an HPC cluster and in a simulated cluster scenario with bioinformatics synthetic workflows, and largely used benchmarks like Montage and Epigenomics. The resulting simulator is tuned and validated with the first test results from the real infrastructure. The evaluation of our proposal shows promising results up to 70% on benchmarks in real HPC clusters using 128 cores and up to 69% of makespan improvement on simulated 512 cores clusters with a deviation between 0.9% and 3% regarding the real HPC cluster.