Skip to main content

COMPUTACION BIG DATA Y DE ALTAS PRESTACIONES SOBRE MULTI-CLOUDS ELASTICOS

TIN2016-79951-R

•
Funding agency name Agencia Estatal de Investigación
Funding agency acronym AEI
Program Programa Estatal de I+D+i Orientado a los Retos de la Sociedad
Call Proyectos I+D+i Retos de Investigación
Call year 2016
State Plan 2013-2016 Plan Estatal de Investigación Científica y Técnica y de Innovación
Beneficiary centre UNIVERSITAT POLITÈCNICA DE VALÈNCIA

Resultados relacionados

Resultados totales (Incluyendo duplicados): 13
Encontrada(s) 1 página(s)

RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publication Artículo científico (JournalArticle)

Multi-elastic Datacenters: Auto-scaled Virtual Clusters on Energy-Aware Physical Infrastructures

  • Alfonso Laguna, Carlos De|||0000-0002-2378-021X
  • Caballer Fernández, Miguel|||0000-0001-9393-3077
  • Calatrava Arroyo, Amanda|||0000-0002-9018-9171
  • Moltó, Germán|||0000-0002-8049-253X
  • Blanquer Espert, Ignacio|||0000-0003-1692-8922
[EN] Computer clusters are widely used platforms to execute different computational workloads. Indeed, the advent of virtualization and Cloud computing has paved the way to deploy virtual elastic clusters on top of Cloud infrastructures, which are typically backed by physical computing clusters. In turn, the advances in Green computing have fostered the ability to dynamically power on the nodes of physical clusters as required. Therefore, this paper introduces an open-source framework to deploy elastic virtual clusters running on elastic physical clusters where the computing capabilities of the virtual clusters are dynamically changed to satisfy both the user application's computing requirements and to minimise the amount of energy consumed by the underlying physical cluster that supports an on-premises Cloud. For that, we integrate: i) an elasticity manager both at the infrastructure level (power management) and at the virtual infrastructure level (horizontal elasticity); ii) an automatic Virtual Machine (VM) consolidation agent that reduces the amount of powered on physical nodes using live migration and iii) a vertical elasticity manager to dynamically and transparently change the memory allocated to VMs, thus fostering enhanced consolidation. A case study based on real datasets executed on a production infrastructure is used to validate the proposed solution. The results show that a multi-elastic virtualized datacenter provides users with the ability to deploy customized scalable computing clusters while reducing its energy footprint., The results of this work have been partially supported by ATMOSPHERE (Adaptive, Trustworthy, Manageable, Orchestrated, Secure, Privacy-assuring Hybrid, Ecosystem for Resilient Cloud Computing), funded by the European Commission under the Cooperation Programme, Horizon 2020 grant agreement No 777154.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publication Artículo científico (JournalArticle)

A Self-managed Mesos Cluster for Data Analytics with QoS Guarantees

  • López-Huguet, Sergio
  • Pérez-González, Alfonso María
  • Calatrava Arroyo, Amanda|||0000-0002-9018-9171
  • Alfonso Laguna, Carlos De|||0000-0002-2378-021X
  • Caballer Fernández, Miguel|||0000-0001-9393-3077
  • Moltó, Germán|||0000-0002-8049-253X
  • Blanquer Espert, Ignacio|||0000-0003-1692-8922
[EN] This article describes the development of an automated configuration of a software platform for Data Analytics that supports horizontal and vertical elasticity to guarantee meeting a specific deadline. It specifies all the components, software dependencies and configurations required to build up the cluster, and analyses the deployment times of different instances, as well as the horizontal and vertical elasticity. The approach followed builds up self-managed hybrid clusters that can deal with different workloads and network requirements. The article describes the structure of the recipes, points out to public repositories where the code is available and discusses the limitations of the approach as well as the results of several experiments., The work presented in this article has been partially funded by a research grant from the regional government of the Comunitat Valenciana (Spain), co-funded by the European Union ERDF funds (European Regional Development Fund) of the Comunitat Valenciana 2014-2020, with reference IDIFEDER/2018/032 (High-Performance Algorithms for the Modelling, Simulation and early Detection of diseases in Personalized Medicine). The authors would also like to thank the Spanish "Ministerio de Economia, Industria y Competitividad" for the project "BigCLOE" with reference number TIN2016-79951-R.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publication Artículo científico (JournalArticle)

A Visual Dashboard to Track Learning Analytics for Educational Cloud Computing

  • Naranjo, Diana M.
  • Prieto, José Ramón
  • Moltó, Germán|||0000-0002-8049-253X
  • Calatrava Arroyo, Amanda|||0000-0002-9018-9171
[EN] Cloud providers such as Amazon Web Services (AWS) stand out as useful platforms to teach distributed computing concepts as well as the development of Cloud-native scalable application architectures on real-world infrastructures. Instructors can benefit from high-level tools to track the progress of students during their learning paths on the Cloud, and this information can be disclosed via educational dashboards for students to understand their progress through the practical activities. To this aim, this paper introduces CloudTrail-Tracker, an open-source platform to obtain enhanced usage analytics from a shared AWS account. The tool provides the instructor with a visual dashboard that depicts the aggregated usage of resources by all the students during a certain time frame and the specific use of AWS for a specific student. To facilitate self-regulation of students, the dashboard also depicts the percentage of progress for each lab session and the pending actions by the student. The dashboard has been integrated in four Cloud subjects that use different learning methodologies (from face-to-face to online learning) and the students positively highlight the usefulness of the tool for Cloud instruction in AWS. This automated procurement of evidences of student activity on the Cloud results in close to real-time learning analytics useful both for semi-automated assessment and student self-awareness of their own training progress., This research was funded by the Spanish Ministerio de Economia, Industria y Competitividad, grant number TIN2016-79951-R (BigCLOE) and by the Vicerrectorado de Estudios, Calidad y Acreditacion of the Universitat Politecnica de Valencia (UPV) to develop the PIME B29.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publication Artículo científico (JournalArticle)

A framework and a performance assessment for serverless MapReduce on AWS Lambda

  • Giménez-Alventosa, Vicent|||0000-0003-1646-6094
  • Moltó, Germán|||0000-0002-8049-253X
  • Caballer Fernández, Miguel|||0000-0001-9393-3077
[EN] MapReduce is one of the most widely used programming models for analysing large-scale datasets, i.e. Big Data. In recent years, serverless computing and, in particular, Functions as a Service (FaaS) has surged as an execution model in which no explicit management of servers (e.g. virtual machines) is performed by the user. Instead, the Cloud provider dynamically allocates resources to the function invocations and fine-grained billing is introduced depending on the execution time and allocated memory, as exemplified by AWS Lambda. In this article, a high-performant serverless architecture has been created to execute MapReduce jobs on AWS Lambda using Amazon S3 as the storage backend. In addition, a thorough assessment has been carried out to study the suitability of AWS Lambda as a platform for the execution of High Throughput Computing jobs. The results indicate that AWS Lambda provides a convenient computing platform for general-purpose applications that fit within the constraints of the service (15 min of maximum execution time, 3008 MB of RAM and 512 MB of disk space) but it exhibits an inhomogeneous performance behaviour that may jeopardise adoption for tightly coupled computing jobs., This study was supported by the program "Ayudas para la contratacion de personal investigador en formacion de caracter pre-doctoral, programa VALid-d" under grant number ACIF/2018/148 from the Conselleria d'Educacio of the Generalitat Valenciana, Spain. The authors would also like to thank the Spanish "Ministerio de Economia, Industria y Competitividad" for the project "BigCLOE" with reference number TIN2016-79951-R.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publication Artículo científico (JournalArticle)

Insights from Learning Analytics for Hands-On Cloud Computing Labs in AWS

  • Moltó, Germán|||0000-0002-8049-253X
  • Segrelles Quilis, José Damián|||0000-0001-5698-7965
  • Naranjo-Delgado, Diana María
[EN] Cloud computing instruction requires hands-on experience with a myriad of distributed computing services from a public cloud provider. Tracking the progress of the students, especially for online courses, requires one to automatically gather evidence and produce learning analytics in order to further determine the behavior and performance of students. With this aim, this paper describes the experience from an online course in cloud computing with Amazon Web Services on the creation of an open-source data processing tool to systematically obtain learning analytics related to the hands-on activities carried out throughout the course. These data, combined with the data obtained from the learning management system, have allowed the better characterization of the behavior of students in the course. Insights from a population of more than 420 online students through three academic years have been assessed, the dataset has been released for increased reproducibility. The results corroborate that course length has an impact on online students dropout. In addition, a gender analysis pointed out that there are no statistically significant differences in the final marks between genders, but women show an increased degree of commitment with the activities planned in the course., This research was funded by the Spanish "Ministerio de Economia, Industria y Competitividad through grant number TIN2016-79951-R (BigCLOE)", the "Vicerrectorado de Estudios, Calidad y Acreditacion" of the Universitat Politecnica de Valencia (UPV) to develop the PIME B29 and PIME/19-20/166, and by the Conselleria d'Innovacio, Universitat, Ciencia i Societat Digital for the project "CloudSTEM" with reference number AICO/2019/313.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publication Artículo científico (JournalArticle)

Accelerated serverless computing based on GPU virtualization

  • Naranjo-Delgado, Diana María
  • Risco, Sebastián
  • Pérez-González, Alfonso María
  • Alfonso Laguna, Carlos De|||0000-0002-2378-021X
  • Blanquer Espert, Ignacio|||0000-0003-1692-8922
  • Moltó, Germán|||0000-0002-8049-253X
[EN] This paper introduces a platform to support serverless computing for scalable event-driven data processing that features a multi-level elasticity approach combined with virtualization of GPUs. The platform supports the execution of applications based on Docker containers in response to file uploads to a data storage in order to perform the data processing in parallel. This is managed by an elastic Kubernetes cluster whose size automatically grows and shrinks depending on the number of files to be processed. To accelerate the processing time of each file, several approaches involving virtualized access to GPUs, either locally or remote, have been evaluated. A use case that involves the inference based on deep learning techniques on transtoracic echocardiography imaging has been carried out to assess the benefits and limitations of the platform. The results indicate that the combination of serverless computing and GPU virtualization introduce an efficient and cost-effective event-driven accelerated computing approach that can be applied for a wide variety of scientific applications., The work presented in this article has been partially funded by a research grant from the regional government of the Comunitat Valenciana (Spain), co-funded by the European Union ERDF funds (European Regional Development Fund) of the Comunitat Valenciana 2014-2020, with reference IDIFEDER/2018/032 (High-Performance Algorithms for the Modeling, Simulation and early Detection of diseases in Personalized Medicine). The authors would also like to thank the Spanish "Ministerio de Economia, Industria y Competitividad" for the project "BigCLOE" with reference number TIN2016-79951-R and the project ATMOSPHERE, funded jointly by the European Commission under the Cooperation Programme, Horizon 2020 grant agreement No 777154 and the Brazilian Ministerio de Ciencia, Tecnologia e Inovacao (MCI-I), number 51119. D.M.N would like to thank the "Generalitat Valenciana, Spain" for the grant GrisoliaP/2017/071.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publication Artículo científico (JournalArticle)

APRICOT: Advanced Platform for Reproducible Infrastructures in the Cloud via Open Tools

  • Giménez-Alventosa, Vicent|||0000-0003-1646-6094
  • Segrelles Quilis, José Damián|||0000-0001-5698-7965
  • Moltó, Germán|||0000-0002-8049-253X
  • Roca-Sogorb, Mar
[EN] Background Scientific publications are meant to exchange knowledge among researchers but the inability to properly reproduce computational experiments limits the quality of scientific research. Furthermore, bibliography shows that irreproducible preclinical research exceeds 50%, which produces a huge waste of resources on nonprofitable research at Life Sciences field. As a consequence, scientific reproducibility is being fostered to promote Open Science through open databases and software tools that are typically deployed on existing computational resources. However, some computational experiments require complex virtual infrastructures, such as elastic clusters of PCs, that can be dynamically provided from multiple clouds. Obtaining these infrastructures requires not only an infrastructure provider, but also advanced knowledge in the cloud computing field. Objectives The main aim of this paper is to improve reproducibility in life sciences to produce better and more cost-effective research. For that purpose, our intention is to simplify the infrastructure usage and deployment for researchers. Methods This paper introduces Advanced Platform for Reproducible Infrastructures in the Cloud via Open Tools (APRICOT), an open source extension for Jupyter to deploy deterministic virtual infrastructures across multiclouds for reproducible scientific computational experiments. To exemplify its utilization and how APRICOT can improve the reproduction of experiments with complex computation requirements, two examples in the field of life sciences are provided. All requirements to reproduce both experiments are disclosed within APRICOT and, therefore, can be reproduced by the users. Results To show the capabilities of APRICOT, we have processed a real magnetic resonance image to accurately characterize a prostate cancer using a Message Passing Interface cluster deployed automatically with APRICOT. In addition, the second example shows how APRICOT scales the deployed infrastructure, according to the workload, using a batch cluster. This example consists of a multiparametric study of a positron emission tomography image reconstruction. Conclusion APRICOT's benefits are the integration of specific infrastructure deployment, the management and usage for Open Science, making experiments that involve specific computational infrastructures reproducible. All the experiment steps and details can be documented at the same Jupyter notebook which includes infrastructure specifications, data storage, experimentation execution, results gathering, and infrastructure termination. Thus, distributing the experimentation notebook and needed data should be enough to reproduce the experiment., This study was supported by the program "Ayudas para la contratación de personal investigador en formación de carácter predoctoral, programa VALi+d" under grant number ACIF/2018/148 from the Conselleria d'Educació of the Generalitat Valenciana and the "Fondo Social Europeo" (FSE). The authors would like to thank the Spanish "Ministerio de Economía, Industria y Competitividad" for the project "BigCLOE" with reference number TIN2016-79951-R and the European Commission, Horizon 2020 grant agreement No 826494 (PRIMAGE). The MRI prostate study case used in this article has been retrospectively collected from a project of prostate MRI biomarkers validation.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publication Artículo científico (JournalArticle)

GPU-Enabled Serverless Workflows for Efficient Multimedia Processing

  • Risco, Sebastián
  • Moltó, Germán|||0000-0002-8049-253X
[EN] Serverless computing has introduced scalable event-driven processing in Cloud infrastructures. However, it is not trivial for multimedia processing to benefit from the elastic capabilities featured by serverless applications. To this aim, this paper introduces the evolution of a framework to support the execution of customized runtime environments in AWS Lambda in order to accommodate workloads that do not satisfy its strict computational requirements: increased execution times and the ability to use GPU-based resources. This has been achieved through the integration of AWS Batch, a managed service to deploy virtual elastic clusters for the execution of containerized jobs. In addition, a Functions Definition Language (FDL) is introduced for the description of data-driven workflows of functions. These workflows can simultaneously leverage both AWS Lambda for the highly-scalable execution of short jobs and AWS Batch, for the execution of compute-intensive jobs that can profit from GPU-based computing. To assess the developed open-source framework, we executed a case study for efficient serverless video processing. The workflow automatically generates subtitles based on the audio and applies GPU-based object recognition to the video frames, thus simultaneously harnessing different computing services. This allows for the creation of cost-effective highly-parallel scale-to-zero serverless workflows in AWS., This research was funded by the Spanish "Ministerio de Economia, Industria y Competitividad" for the project "BigCLOE" with reference number TIN2016-79951-R
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publication Artículo científico (JournalArticle)

Vertical Elasticity on Marathon and Chronos Mesos frameworks

  • López-Huguet, Sergio
  • Natanael, I.
  • Brito, A.
  • Blanquer Espert, Ignacio|||0000-0003-1692-8922
[EN] Marathon and Chronos are two popular Mesos frameworks that are widely used for deploying fault-tolerant services and periodic batch jobs. Marathon and Chronos provide by design mechanisms for horizontal elasticity, scaling up and down the number of job and service instances. Horizontal elasticity is appropriate when the problems that are solved are inherently parallel. However, when the problem cannot benefit from an increase of the amount of resources, vertical elasticity must be considered. This work implements on top of Marathon and Chronos Mesos frameworks, a mechanism to vary the resources associated to an executor dynamically, according to its progress and considering specific Quality of Service (QoS). The mechanism developed provides a wrapper executable and a service that takes the decision of increasing or decreasing the resources allocated to different Chronos iterations or a long-living Marathon application. The mechanism makes use of checkpointing techniques to preserve the execution of Marathon applications and leverages OpenStack Monasca for the monitoring. \footnote{The work in this article has been funded by projects BIGCLOE and EUBra-BIGSEA, BIGLOE is funded by the Spanish ``Ministerio de Econom\'ia, Industria y Competitividad" with reference number TIN2016-79951-R and EUBra-BIGSEA is funded jointly by the European Commission under the Cooperation Programme, Horizon 2020 grant agreement No 690116 and the Brazilian Ministério de Ciência, Tecnologia e Inovação (MCTI)., The work in this article has been funded by projects BIGCLOE and EUBra BIGSEA, BIGLOE is funded by the Spanish "Ministerio de Economia, Industria y Competitividad" with reference number TIN2016-79951-R and EUBra-BIGSEA is funded jointly by the European Commission under the Cooperation Programme, Horizon 2020 grant agreement No 690116 and the Brazilian Ministerio de Ciencia, Tecnologia e Inovacao (MCTI).
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publication Artículo científico (JournalArticle)

Deployment of Elastic Virtual Hybrid Clusters Across Cloud Sites

  • Caballer Fernández, Miguel|||0000-0001-9393-3077
  • Moltó, Germán|||0000-0002-8049-253X
  • Antonacci, Marica
  • Sustr, Zdenek
  • Perniola, Michele
[EN] Virtual clusters are widely used computing platforms than can be deployed in multiple cloud platforms. The ability to dynamically grow and shrink the number of nodes has paved the way for customised elastic computing both for High Performance Computing and High Throughput Computing workloads. However, elasticity is typically restricted to a single cloud site, thus hindering the ability to provision computational resources from multiple geographically distributed cloud sites. To this aim, this paper introduces an architecture of open-source components that coherently deploy a virtual elastic cluster across multiple cloud sites to perform large-scale computing. These hybrid virtual elastic clusters are automatically deployed and configured using an Infrastructure as Code (IaC) approach on a distributed hybrid testbed that spans different organizations, including on-premises and public clouds, supporting automated tunneling of communications across the cluster nodes with advanced VPN topologies. The results indicate that cluster-based computing of embarrassingly parallel jobs can benefit from hybrid virtual clusters that aggregate computing resources from multiple cloud back-ends and bring them together into a dedicated, albeit virtual network., The work presented in this article has been partially funded by project DEEP Hybrid-DataCloud (grant agreement No 777435). GM and MC would also like to thank the Spanish "Ministerio de Economia, Industria y Competitividad" for the project "BigCLOE" with reference number TIN2016-79951-R. Computational resources at CESNET, used in the real-world use case, were supplied by the project "e-Infrastruktura CZ" (e-INFRA LM2018140) provided within the program Projects of Large Research, Development and Innovations Infrastructures
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publication Artículo científico (JournalArticle)

Toward Bio-Inspired Auto-Scaling Algorithms: An Elasticity Approach for Container Orchestration Platforms

  • Herrera, Jose
  • Moltó, Germán|||0000-0002-8049-253X
[EN] The wide adoption of microservices architectures has introduced an unprecedented granularisation of computing that requires the coordinated execution of multiple containers with diverse lifetimes and with potentially different auto-scaling requirements. These applications are managed by means of container orchestration platforms and existing centralised approaches for auto-scaling face challenges when used for the timely adaptation of the elasticity required for the different application components. This paper studies the impact of integrating bio-inspired approaches for dynamic distributed auto-scaling on container orchestration platforms. With a focus on running self-managed containers, we compare alternative configuration options for the container life cycle. The performance of the proposed models is validated through simulations subjected to both synthetic and real-world workloads. Also, multiple scaling options are assessed with the purpose of identifying exceptional cases and improvement areas. Furthermore, a nontraditional metric for scaling measurement is introduced to substitute classic analytical approaches. We found out connections for two related worlds (biological systems and software container elasticity procedures) and we open a new research area in software containers that features potential self-guided container elasticity activities., This work was supported by the Ministerio de Economía, Industria y Competitividad, Spanish Government, for the Project BigCLOE under Grant TIN2016-79951-R
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publication Artículo científico (JournalArticle)

Virtualized Computational Environments on the cloud to foster group skills through PBL: A case study in architecture

  • Segrelles Quilis, José Damián|||0000-0001-5698-7965
  • Martínez Antón, Alicia|||0000-0002-8897-8809
  • Castilla-Cabanes, Nuria|||0000-0002-9993-828X
  • Moltó, Germán|||0000-0002-8049-253X
The ODISEA platform provides Virtualized Computational Environments (VCEs) on cloud providers as the computational infrastructure to support educational activities. A VCE consists of a collection of one or more Virtual Machines (VMs) to which the students connect from their own computers. In this paper a case study is presented in the architecture domain where a PBL activity is carried out in working groups. The study involves 293 students organized in 28 pilot groups that use customized VCEs created and deployed through the ODISEA platform on a Cloud, and 30 traditional groups that use a LMS platform. The VCE provides the software, hardware and specific configuration to ease the interrelation and cooperative work between the working groups, enhancing the process tracking and feedback gathering as well as providing a better organization of the teaching material. The results demonstrate that the VCE allows to improve the cooperative work, improving the final marks in the PBL developed by the pilot working groups. Also, an economical study is presented, highlighting the economic benefits of the Cloud with respect to raditional physical laboratories of PCs., The authors wish to thank the financial support received from Vicerrectorado de Estudios, Calidad y Acreditacion of the Universitat Politecnica de Valencia (UPV) to develop the PIME project "Entomos Virtuales Computacionales para la Evaluation de Competencias Transversales en la Nube", with reference A04 and to the Ministerio de Economia, Industria y Cornpetitividad for the project BigCLOE (TIN2016-79951-R). GM would like to thank AWS for the AWS Educate program that provides the credits required to support the educational activities.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publication Artículo científico (JournalArticle)

Serverless computing for container-based architectures

  • Pérez-González, Alfonso María
  • Moltó, Germán|||0000-0002-8049-253X
  • Caballer Fernández, Miguel|||0000-0001-9393-3077
  • Calatrava Arroyo, Amanda|||0000-0002-9018-9171
[EN] New architectural patterns (e.g. microservices), the massive adoption of Linux contain- ers (e.g. Docker containers), and improvements in key features of Cloud computing such as auto-scaling, have helped developers to decouple complex and monolithic sys- tems into smaller stateless services. In turn, Cloud providers have introduced serverless computing, where applications can be defined as a workflow of event-triggered functions. However, serverless services, such as AWS Lambda, impose serious restrictions for these applications (e.g. using a predefined set of programming languages or difficulting the installation and deployment of external libraries). This paper addresses such issues by introducing a framework and a methodology to create Serverless Container-aware AR- chitectures (SCAR). The SCAR framework can be used to create highly-parallel event- driven serverless applications that run on customized runtime environments defined as Docker images on top of AWS Lambda. This paper describes the architecture of SCAR together with the cache-based optimizations applied to minimize cost, exemplified on a massive image processing use case. The results show that, by means of SCAR, AWS Lambda becomes a convenient platform for High Throughput Computing, specially for highly-parallel bursty workloads of short stateless jobs., The authors would like to thank the Spanish "Ministerio de Economia, Industria y Competitividad" for the project "BigCLOE" under grant reference TIN2016-79951-R. The authors would also like to thank Jorge Gomes from LIP for the development of the udocker tool.