Skip to main content

COMPUTACION BIG DATA Y DE ALTAS PRESTACIONES SOBRE MULTI-CLOUDS ELASTICOS

TIN2016-79951-R

•
Funding agency name Agencia Estatal de Investigación
Funding agency acronym AEI
Program Programa Estatal de I+D+i Orientado a los Retos de la Sociedad
Call Proyectos I+D+i Retos de Investigación
Call year 2016
State Plan 2013-2016 Plan Estatal de Investigación Científica y Técnica y de Innovación
Beneficiary centre UNIVERSITAT POLITÈCNICA DE VALÈNCIA

Resultados relacionados

Resultados totales (Incluyendo duplicados): 16
Encontrada(s) 1 página(s)

RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publication Artículo científico (JournalArticle)

Multi-elastic Datacenters: Auto-scaled Virtual Clusters on Energy-Aware Physical Infrastructures

  • Alfonso Laguna, Carlos De|||0000-0002-2378-021X
  • Caballer Fernández, Miguel|||0000-0001-9393-3077
  • Calatrava Arroyo, Amanda|||0000-0002-9018-9171
  • Moltó, Germán|||0000-0002-8049-253X
  • Blanquer Espert, Ignacio|||0000-0003-1692-8922
[EN] Computer clusters are widely used platforms to execute different computational workloads. Indeed, the advent of virtualization and Cloud computing has paved the way to deploy virtual elastic clusters on top of Cloud infrastructures, which are typically backed by physical computing clusters. In turn, the advances in Green computing have fostered the ability to dynamically power on the nodes of physical clusters as required. Therefore, this paper introduces an open-source framework to deploy elastic virtual clusters running on elastic physical clusters where the computing capabilities of the virtual clusters are dynamically changed to satisfy both the user application's computing requirements and to minimise the amount of energy consumed by the underlying physical cluster that supports an on-premises Cloud. For that, we integrate: i) an elasticity manager both at the infrastructure level (power management) and at the virtual infrastructure level (horizontal elasticity); ii) an automatic Virtual Machine (VM) consolidation agent that reduces the amount of powered on physical nodes using live migration and iii) a vertical elasticity manager to dynamically and transparently change the memory allocated to VMs, thus fostering enhanced consolidation. A case study based on real datasets executed on a production infrastructure is used to validate the proposed solution. The results show that a multi-elastic virtualized datacenter provides users with the ability to deploy customized scalable computing clusters while reducing its energy footprint., The results of this work have been partially supported by ATMOSPHERE (Adaptive, Trustworthy, Manageable, Orchestrated, Secure, Privacy-assuring Hybrid, Ecosystem for Resilient Cloud Computing), funded by the European Commission under the Cooperation Programme, Horizon 2020 grant agreement No 777154.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publication Artículo científico (JournalArticle)

A Self-managed Mesos Cluster for Data Analytics with QoS Guarantees

  • López-Huguet, Sergio
  • Pérez-González, Alfonso María
  • Calatrava Arroyo, Amanda|||0000-0002-9018-9171
  • Alfonso Laguna, Carlos De|||0000-0002-2378-021X
  • Caballer Fernández, Miguel|||0000-0001-9393-3077
  • Moltó, Germán|||0000-0002-8049-253X
  • Blanquer Espert, Ignacio|||0000-0003-1692-8922
[EN] This article describes the development of an automated configuration of a software platform for Data Analytics that supports horizontal and vertical elasticity to guarantee meeting a specific deadline. It specifies all the components, software dependencies and configurations required to build up the cluster, and analyses the deployment times of different instances, as well as the horizontal and vertical elasticity. The approach followed builds up self-managed hybrid clusters that can deal with different workloads and network requirements. The article describes the structure of the recipes, points out to public repositories where the code is available and discusses the limitations of the approach as well as the results of several experiments., The work presented in this article has been partially funded by a research grant from the regional government of the Comunitat Valenciana (Spain), co-funded by the European Union ERDF funds (European Regional Development Fund) of the Comunitat Valenciana 2014-2020, with reference IDIFEDER/2018/032 (High-Performance Algorithms for the Modelling, Simulation and early Detection of diseases in Personalized Medicine). The authors would also like to thank the Spanish "Ministerio de Economia, Industria y Competitividad" for the project "BigCLOE" with reference number TIN2016-79951-R.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publication Artículo científico (JournalArticle)

A Visual Dashboard to Track Learning Analytics for Educational Cloud Computing

  • Naranjo, Diana M.
  • Prieto, José Ramón
  • Moltó, Germán|||0000-0002-8049-253X
  • Calatrava Arroyo, Amanda|||0000-0002-9018-9171
[EN] Cloud providers such as Amazon Web Services (AWS) stand out as useful platforms to teach distributed computing concepts as well as the development of Cloud-native scalable application architectures on real-world infrastructures. Instructors can benefit from high-level tools to track the progress of students during their learning paths on the Cloud, and this information can be disclosed via educational dashboards for students to understand their progress through the practical activities. To this aim, this paper introduces CloudTrail-Tracker, an open-source platform to obtain enhanced usage analytics from a shared AWS account. The tool provides the instructor with a visual dashboard that depicts the aggregated usage of resources by all the students during a certain time frame and the specific use of AWS for a specific student. To facilitate self-regulation of students, the dashboard also depicts the percentage of progress for each lab session and the pending actions by the student. The dashboard has been integrated in four Cloud subjects that use different learning methodologies (from face-to-face to online learning) and the students positively highlight the usefulness of the tool for Cloud instruction in AWS. This automated procurement of evidences of student activity on the Cloud results in close to real-time learning analytics useful both for semi-automated assessment and student self-awareness of their own training progress., This research was funded by the Spanish Ministerio de Economia, Industria y Competitividad, grant number TIN2016-79951-R (BigCLOE) and by the Vicerrectorado de Estudios, Calidad y Acreditacion of the Universitat Politecnica de Valencia (UPV) to develop the PIME B29.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publication Artículo científico (JournalArticle)

A framework and a performance assessment for serverless MapReduce on AWS Lambda

  • Giménez-Alventosa, Vicent|||0000-0003-1646-6094
  • Moltó, Germán|||0000-0002-8049-253X
  • Caballer Fernández, Miguel|||0000-0001-9393-3077
[EN] MapReduce is one of the most widely used programming models for analysing large-scale datasets, i.e. Big Data. In recent years, serverless computing and, in particular, Functions as a Service (FaaS) has surged as an execution model in which no explicit management of servers (e.g. virtual machines) is performed by the user. Instead, the Cloud provider dynamically allocates resources to the function invocations and fine-grained billing is introduced depending on the execution time and allocated memory, as exemplified by AWS Lambda. In this article, a high-performant serverless architecture has been created to execute MapReduce jobs on AWS Lambda using Amazon S3 as the storage backend. In addition, a thorough assessment has been carried out to study the suitability of AWS Lambda as a platform for the execution of High Throughput Computing jobs. The results indicate that AWS Lambda provides a convenient computing platform for general-purpose applications that fit within the constraints of the service (15 min of maximum execution time, 3008 MB of RAM and 512 MB of disk space) but it exhibits an inhomogeneous performance behaviour that may jeopardise adoption for tightly coupled computing jobs., This study was supported by the program "Ayudas para la contratacion de personal investigador en formacion de caracter pre-doctoral, programa VALid-d" under grant number ACIF/2018/148 from the Conselleria d'Educacio of the Generalitat Valenciana, Spain. The authors would also like to thank the Spanish "Ministerio de Economia, Industria y Competitividad" for the project "BigCLOE" with reference number TIN2016-79951-R.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publication Tesis doctoral (DoctoralThesis)

Plataformes avançades en el Núvol per a la reproductibilitat d&apos, experiments computacionals

  • Giménez-Alventosa, Vicent|||0000-0003-1646-6094
[ES] La tesis presentada se enmarca dentro del ámbito de la ciencia computacional. Dentro de esta, se centra en el desarrollo de herramientas para la ejecución de experimentación científica computacional, el impacto de la cual es cada vez mayor en todos los ámbitos de la ciencia y la ingeniería. Debido a la creciente complejidad de los cálculos realizados, cada vez es necesario un mayor conocimiento de las técnicas y herramientas disponibles para llevar a cabo este tipo de experimentos, ya que pueden requerir, en general, una gran infraestructura computacional para afrontar los altos costes de cómputo. Más aún, la reciente popularización del cómputo en la Nube ofrece una gran variedad de posibilidades para configurar nuestras propias infraestructuras con requisitos específicos. No obstante, el precio a pagar es la complejidad de configurar dichas infraestructuras en este tipo de entornos. Además, el aumento en la complejidad de configuración de los entornos en la nube no hace más que agravar un problema ya existente en el ámbito científico, y es el de la reproducibilidad de los resultados publicados. La falta de documentación, como las versiones de software que se han usado para llevar a cabo el cómputo, o los datos requeridos, provocan que una parte significativa de los resultados de experimentos computacionales publicados no sean reproducibles por otros investigadores. Como consecuencia, se produce un derroche de recursos destinados a la investigación. Como respuesta a esta situación, existen, y continúan desarrollándose, diferentes herramientas para facilitar procesos como el despliegue y configuración de infraestructura, el acceso a los datos, el diseño de flujos de cómputo, etc. con el objetivo de que los investigadores puedan centrarse en el problema a abordar. Precisamente, esta es la base de los trabajos desarrollados en la presente tesis, el desarrollo de herramientas para facilitar que el cómputo científico se beneficie de entornos de computación en la Nube de forma eficiente. El primer trabajo presentado empieza con un estudio exhaustivo de las prestaciones d'un servicio relativamente nuevo, la ejecución serverless de funciones. En este, se determinará la conveniencia de usar este tipo de entornos en el cálculo científico midiendo tanto sus prestaciones de forma aislada, como velocidad de CPU y comunicaciones, como en conjunto mediante el desarrollo de una aplicación de procesamiento MapReduce para entornos serverless. En el siguiente trabajo, se abordará una problemática diferente, y es la reproducibilidad de experimentos computacionales. Para conseguirlo, se presentará un entorno, basado en Jupyter, donde se encapsule tanto el proceso de despliegue y configuración de infraestructura computacional como el acceso a datos y la documentación de la experimentación. Toda esta información quedará registrada en el notebook de Jupyter donde se ejecuta el experimento, permitiendo así a otros investigadores reproducir los resultados simplemente compartiendo el notebook correspondiente. Volviendo al estudio de las prestaciones del primer trabajo, teniendo en cuenta las medidas y bien estudiadas fluctuaciones de éstas en entornos compartidos, como el cómputo en la Nube, en el tercer trabajo se desarrollará un sistema de balanceo de carga diseñado expresamente para este tipo de entornos. Como se mostrará, este componente es capaz de gestionar y corregir de forma precisa fluctuaciones impredecibles en las prestaciones del cómputo en entornos compartidos. Finalmente, y aprovechando el desarrollo anterior, se diseñará una plataforma completamente serverless encargada de repartir y balancear tareas ejecutadas en múltiples infraestructuras independientes. La motivación de este último trabajo viene dada por los altos costes computacionales de ciertos experimentos, los cuales fuerzan a los investigadores a usar múltiples infraestructuras que, en general, pertenecen a diferentes organizaciones., [CA] La tesi presentada a aquest document s'emmarca dins de l'àmbit de la ciència computacional. Dintre d'aquesta, es centra en el desenvolupament d'eines per a l'execució d'experimentació científica computacional, la qual té un impacte cada vegada major en tots els àmbits de la ciència i l'enginyeria. Donada la creixent complexitat dels càlculs realitzats, cada vegada és necessari un major coneixement sobre les tècniques i eines disponibles per a dur a terme aquestes experimentacions, ja que poden requerir, en general, una gran infraestructura computacional per afrontar els alts costos de còmput. Més encara, la recent popularització del còmput en el Núvol ofereix una gran varietat de possibilitats per a configurar les nostres pròpies infraestructures amb requisits específiques. No obstant, el preu a pagar és la complexitat de configurar les esmenades infraestructures a aquest tipus d'entorns. A més, l'augment de la complexitat de configuració dels entorns de còmput no ha fet més que agreujar un problema ja existent a l'àmbit científic, i és la reproductibilitat de resultats publicats. La manca de documentació, com les versions del programari emprat per a dur a terme el còmput, o les dades requerides ocasionen que una part no negligible dels resultats d'experiments computacionals publicats no siguen reproduïbles per altres investigadors. Com a conseqüència, es produeix un malbaratament dels recursos destinats a la investigació. Com a resposta a aquesta situació, existeixen, i continuen desenvolupant-se, diverses eines per facilitar processos com el desplegament i configuració d'infraestructura, l'accés a les dades, el disseny de fluxos de còmput, etc. amb l'objectiu de que els investigadors puguen centrar-se en el problema a abordar. Precisament, aquesta és la base dels treballs desenvolupats durant la tesi que segueix, el desenvolupar eines per a facilitar que el còmput científic es beneficiar-se d'entorns de computació en el Núvol d'una forma eficient. El primer treball presentat comença amb un estudi exhaustiu de les prestacions d'un servei relativament nou, l'execució serverless de funcions. En aquest, es determinarà la conveniència d'emprar este tipus d'entorns en el càlcul científic mesurant tant les seues prestacions de forma aïllada, com velocitat de CPU i la velocitat de les comunicacions, com en conjunt a través del desenvolupament d'una aplicació de processament MapReduce per a entorns serverless. Al següent treball, s'abordarà una problemàtica diferent, i és la reproductibilitat dels experiments computacionals. Per a aconseguir-ho, es presentarà una entorn, basat en Jupyter, on s'englobe tant el desplegament i configuració d'infraestructura computacional, com l'accés a les dades requerides i la documentació de l'experimentació. Tota aquesta informació quedarà registrada al notebook de Jupyter on s'executa l'experiment, permetent així a altres investigadors reproduir els resultats simplement compartint el notebook corresponent. Tornant a l'estudi de les prestacions del primer treball, donades les mesurades i ben estudiades fluctuacions d'aquestes en entorns compartits, com en el còmput en el Núvol, al tercer treball es desenvoluparà un sistema de balanceig de càrrega dissenyat expressament per aquest tipus d'entorns. Com es veurà, aquest component és capaç de gestionar i corregir de forma precisa fluctuacions impredictibles en les prestacions de còmput d'entorns compartits. Finalment, i aprofitant el desenvolupament anterior, es dissenyarà una plataforma completament serverless per a repartir i balancejar tasques executades en múltiples infraestructures de còmput independents. La motivació d'aquest últim treball ve donada pels alts costos computacionals de certes experimentacions, els quals forcen als investigadors a emprar múltiples infraestructures que, en general, pertanyen a diferents organitzacions. Es demostrarà la capacitat de la plataforma per balancejar treballs i minimitzar el malbaratament de recursos, [EN] This document is focused on computational science, specifically in the development of tools for executions of scientific computational experiments, whose impact has increased, and still increasing, in all scientific and engineering scopes. Considering the growing complexity of scientific calculus, it is required large and complex computational infrastructures to carry on the experimentation. However, to use this infrastructures, it is required a deep knowledge of the available tools and techniques to be handled efficiently. Moreover, the popularity of Cloud computing environments offers a wide variety of possible configurations for our computational infrastructures, thus complicating the configuration process. Furthermore, this increase in complexity has exacerbated the well known problem of reproducibility in science. The lack of documentation, as the used software versions, or the data required by the experiment, produces non reproducible results in computational experiments. This situation produce a non negligible waste of the resources invested in research. As consequence, several tools have been developed to facilitate the deployment, usage and configuration of complex infrastructures, provide access to data, etc. with the objective to simplify the common steps of computational experiments to researchers. Moreover, the works presented in this document share the same objective, i.e. develop tools to provide an easy, efficient and reproducible usage of cloud computing environments for scientific experimentation. The first presented work begins with an exhaustive study of the suitability of the AWS serverless environment for scientific calculus. In this one, the suitability of this kind of environments for scientific research will be studied. With this aim, the study will measure the CPU and network performance, both isolated and combined, via a MapReduce framework developed completely using serverless services. The second one is focused on the reproducibility problem in computational experiments. To improve reproducibility, the work presents an environment, based on Jupyter, which handles and simplify the deployment, configuration and usage of complex computational infrastructures. Also, includes a straight forward procedure to provide access to data and documentation of the experimentation via the Jupyter notebooks. Therefore, the whole experiment could be reproduced sharing the corresponding notebook. In the third work, a load balance library has been developed to address fluctuations of shared infrastructure capabilities. This effect has been wide studied in the literature and affects specially to cloud computing environments. The developed load balance system, as we will see, can handle and correct accurately unpredictable fluctuations in such environments. Finally, based on the previous work, a completely serverless platform is presented to split and balance job executions among several shared, heterogeneous and independent computing infrastructures. The motivation of this last work is the huge computational cost of many experiments, which forces the researchers to use multiple infrastructures belonging, in general, to different organisations. It will be shown how the developed platform is capable to balance the workload accurately. Moreover, it can fit execution time constrains specified by the user. In addition, the platform assists the computational infrastructures to scale as a function of the incoming workload, avoiding an over-provisioning or under-provisioning. Therefore, the platform provides an efficient usage of the available resources., This study was supported by the program “Ayudas para la contratación de personal investigador en formación de carácter predoctoral, programa VALi+d” under grant number ACIF/2018/148 from the Conselleria d’Educació of the Generalitat Valenciana. The authors would also like to thank the Spanish "Ministerio de Economía, Industria y Competitividad"for the project “BigCLOE” with reference number TIN2016-79951-R.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publication Artículo científico (JournalArticle)

Insights from Learning Analytics for Hands-On Cloud Computing Labs in AWS

  • Moltó, Germán|||0000-0002-8049-253X
  • Segrelles Quilis, José Damián|||0000-0001-5698-7965
  • Naranjo-Delgado, Diana María
[EN] Cloud computing instruction requires hands-on experience with a myriad of distributed computing services from a public cloud provider. Tracking the progress of the students, especially for online courses, requires one to automatically gather evidence and produce learning analytics in order to further determine the behavior and performance of students. With this aim, this paper describes the experience from an online course in cloud computing with Amazon Web Services on the creation of an open-source data processing tool to systematically obtain learning analytics related to the hands-on activities carried out throughout the course. These data, combined with the data obtained from the learning management system, have allowed the better characterization of the behavior of students in the course. Insights from a population of more than 420 online students through three academic years have been assessed, the dataset has been released for increased reproducibility. The results corroborate that course length has an impact on online students dropout. In addition, a gender analysis pointed out that there are no statistically significant differences in the final marks between genders, but women show an increased degree of commitment with the activities planned in the course., This research was funded by the Spanish "Ministerio de Economia, Industria y Competitividad through grant number TIN2016-79951-R (BigCLOE)", the "Vicerrectorado de Estudios, Calidad y Acreditacion" of the Universitat Politecnica de Valencia (UPV) to develop the PIME B29 and PIME/19-20/166, and by the Conselleria d'Innovacio, Universitat, Ciencia i Societat Digital for the project "CloudSTEM" with reference number AICO/2019/313.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publication Contribución de congreso (ConferenceOutput)

A Cloud Architecture for the Execution of Medical Imaging Biomarkers

  • López-Huguet, Sergio
  • García-Castro, Fabio
  • Alberich-Bayarri, Angel
  • Blanquer Espert, Ignacio|||0000-0003-1692-8922
Digital Medical Imaging is increasingly being used in clinical routine and research. As a consequence, the workload in medical imaging departments in hospitals has multiplied by over 20 in the last decade. Medical Image processing requires intensive computing resources not available at hospitals, but which could be provided by public clouds. The article analyses the requirements of processing digital medical images and introduces a cloud-based architecture centred on a DevOps approach to deploying resources on demand, adjusting them based on the request of resources and the expected execution time to deal with an unplanned workload. Results presented show a low overhead and high flexibility executing a lung disease biomarker on a public cloud., The work in this article has been co-funded by project SME Instrument Phase II - 778064, QUIBIM Precision, funded by the European Commission under the INDUSTRIAL LEADERSHIP - Leadership in enabling and industrial technologies - Information and Communication Technologies (ICT), Horizon 2020, project ATMOSPHERE, funded jointly by the European Commission under the Cooperation Programme, Horizon 2020 grant agreement No 777154 and the Brazilian Ministerio de Ciencia, Tecnologia e Inovacao (MCTI), number 51119. The authors would like also to thank the Spanish Ministerio de Economia, Industria y Competitividad¿ for the project BigCLOE with reference number TIN2016-79951-R.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publication Artículo científico (JournalArticle)

Accelerated serverless computing based on GPU virtualization

  • Naranjo-Delgado, Diana María
  • Risco, Sebastián
  • Pérez-González, Alfonso María
  • Alfonso Laguna, Carlos De|||0000-0002-2378-021X
  • Blanquer Espert, Ignacio|||0000-0003-1692-8922
  • Moltó, Germán|||0000-0002-8049-253X
[EN] This paper introduces a platform to support serverless computing for scalable event-driven data processing that features a multi-level elasticity approach combined with virtualization of GPUs. The platform supports the execution of applications based on Docker containers in response to file uploads to a data storage in order to perform the data processing in parallel. This is managed by an elastic Kubernetes cluster whose size automatically grows and shrinks depending on the number of files to be processed. To accelerate the processing time of each file, several approaches involving virtualized access to GPUs, either locally or remote, have been evaluated. A use case that involves the inference based on deep learning techniques on transtoracic echocardiography imaging has been carried out to assess the benefits and limitations of the platform. The results indicate that the combination of serverless computing and GPU virtualization introduce an efficient and cost-effective event-driven accelerated computing approach that can be applied for a wide variety of scientific applications., The work presented in this article has been partially funded by a research grant from the regional government of the Comunitat Valenciana (Spain), co-funded by the European Union ERDF funds (European Regional Development Fund) of the Comunitat Valenciana 2014-2020, with reference IDIFEDER/2018/032 (High-Performance Algorithms for the Modeling, Simulation and early Detection of diseases in Personalized Medicine). The authors would also like to thank the Spanish "Ministerio de Economia, Industria y Competitividad" for the project "BigCLOE" with reference number TIN2016-79951-R and the project ATMOSPHERE, funded jointly by the European Commission under the Cooperation Programme, Horizon 2020 grant agreement No 777154 and the Brazilian Ministerio de Ciencia, Tecnologia e Inovacao (MCI-I), number 51119. D.M.N would like to thank the "Generalitat Valenciana, Spain" for the grant GrisoliaP/2017/071.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publication Artículo científico (JournalArticle)

APRICOT: Advanced Platform for Reproducible Infrastructures in the Cloud via Open Tools

  • Giménez-Alventosa, Vicent|||0000-0003-1646-6094
  • Segrelles Quilis, José Damián|||0000-0001-5698-7965
  • Moltó, Germán|||0000-0002-8049-253X
  • Roca-Sogorb, Mar
[EN] Background Scientific publications are meant to exchange knowledge among researchers but the inability to properly reproduce computational experiments limits the quality of scientific research. Furthermore, bibliography shows that irreproducible preclinical research exceeds 50%, which produces a huge waste of resources on nonprofitable research at Life Sciences field. As a consequence, scientific reproducibility is being fostered to promote Open Science through open databases and software tools that are typically deployed on existing computational resources. However, some computational experiments require complex virtual infrastructures, such as elastic clusters of PCs, that can be dynamically provided from multiple clouds. Obtaining these infrastructures requires not only an infrastructure provider, but also advanced knowledge in the cloud computing field. Objectives The main aim of this paper is to improve reproducibility in life sciences to produce better and more cost-effective research. For that purpose, our intention is to simplify the infrastructure usage and deployment for researchers. Methods This paper introduces Advanced Platform for Reproducible Infrastructures in the Cloud via Open Tools (APRICOT), an open source extension for Jupyter to deploy deterministic virtual infrastructures across multiclouds for reproducible scientific computational experiments. To exemplify its utilization and how APRICOT can improve the reproduction of experiments with complex computation requirements, two examples in the field of life sciences are provided. All requirements to reproduce both experiments are disclosed within APRICOT and, therefore, can be reproduced by the users. Results To show the capabilities of APRICOT, we have processed a real magnetic resonance image to accurately characterize a prostate cancer using a Message Passing Interface cluster deployed automatically with APRICOT. In addition, the second example shows how APRICOT scales the deployed infrastructure, according to the workload, using a batch cluster. This example consists of a multiparametric study of a positron emission tomography image reconstruction. Conclusion APRICOT's benefits are the integration of specific infrastructure deployment, the management and usage for Open Science, making experiments that involve specific computational infrastructures reproducible. All the experiment steps and details can be documented at the same Jupyter notebook which includes infrastructure specifications, data storage, experimentation execution, results gathering, and infrastructure termination. Thus, distributing the experimentation notebook and needed data should be enough to reproduce the experiment., This study was supported by the program "Ayudas para la contratación de personal investigador en formación de carácter predoctoral, programa VALi+d" under grant number ACIF/2018/148 from the Conselleria d'Educació of the Generalitat Valenciana and the "Fondo Social Europeo" (FSE). The authors would like to thank the Spanish "Ministerio de Economía, Industria y Competitividad" for the project "BigCLOE" with reference number TIN2016-79951-R and the European Commission, Horizon 2020 grant agreement No 826494 (PRIMAGE). The MRI prostate study case used in this article has been retrospectively collected from a project of prostate MRI biomarkers validation.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publication Artículo científico (JournalArticle)

GPU-Enabled Serverless Workflows for Efficient Multimedia Processing

  • Risco, Sebastián
  • Moltó, Germán|||0000-0002-8049-253X
[EN] Serverless computing has introduced scalable event-driven processing in Cloud infrastructures. However, it is not trivial for multimedia processing to benefit from the elastic capabilities featured by serverless applications. To this aim, this paper introduces the evolution of a framework to support the execution of customized runtime environments in AWS Lambda in order to accommodate workloads that do not satisfy its strict computational requirements: increased execution times and the ability to use GPU-based resources. This has been achieved through the integration of AWS Batch, a managed service to deploy virtual elastic clusters for the execution of containerized jobs. In addition, a Functions Definition Language (FDL) is introduced for the description of data-driven workflows of functions. These workflows can simultaneously leverage both AWS Lambda for the highly-scalable execution of short jobs and AWS Batch, for the execution of compute-intensive jobs that can profit from GPU-based computing. To assess the developed open-source framework, we executed a case study for efficient serverless video processing. The workflow automatically generates subtitles based on the audio and applies GPU-based object recognition to the video frames, thus simultaneously harnessing different computing services. This allows for the creation of cost-effective highly-parallel scale-to-zero serverless workflows in AWS., This research was funded by the Spanish "Ministerio de Economia, Industria y Competitividad" for the project "BigCLOE" with reference number TIN2016-79951-R
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publication Artículo científico (JournalArticle)

Vertical Elasticity on Marathon and Chronos Mesos frameworks

  • López-Huguet, Sergio
  • Natanael, I.
  • Brito, A.
  • Blanquer Espert, Ignacio|||0000-0003-1692-8922
[EN] Marathon and Chronos are two popular Mesos frameworks that are widely used for deploying fault-tolerant services and periodic batch jobs. Marathon and Chronos provide by design mechanisms for horizontal elasticity, scaling up and down the number of job and service instances. Horizontal elasticity is appropriate when the problems that are solved are inherently parallel. However, when the problem cannot benefit from an increase of the amount of resources, vertical elasticity must be considered. This work implements on top of Marathon and Chronos Mesos frameworks, a mechanism to vary the resources associated to an executor dynamically, according to its progress and considering specific Quality of Service (QoS). The mechanism developed provides a wrapper executable and a service that takes the decision of increasing or decreasing the resources allocated to different Chronos iterations or a long-living Marathon application. The mechanism makes use of checkpointing techniques to preserve the execution of Marathon applications and leverages OpenStack Monasca for the monitoring. \footnote{The work in this article has been funded by projects BIGCLOE and EUBra-BIGSEA, BIGLOE is funded by the Spanish ``Ministerio de Econom\'ia, Industria y Competitividad" with reference number TIN2016-79951-R and EUBra-BIGSEA is funded jointly by the European Commission under the Cooperation Programme, Horizon 2020 grant agreement No 690116 and the Brazilian Ministério de Ciência, Tecnologia e Inovação (MCTI)., The work in this article has been funded by projects BIGCLOE and EUBra BIGSEA, BIGLOE is funded by the Spanish "Ministerio de Economia, Industria y Competitividad" with reference number TIN2016-79951-R and EUBra-BIGSEA is funded jointly by the European Commission under the Cooperation Programme, Horizon 2020 grant agreement No 690116 and the Brazilian Ministerio de Ciencia, Tecnologia e Inovacao (MCTI).
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publication Artículo científico (JournalArticle)

Deployment of Elastic Virtual Hybrid Clusters Across Cloud Sites

  • Caballer Fernández, Miguel|||0000-0001-9393-3077
  • Moltó, Germán|||0000-0002-8049-253X
  • Antonacci, Marica
  • Sustr, Zdenek
  • Perniola, Michele
[EN] Virtual clusters are widely used computing platforms than can be deployed in multiple cloud platforms. The ability to dynamically grow and shrink the number of nodes has paved the way for customised elastic computing both for High Performance Computing and High Throughput Computing workloads. However, elasticity is typically restricted to a single cloud site, thus hindering the ability to provision computational resources from multiple geographically distributed cloud sites. To this aim, this paper introduces an architecture of open-source components that coherently deploy a virtual elastic cluster across multiple cloud sites to perform large-scale computing. These hybrid virtual elastic clusters are automatically deployed and configured using an Infrastructure as Code (IaC) approach on a distributed hybrid testbed that spans different organizations, including on-premises and public clouds, supporting automated tunneling of communications across the cluster nodes with advanced VPN topologies. The results indicate that cluster-based computing of embarrassingly parallel jobs can benefit from hybrid virtual clusters that aggregate computing resources from multiple cloud back-ends and bring them together into a dedicated, albeit virtual network., The work presented in this article has been partially funded by project DEEP Hybrid-DataCloud (grant agreement No 777435). GM and MC would also like to thank the Spanish "Ministerio de Economia, Industria y Competitividad" for the project "BigCLOE" with reference number TIN2016-79951-R. Computational resources at CESNET, used in the real-world use case, were supplied by the project "e-Infrastruktura CZ" (e-INFRA LM2018140) provided within the program Projects of Large Research, Development and Innovations Infrastructures
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publication Artículo científico (JournalArticle)

Toward Bio-Inspired Auto-Scaling Algorithms: An Elasticity Approach for Container Orchestration Platforms

  • Herrera, Jose
  • Moltó, Germán|||0000-0002-8049-253X
[EN] The wide adoption of microservices architectures has introduced an unprecedented granularisation of computing that requires the coordinated execution of multiple containers with diverse lifetimes and with potentially different auto-scaling requirements. These applications are managed by means of container orchestration platforms and existing centralised approaches for auto-scaling face challenges when used for the timely adaptation of the elasticity required for the different application components. This paper studies the impact of integrating bio-inspired approaches for dynamic distributed auto-scaling on container orchestration platforms. With a focus on running self-managed containers, we compare alternative configuration options for the container life cycle. The performance of the proposed models is validated through simulations subjected to both synthetic and real-world workloads. Also, multiple scaling options are assessed with the purpose of identifying exceptional cases and improvement areas. Furthermore, a nontraditional metric for scaling measurement is introduced to substitute classic analytical approaches. We found out connections for two related worlds (biological systems and software container elasticity procedures) and we open a new research area in software containers that features potential self-guided container elasticity activities., This work was supported by the Ministerio de Economía, Industria y Competitividad, Spanish Government, for the Project BigCLOE under Grant TIN2016-79951-R
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publication Artículo científico (JournalArticle)

Virtualized Computational Environments on the cloud to foster group skills through PBL: A case study in architecture

  • Segrelles Quilis, José Damián|||0000-0001-5698-7965
  • Martínez Antón, Alicia|||0000-0002-8897-8809
  • Castilla-Cabanes, Nuria|||0000-0002-9993-828X
  • Moltó, Germán|||0000-0002-8049-253X
The ODISEA platform provides Virtualized Computational Environments (VCEs) on cloud providers as the computational infrastructure to support educational activities. A VCE consists of a collection of one or more Virtual Machines (VMs) to which the students connect from their own computers. In this paper a case study is presented in the architecture domain where a PBL activity is carried out in working groups. The study involves 293 students organized in 28 pilot groups that use customized VCEs created and deployed through the ODISEA platform on a Cloud, and 30 traditional groups that use a LMS platform. The VCE provides the software, hardware and specific configuration to ease the interrelation and cooperative work between the working groups, enhancing the process tracking and feedback gathering as well as providing a better organization of the teaching material. The results demonstrate that the VCE allows to improve the cooperative work, improving the final marks in the PBL developed by the pilot working groups. Also, an economical study is presented, highlighting the economic benefits of the Cloud with respect to raditional physical laboratories of PCs., The authors wish to thank the financial support received from Vicerrectorado de Estudios, Calidad y Acreditacion of the Universitat Politecnica de Valencia (UPV) to develop the PIME project "Entomos Virtuales Computacionales para la Evaluation de Competencias Transversales en la Nube", with reference A04 and to the Ministerio de Economia, Industria y Cornpetitividad for the project BigCLOE (TIN2016-79951-R). GM would like to thank AWS for the AWS Educate program that provides the credits required to support the educational activities.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publication Artículo científico (JournalArticle)

Serverless computing for container-based architectures

  • Pérez-González, Alfonso María
  • Moltó, Germán|||0000-0002-8049-253X
  • Caballer Fernández, Miguel|||0000-0001-9393-3077
  • Calatrava Arroyo, Amanda|||0000-0002-9018-9171
[EN] New architectural patterns (e.g. microservices), the massive adoption of Linux contain- ers (e.g. Docker containers), and improvements in key features of Cloud computing such as auto-scaling, have helped developers to decouple complex and monolithic sys- tems into smaller stateless services. In turn, Cloud providers have introduced serverless computing, where applications can be defined as a workflow of event-triggered functions. However, serverless services, such as AWS Lambda, impose serious restrictions for these applications (e.g. using a predefined set of programming languages or difficulting the installation and deployment of external libraries). This paper addresses such issues by introducing a framework and a methodology to create Serverless Container-aware AR- chitectures (SCAR). The SCAR framework can be used to create highly-parallel event- driven serverless applications that run on customized runtime environments defined as Docker images on top of AWS Lambda. This paper describes the architecture of SCAR together with the cache-based optimizations applied to minimize cost, exemplified on a massive image processing use case. The results show that, by means of SCAR, AWS Lambda becomes a convenient platform for High Throughput Computing, specially for highly-parallel bursty workloads of short stateless jobs., The authors would like to thank the Spanish "Ministerio de Economia, Industria y Competitividad" for the project "BigCLOE" under grant reference TIN2016-79951-R. The authors would also like to thank Jorge Gomes from LIP for the development of the udocker tool.
RUA. Repositorio Institucional de la Universidad de Alicante
Publication Otros (Other)

Herramienta web para el seguimiento automatizado de actividades educativas prácticas en la nube

  • Moltó, Germán
  • Naranjo, Diana M.
  • Prieto, José Ramón
Esta contribución presenta un recurso docente destinado a la recopilación y análisis automatizado de evidencias generadas en actividades educativas prácticas en la nube, ejemplificado para Amazon Web Services (AWS). Incluye una arquitectura que posibilita la captura de datos y procesado utilizando servicios en la nube, así como un panel web de control educativo donde profesor y alumnos pueden consultar información relativa al uso de los diferentes servicios de AWS. Además, permite la autorregulación de los estudiantes proporcionándoles información sobre el porcentaje de progreso de cada sesión de laboratorio y las acciones que faltan para culminar cada práctica. La herramienta permite extraer automáticamente analíticas de aprendizaje en base a dichos datos, que permiten evidenciar el grado de desarrollo de una práctica para un alumno concreto. También obtiene información agregada sobre el uso de recursos en AWS de diferentes alumnos a lo largo de un curso académico. La herramienta, que se ha liberado a la comunidad como código abierto, se está utilizando en producción en tres másteres y un curso online de formación en AWS, y puede ser aplicada en entornos educativos que involucren el uso de este proveedor Cloud.