Pasar al contenido principal

Resultados relacionados

Resultados totales (Incluyendo duplicados): 4
Encontrada(s) 1 página(s)

UPCommons. Portal del coneixement obert de la UPC
Publicación Contribución de congreso (ConferenceOutput)

An oracle for guiding large-scale model/hybrid parallel training of convolutional neural networks

  • Njoroge Kahira, Albert|||0000-0002-1138-0577
  • Nguyen, Truong Thao
  • Bautista Gomez, Leonardo|||0000-0002-0814-5779
  • Takano, Ryousei
  • Badia Sala, Rosa Maria|||0000-0003-2941-5499
  • Wahib, Mohamed
Deep Neural Network (DNN) frameworks use distributed training to enable faster time to convergence and alleviate memory capacity limitations when training large models and/or using high dimension inputs. With the steady increase in datasets and model sizes, model/hybrid parallelism is deemed to have an important role in the future of distributed training of DNNs. We analyze the compute, communication, and memory requirements of Convolutional Neural Networks (CNNs) to understand the trade-offs between different parallelism approaches on performance and scalability. We leverage our model-driven analysis to be the basis for an oracle utility which can help in detecting the limitations and bottlenecks of different parallelism approaches at scale. We evaluate the oracle on six parallelization strategies, with four CNN models and multiple datasets (2D and 3D), on up to 1024 GPUs. The results demonstrate that the oracle has an average accuracy of about 86.74% when compared to empirical results, and as high as 97.57% for data parallelism., The project that gave rise to these results received the support of a fellowship from the "la Caixa" Foundation (ID 100010434). The fellowship code is LCF/BQ/DI17/11620059. This project has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No. 713673. The Eurolab4HPC project has received funding from the European Union Horizon 2020 Framework Programme (H2020-EU.1.2.2. - FET Proactive) under grant agreement number 800962. This work was supported by JST, ACT-X Grant Number JPMJAX190C, Japan; by JST, PRESTO Grant Number JPMJPR20MA, Japan., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Estudio, informe, memoria (Report)

Towards resilient EU HPC systems: A blueprint

  • Radojković, Petar
  • Marazakis, Manolis
  • Carpenter, Paul Matthew
  • Jeyapaul, Reiley
  • Gizopoulos, Dimitris
  • Schulz, Martin
  • Armejach Sanosa, Adrià|||0000-0003-2869-668X
  • Ayguadé Parra, Eduard|||0000-0002-5146-103X
  • Canal Corretger, Ramon|||0000-0003-4542-204X
  • Moretó Planas, Miquel|||0000-0002-9848-8758
  • Salami, Behzad
  • Unsal, Osman Sabri
This document aims to spearhead a Europe-wide discussion on HPC system resilience and to help the European HPC community define best practices for resilience. We analyse a wide range of state-of-the-art resilience mechanisms and recommend the most effective approaches to employ in large-scale HPC systems. Our guidelines will be useful in the allocation of available resources, as well as guiding researchers and research funding towards the enhancement of resilience approaches with the highest priority and utility. Although our work is focused on the needs of next generation HPC systems in Europe, the principles and evaluations are applicable globally., This work has received funding from the European Union’s Horizon 2020 research and innovation programme under the projects ECOSCALE (grant agreement No 671632), EPI (grant agreement No 826647), EuroEXA (grant agreement No 754337), Eurolab4HPC (grant agreement No 800962), EVOLVE (grant agreement No 825061), EXA2PRO (grant agreement No 801015), ExaNest (grant agreement No 671553), ExaNoDe (grant agreement No 671578), EXDCI-2 (grant agreement No 800957), LEGaTO (grant agreement No 780681), MB2020 (grant agreement No 779877), RECIPE (grant agreement No 801137) and SDK4ED (grant agreement No 780572). The work was also supported by the European Commission’s Seventh Framework Programme under the projects CLERECO (grant agreement No 611404), the NCSA-Inria-ANL-BSC-JSCRiken-UTK Joint-Laboratory for Extreme Scale Computing – JLESC (https://jlesc.github.io/), OMPI-X project (No ECP-2.3.1.17) and the Spanish Government through Severo Ochoa programme (SEV-2015-0493). This work was sponsored in part by the U.S. Department of Energy's Office of Advanced Scientific Computing Research, program managers Robinson Pino and Lucy Nowell. This manuscript has been authored by UT-Battelle, LLC under Contract No DE-AC05-00OR22725 with the U.S. Department of Energy.
Recercat. Dipósit de la Recerca de Catalunya
Publicación

Towards resilient EU HPC systems: A blueprint

  • Radojković, Petar
  • Marazakis, Manolis
  • Carpenter, Paul Matthew
  • Jeyapaul, Reiley
  • Gizopoulos, Dimitris
  • Schulz, Martin
  • Armejach Sanosa, Adrià
  • Ayguadé Parra, Eduard
  • Canal Corretger, Ramon
  • Moretó Planas, Miquel
  • Salami, Behzad
  • Unsal, Osman Sabri
This document aims to spearhead a Europe-wide discussion on HPC system resilience and to help the European HPC community define best practices for resilience. We analyse a wide range of state-of-the-art resilience mechanisms and recommend the most effective approaches to employ in large-scale HPC systems. Our guidelines will be useful in the allocation of available resources, as well as guiding researchers and research funding towards the enhancement of resilience approaches with the highest priority and utility. Although our work is focused on the needs of next generation HPC systems in Europe, the principles and evaluations are applicable globally., This work has received funding from the European Union’s Horizon 2020 research and innovation programme under the projects ECOSCALE (grant agreement No 671632), EPI (grant agreement No 826647), EuroEXA (grant agreement No 754337), Eurolab4HPC (grant agreement No 800962), EVOLVE (grant agreement No 825061), EXA2PRO (grant agreement No 801015), ExaNest (grant agreement No 671553), ExaNoDe (grant agreement No 671578), EXDCI-2 (grant agreement No 800957), LEGaTO (grant agreement No 780681), MB2020 (grant agreement No 779877), RECIPE (grant agreement No 801137) and SDK4ED (grant agreement No 780572). The work was also supported by the European Commission’s Seventh Framework Programme under the projects CLERECO (grant agreement No 611404), the NCSA-Inria-ANL-BSC-JSCRiken-UTK Joint-Laboratory for Extreme Scale Computing – JLESC (https://jlesc.github.io/), OMPI-X project (No ECP-2.3.1.17) and the Spanish Government through Severo Ochoa programme (SEV-2015-0493). This work was sponsored in part by the U.S. Department of Energy's Office of Advanced Scientific Computing Research, program managers Robinson Pino and Lucy Nowell. This manuscript has been authored by UT-Battelle, LLC under Contract No DE-AC05-00OR22725 with the U.S. Department of Energy., Preprint
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

An oracle for guiding large-scale model/hybrid parallel training of convolutional neural networks

  • Njoroge Kahira, Albert
  • Nguyen, Truong Thao
  • Bautista Gomez, Leonardo
  • Takano, Ryousei
  • Badia Sala, Rosa Maria
  • Wahib, Mohamed
Deep Neural Network (DNN) frameworks use distributed training to enable faster time to convergence and alleviate memory capacity limitations when training large models and/or using high dimension inputs. With the steady increase in datasets and model sizes, model/hybrid parallelism is deemed to have an important role in the future of distributed training of DNNs. We analyze the compute, communication, and memory requirements of Convolutional Neural Networks (CNNs) to understand the trade-offs between different parallelism approaches on performance and scalability. We leverage our model-driven analysis to be the basis for an oracle utility which can help in detecting the limitations and bottlenecks of different parallelism approaches at scale. We evaluate the oracle on six parallelization strategies, with four CNN models and multiple datasets (2D and 3D), on up to 1024 GPUs. The results demonstrate that the oracle has an average accuracy of about 86.74% when compared to empirical results, and as high as 97.57% for data parallelism., The project that gave rise to these results received the support of a fellowship from the "la Caixa" Foundation (ID 100010434). The fellowship code is LCF/BQ/DI17/11620059. This project has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No. 713673. The Eurolab4HPC project has received funding from the European Union Horizon 2020 Framework Programme (H2020-EU.1.2.2. - FET Proactive) under grant agreement number 800962. This work was supported by JST, ACT-X Grant Number JPMJAX190C, Japan; by JST, PRESTO Grant Number JPMJPR20MA, Japan., Peer Reviewed, Postprint (author's final draft)