Skip to main content

Resultados relacionados

Resultados totales (Incluyendo duplicados): 6
Encontrada(s) 1 página(s)

UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Beyond parameter averaging in model aggregation

  • Garcia Recasens, Pol|||0009-0008-5535-0012
  • Torres Viñals, Jordi|||0000-0003-1963-7418
  • Berral García, Josep Lluís|||0000-0003-3037-3580
  • Hauberg, Søren
  • Moreno Muñoz, Pablo
The success of foundation models is strongly linked to scale, which has reinforced the interest in federated learning. With the prohibitive cost of training a large language model (LLM) in mind, little attention has been placed on reusing pre-trained models in collaborative training settings. Self-supervision has also played an important role in this success, but its emphasis has been primarily on data. This paper leverages Bayesian principles to bring self-supervision into the model aggregation toolbox. It introduces self-supervised Fisher merging, a framework that successfully merges models in parameter space without re-visiting data, opening a new door in model reusability. Experimental results build the foundation of our method on tractable linear models, and highlight its potential on aggregating neural networks., This work was supported by a research grant (42062) from VILLUM FONDEN. This project has also received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement 757360). Our work was funded in part by the Novo Nordisk Foundation through the Center for Basic Machine Learning Research in Life Sciences (MLSS) with reference number NNF20OC0062606. This work has been partially financed by the EU-H2020 programme under grant agreement EU-H2020 GA.952179 and grant agreement EU-HORIZON GA.101095717. Also, it has been partially financed by Generalitat de Catalunya (AGAUR) under grant agreement 2021-SGR-00478., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

Distributing deep learning hyperparameter tuning for 3D medical image segmentation

  • Berral García, Josep Lluís|||0000-0003-3037-3580
  • Aranda Llorens, Oriol
  • Domínguez Bermúdez, Juan Luis
  • Torres Viñals, Jordi|||0000-0003-1963-7418
Most research on novel techniques for 3D Medical Image Segmentation (MIS) is currently done using Deep Learning with GPU accelerators. The principal challenge of such technique is that a single input can easily cope computing resources, and require prohibitive amounts of time to be processed. Distribution of deep learning and scalability over computing devices is an actual need for progressing on such research field. Conventional distribution of neural networks consist in “data parallelism”, where data is scattered over resources (e.g., GPUs) to parallelize the training of the model. However, “experiment parallelism” is also an option, where different training processes (i.e., on a hyper-parameter search) are parallelized across resources. While the first option is much more common on 3D image segmentation, the second provides a pipeline design with less dependence among parallelized processes, allowing overhead reduction and more potential scalability. In this work we present a design for distributed deep learning training pipelines, focusing on multi-node and multi-GPU environments, where the two different distribution approaches are deployed and benchmarked. We take as proof of concept the 3D U-Net architecture, using the MSD Brain Tumor Segmentation dataset, a state-of-art problem in medical image segmentation with high computing and space requirements. Using the BSC MareNostrum supercomputer as benchmarking environment, we use TensorFlow and Ray as neural network training and experiment distribution platforms. We evaluate the experiment speed-up when parallelizing, showing the potential for scaling out on GPUs and nodes. Also comparing the different parallelism techniques, showing how experiment distribution leverages better such resources through scaling, e.g. by a speed-up factor from x12 to x14 using 32 GPUs. Finally, we provide the implementation of the design open to the community, and the non-trivial steps and methodology for adapting and deploying a MIS case as the here presented., This work has been partially financed by the European Commission (EU-H2020 INCISIVE GA.952179, and CALLISTO GA.101004152). Also the Spanish Ministry of Science (PID2019- 107255GB-C22/ AEI / 10.13039/501100011033), and Generalitat de Catalunya through the 2017-SGR-1414 project., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publication Contribución de congreso (ConferenceOutput)

HealthMesh: An architectural framework for federated healthcare data management

  • Bisquert Parés, Aniol
  • Hmimou Ham Man, Achraf|||0009-0004-0448-3406
  • Berral García, Josep Lluís|||0000-0003-3037-3580
  • Gutiérrez Torre, Alberto|||0000-0002-5548-3359
  • Romero Moral, Óscar|||0000-0001-6350-8328
Recently, significant milestones have been achieved in the field of healthcare data analysis. However, alongside these accomplishments, substantial data-related challenges have emerged in the domain of big data management. Modern healthcare projects are no more dealing with a single data repository but many heterogeneous ones and must overcome data variety, privacy and governance issues. Yet, current solutions face a privacy-decentralization trade-off. To address this dual challenge, we introduce HealthMesh, a novel layered architectural framework based on the Data Mesh principles, providing a domaindecentralised paradigm. In addition, the framework incorporates a Semantic Data Model which establishes robust governance, enables interoperability and guarantees policy compliance for all the data assets. To demonstrate the capabilities of the proposed approach, we provide an illustrative example inspired by the use case of the INCISIVE project for breast cancer analytics. Overall, this work makes a significant contribution on collecting key challenges, identifying actors and providing a set of components and guidelines for establishing a holistic framework for the complex field of healthcare data management., The work reported here was supported by the EU-H2020 programme under GA.952179 INCISIVE, Horizon Europe Programme under GA.101095717 (SECURED) and GA.101135513 (CYCLOPS), the Spanish Research State Agency (MICINN/AEI, ERDF/FEDER) under GA.MCIN/AEI/10.13039/501100011033/FEDER (UE DALEST), the Spanish Ministerio de Ciencia e Innovación under project PID2020-117191RB-I00 / AEI/10.13039/501100011033 (DOGO4ML), and the Generalitat de Catalunya (AGAUR) 2021-SGR-00478 (CROMAI)., Peer Reviewed
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Beyond parameter averaging in model aggregation

  • Garcia Recasens, Pol
  • Torres Viñals, Jordi
  • Berral García, Josep Lluís
  • Hauberg, Søren
  • Moreno Muñoz, Pablo
The success of foundation models is strongly linked to scale, which has reinforced the interest in federated learning. With the prohibitive cost of training a large language model (LLM) in mind, little attention has been placed on reusing pre-trained models in collaborative training settings. Self-supervision has also played an important role in this success, but its emphasis has been primarily on data. This paper leverages Bayesian principles to bring self-supervision into the model aggregation toolbox. It introduces self-supervised Fisher merging, a framework that successfully merges models in parameter space without re-visiting data, opening a new door in model reusability. Experimental results build the foundation of our method on tractable linear models, and highlight its potential on aggregating neural networks., This work was supported by a research grant (42062) from VILLUM FONDEN. This project has also received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement 757360). Our work was funded in part by the Novo Nordisk Foundation through the Center for Basic Machine Learning Research in Life Sciences (MLSS) with reference number NNF20OC0062606. This work has been partially financed by the EU-H2020 programme under grant agreement EU-H2020 GA.952179 and grant agreement EU-HORIZON GA.101095717. Also, it has been partially financed by Generalitat de Catalunya (AGAUR) under grant agreement 2021-SGR-00478., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

HealthMesh: An architectural framework for federated healthcare data management

  • Bisquert Parés, Aniol
  • Hmimou Ham Man, Achraf
  • Berral García, Josep Lluís
  • Gutiérrez Torre, Alberto
  • Romero Moral, Óscar
Recently, significant milestones have been achieved in the field of healthcare data analysis. However, alongside these accomplishments, substantial data-related challenges have emerged in the domain of big data management. Modern healthcare projects are no more dealing with a single data repository but many heterogeneous ones and must overcome data variety, privacy and governance issues. Yet, current solutions face a privacy-decentralization trade-off. To address this dual challenge, we introduce HealthMesh, a novel layered architectural framework based on the Data Mesh principles, providing a domaindecentralised paradigm. In addition, the framework incorporates a Semantic Data Model which establishes robust governance, enables interoperability and guarantees policy compliance for all the data assets. To demonstrate the capabilities of the proposed approach, we provide an illustrative example inspired by the use case of the INCISIVE project for breast cancer analytics. Overall, this work makes a significant contribution on collecting key challenges, identifying actors and providing a set of components and guidelines for establishing a holistic framework for the complex field of healthcare data management., The work reported here was supported by the EU-H2020 programme under GA.952179 INCISIVE, Horizon Europe Programme under GA.101095717 (SECURED) and GA.101135513 (CYCLOPS), the Spanish Research State Agency (MICINN/AEI, ERDF/FEDER) under GA.MCIN/AEI/10.13039/501100011033/FEDER (UE DALEST), the Spanish Ministerio de Ciencia e Innovación under project PID2020-117191RB-I00 / AEI/10.13039/501100011033 (DOGO4ML), and the Generalitat de Catalunya (AGAUR) 2021-SGR-00478 (CROMAI)., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

Distributing deep learning hyperparameter tuning for 3D medical image segmentation

  • Berral García, Josep Lluís
  • Aranda Llorens, Oriol
  • Domínguez Bermúdez, Juan Luis
  • Torres Viñals, Jordi
Most research on novel techniques for 3D Medical Image Segmentation (MIS) is currently done using Deep Learning with GPU accelerators. The principal challenge of such technique is that a single input can easily cope computing resources, and require prohibitive amounts of time to be processed. Distribution of deep learning and scalability over computing devices is an actual need for progressing on such research field. Conventional distribution of neural networks consist in “data parallelism”, where data is scattered over resources (e.g., GPUs) to parallelize the training of the model. However, “experiment parallelism” is also an option, where different training processes (i.e., on a hyper-parameter search) are parallelized across resources. While the first option is much more common on 3D image segmentation, the second provides a pipeline design with less dependence among parallelized processes, allowing overhead reduction and more potential scalability. In this work we present a design for distributed deep learning training pipelines, focusing on multi-node and multi-GPU environments, where the two different distribution approaches are deployed and benchmarked. We take as proof of concept the 3D U-Net architecture, using the MSD Brain Tumor Segmentation dataset, a state-of-art problem in medical image segmentation with high computing and space requirements. Using the BSC MareNostrum supercomputer as benchmarking environment, we use TensorFlow and Ray as neural network training and experiment distribution platforms. We evaluate the experiment speed-up when parallelizing, showing the potential for scaling out on GPUs and nodes. Also comparing the different parallelism techniques, showing how experiment distribution leverages better such resources through scaling, e.g. by a speed-up factor from x12 to x14 using 32 GPUs. Finally, we provide the implementation of the design open to the community, and the non-trivial steps and methodology for adapting and deploying a MIS case as the here presented., This work has been partially financed by the European Commission (EU-H2020 INCISIVE GA.952179, and CALLISTO GA.101004152). Also the Spanish Ministry of Science (PID2019- 107255GB-C22/ AEI / 10.13039/501100011033), and Generalitat de Catalunya through the 2017-SGR-1414 project., Peer Reviewed, Postprint (author's final draft)