DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication
Artículo científico (JournalArticle)
Effective theory of collective deep learning
Unraveling the emergence of collective learning in systems of coupled artificial neural networks points to broader implications for physics, machine learning, neuroscience, and society. Here we introduce a minimal model of interacting deep neural nets that condenses several recent decentralized algorithms by considering a competition between two terms: the local learning dynamics in the parameters of each neural network unit, and a diffusive coupling among units that tends to homogenize the parameters of the ensemble. We derive an effective theory for linear networks to show that the coarse-grained behavior of our system is equivalent to a deformed Ginzburg-Landau model with quenched disorder. This framework predicts depth-dependent disorder-order-disorder phase transitions in the parameters' solutions that reveal a depth-delayed onset of a collective learning phase and a low-rank microscopic learning path. We validate the theory in coupled ensembles of realistic neural networks trained on MNIST and CIFAR-10 datasets under privacy constraints. Interestingly, experiments confirm that individual networks-trained on private data-can fully generalize to unseen data classes when the collective learning phase emerges. Our work elucidates the physics of collective learning and contributes to the mechanistic interpretability of deep learning in decentralized settings., This work has been funded by the Spanish State Research Agency, through projects DYNDEEP, MISLAND and a María de Maeztu project (EUR2021- 122007, PID2020-114324GB-C22 and CEX2021-001164-M, respectively), funded by the MICIU/AEI/10.13039/501100011033. The authors also acknowledge the support from the European Commission Chips Joint Undertaking Project No. 101194363 (NEHIL)., With funding from the Spanish government through the "Severo Ochoa Centre of Excellence" accreditation (CEX2021-001164-M)., Peer reviewed
Project:
EC / HE 101194363
,
AEI / Plan Estatal de Investigación Científica y Técnica y de Innovación 2021-2023 EUR2021-122007
,
AEI / Plan Estatal de Investigación Científica y Técnica y de Innovación 2017-2020 CEX2021-001164-M
,
AEI PID2020-114324GB-C22
DOI: http://hdl.handle.net/10261/380842, https://api.elsevier.com/content/abstract/scopus_id/85210321018
HANDLE: http://hdl.handle.net/10261/380842, https://api.elsevier.com/content/abstract/scopus_id/85210321018