Pasar al contenido principal

COMPLEX SYSTEMS APPROACHES FOR NOVEL MACHINE LEARNING PARADIGMS

PRE2022-104148

•
Nombre agencia financiadora Agencia Estatal de Investigación
Acrónimo agencia financiadora AEI
Programa Programa Estatal para Desarrollar, Atraer y Retener Talento
Subprograma Subprograma Estatal de Formación
Convocatoria Contratos Predoctorales
Año convocatoria 2022
Plan Estatal 2021-2023 Plan Estatal de Investigación Científica y Técnica y de Innovación
Centro beneficiario UNIVERSIDAD DE LAS ISLAS BALEARES

Resultados relacionados

Resultados totales (Incluyendo duplicados): 1
Encontrada(s) 1 página(s)

DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

Leveraging chaotic transients in the training of artificial neural networks

  • Jiménez-González, Pedro
  • Soriano, Miguel C.
  • Lacasa, Lucas
Traditional algorithms to optimize artificial neural networks when confronted with a supervised learning task are usually exploitation-type relaxational dynamics such as gradient descent (GD). Here, we explore the dynamics of the neural network trajectory along training for unconventionally large learning rates. We show that for a region of values of the learning rate, the GD optimization shifts away from purely exploitationlike algorithm into a regime of exploration-exploitation balance, as the neural network is still capable of learning but the trajectory shows sensitive dependence on initial conditions—as characterized by positive network maximum Lyapunov exponent. Interestingly, the characteristic training time required to reach an acceptable accuracy in the test set reaches a minimum precisely in such learning rate region, further suggesting that one can accelerate the training of artificial neural networks by locating at the onset of chaos. Our results—initially illustrated for the MNIST classification task—qualitatively hold for a range of supervised learning tasks, learning architectures (including both shallow and deep multilayer perceptrons and convolutional neural networks) and other hyperparameters (different activation functions and weight regularization), and showcase the emergent, constructive role of transient chaotic dynamics in the training of artificial neural networks., P.J. acknowledges funding from Maria de Maeztu (MdM) Seal of Excellence (CEX2021-001164-M) via the FPI programme (Grant No. PRE2022-104148), funded by the MICIU/AEI/10.13039/501100011033. L.L. acknowledges partial support from project CSxAI (PID2024-157526NB-I00) funded by MICIU/AEI/10.13039/501100011033/FEDER, UE. M.C.S. and L.L. acknowledge partial support from a Maria de Maeztu grant (CEX2021-001164-M) funded by the MICIU/AEI/10.13039/501100011033 and from the European Commission Chips Joint Undertaking Project No. 101194363 (NEHIL)., With funding from the Spanish government through the "Maria de Maeztu Centre of Excellence" accreditation (CEX2021-001164-M)., Peer reviewed