DIGITAL.CSIC. Repositorio Institucional del CSIC
Publication
Otros (Other)
Dynamical stability and chaos in artificial neural network trajectories along training
29 pages, 18 figures, The process of training an artificial neural network involves iteratively adapting its parameters so as to minimize the error of the network's prediction, when confronted with a learning task. This iterative change can be naturally interpreted as a trajectory in network space -- a time series of networks -- and thus the training algorithm (e.g. gradient descent optimization of a suitable loss function) can be interpreted as a dynamical system in graph space. In order to illustrate this interpretation, here we study the dynamical properties of this process by analyzing through this lens the network trajectories of a shallow neural network, and its evolution through learning a simple classification task. We systematically consider different ranges of the learning rate and explore both the dynamical and orbital stability of the resulting network trajectories, finding hints of regular and chaotic behavior depending on the learning rate regime. Our findings are put in contrast to
common wisdom on convergence properties of neural networks and dynamical systems theory. This work also contributes to the cross-fertilization of ideas between dynamical systems theory, network theory and machine learning., We acknowledge funding from the Spanish Research Agency MCIN/AEI/10.13039/501100011033 via projects DYNDEEP (EUR2021-122007), MISLAND (PID2020-114324GB-C22), INFOLANET (PID2022-139409NB-I00) and the María de Maeztu project CEX2021-001164-M., With funding from the Spanish government through the "Severo Ochoa Centre of Excellence" accreditation (CEX2021-001164-M)., No
Project:
AEI / Plan Estatal de Investigación Científica y Técnica y de Innovación 2021-2023 CEX2021-001164-M
,
AEI / Plan Estatal de Investigación Científica y Técnica y de Innovación 2021-2023 PID2022-139409NB-I00
,
AEI / Plan Estatal de Investigación Científica y Técnica y de Innovación 2017-2020 EUR2021-122007
,
AEI PID2020-114324GB-C22