RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publication
Artículo científico (JournalArticle)
Q-Learning for Online PID Controller Tuning in Continuous Dynamic Systems: An Interpretable Framework for Exploring Multi-Agent Systems
[EN] This study proposes a discrete multi-agent Q-learning framework for online tuning of PID controllers in continuous dynamic systems with limited observability. Each PID gain (Kp, Ki, Kd) is adjusted by an independent learning agent operating in a discrete state space defined by its own gain value. At each decision step, agents choose among three actions: decrease, keep, or increase the gain, and they act simultaneously at fixed decision intervals. This design helps preserve quasi-stationary conditions from the agents’ perspective and supports convergence. Coordination is achieved through a shared cumulative global reward that combines system-performance terms with time and control-effort penalties, as well as stability incentives, guiding exploration toward control objectives. Implemented in Python, the framework is validated on two nonlinear control problems: a water-tank system and an inverted pendulum (cart-pole). The agents reach initial convergence after approximately 300 and 500 episodes, respectively, and obtain overall success rates of 49.6% and 46.2% over 5,000 training episodes. Results show sustained learning toward effective PID parameter settings capable of stabilizing both systems without requiring explicit dynamic models. These findings support the feasibility of a low-complexity, discrete reinforcement-learning approach for online adaptive PID tuning, producing interpretable and reproducible control policies and providing a foundation for future hybrid schemes combining classical control theory with reinforcement-learning agents., This research was funded by Vicerrectorado de Investigación de la Universitat Politècnica
de València (PAID-11-24) and supported by Agencia Nacional de Investigación de Chile ANID
BECAS/DOCTORADO EN EL EXTRANJERO 72230436-2023.