DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación
Artículo científico (JournalArticle)
Multi-objective reinforcement learning for provably incentivising alignment with value systems
This paper addresses the problem of ensuring that autonomous learning agents align with multiple moral values. Specifically, we present the theoretical principles and algorithmic tools necessary for creating an environment where we ensure that the agent learns a behaviour aligned with multiple moral values while striving to achieve its individual objective. To address this value alignment problem, we adopt the Multi-Objective Reinforcement Learning framework and propose a novel algorithm that combines techniques from Multi-Objective Reinforcement Learning and Linear Programming. In addition, we illustrate our value alignment process with an example involving an autonomous vehicle. Here, we demonstrate that the agent learns to behave in alignment with the ethical values of safety, achievement, and comfort, with achievement representing the agent’s individual objective. Such ethical behaviour differs depending on the ordering between values. We also use a synthetic multi-objective environment to evaluate the computational costs of guaranteeing ethical learning as the number of values increases., This research project was made possible through the access granted by the Galician Supercomputing Center (CESGA) to its supercomputing infrastructure. The supercomputer FinisTerrae III and its permanent data storage system have been funded by the Spanish Ministry of Science and Innovation, the Galician Government, and the European Regional Development Fund (ERDF). Funding sources The research presented in this paper was supported by the EU-funded VALAWAI (# 101070930) project, the Spanish-funded EVASAI (# PID2024-158227NB-C31), ANNOTATE-SenSIT (# PID2024-156022OB-C33), VAE (# TED2021-131295B-C31), EMOROBCARE (# IASOMMA2024) and Rhymas (# PID2020-113594RB-100) projects. This work was supported by grant PID2022-136787NB-I00 funded by MCIN/AEI/10.13039/501100011033. It was also funded by GUARDEN (101060693), Fairtrans (PID2021-124361OB-C33), AUTODEMO (SR21-00329), TAILOR (H2020-952215), PrepParticip2.0 ( 24S03545-001), grants 2021 SGR 00313 and 2021 SGR 00754. Maite Lopez-Sanchez belongs to the WAI research group (University of Barcelona) associated unit to CSIC by IIIA. Roxana Rădulescu was partly supported by the Research Foundation Flanders (FWO), grant number 1286223N. This research was partially supported by funding from the Flemish Government under the “Onderzoeksprogramma Artificiële Intelligentie (AI) Vlaanderen” program and by the FWO, grant number G062819N., Peer reviewed
Proyecto:
EC / HE 101070930
,
AEI / Plan Estatal de Investigación Científica y Técnica y de Innovación 2017-2020 PID2024-158227NB-C31
,
AEI / Plan Estatal de Investigación Científica y Técnica y de Innovación 2021-2023 PID2024-156022OB-C33
,
AEI / HE TED2021-131295B-C31
,
AEI / Plan Estatal de Investigación Científica y Técnica y de Innovación 2021-2023 PID2020-113594RB-I00
,
AEI / H2020 PID2022-136787NB-I00
,
EC 101060693
,
AEI PID2021-124361OB-C33
,
EC 952215
DOI: http://hdl.handle.net/10261/431739, https://api.elsevier.com/content/abstract/scopus_id/105024433394
HANDLE: http://hdl.handle.net/10261/431739, https://api.elsevier.com/content/abstract/scopus_id/105024433394