Academica-e. Repositorio Institucional de la Universidad Pública de Navarra
Publication
Artículo científico (JournalArticle)
Metrics for dataset demographic bias: a case study on facial expression recognition
Demographic biases in source datasets have been shown as one of the causes of unfairness and discrimination in the predictions of Machine Learning models. One of the most prominent types of demographic bias are statistical imbalances in the representation of demographic groups in the datasets. In this paper, we study the measurement of these biases by reviewing the existing metrics, including those that can be borrowed from other disciplines. We develop a taxonomy for the classification of these metrics, providing a practical guide for the selection of appropriate metrics. To illustrate the utility of our framework, and to further understand the practical characteristics of the metrics, we conduct a case study of 20 datasets used in Facial Emotion Recognition (FER), analyzing the biases present in them. Our experimental results show that many metrics are redundant and that a reduced subset of metrics may be sufficient to measure the amount of demographic bias. The paper provides valuable insights for researchers in AI and related fields to mitigate dataset bias and improve the fairness and accuracy of AI models., This work was supported in part by a predoctoral fellowship from the Research Service of the Universidad Pública de Navarra through open access funding, in part by the Spanish MICIN under Grants PID2019-108392GB-I00, PID2020-118014RB-I00, and PID2022-136627NB-I00/AEI/10.13039/501100011033 FEDER, UE, and in part by the Government of Navarre under Grant 0011-1411-2020-000079 - Emotional Films.
Project:
AEI / Plan Estatal de Investigación Científica y Técnica y de Innovación 2017-2020 PID2019-108392GB-I00
,
AEI / Plan Estatal de Investigación Científica y Técnica y de Innovación 2017-2020 PID2020-118014RB-I00
,
AEI / Plan Estatal de Investigación Científica y Técnica y de Innovación 2021-2023 PID2022-136627NB-I00
,
Gobierno de Navarra 0011-1411-2020-000079
View at: https://hdl.handle.net/2454/51887