Skip to main content

ANALISIS DE ESTRATEGIAS PARA INCORPORAR HUMANOS AL PROCESO DE APRENDIZAJE AUTOMATICO Y SU APLICACION A LA INVESTIGACION DEL CANCER PANCREATICO

PID2019-107194GB-I00

•
Funding agency name Agencia Estatal de Investigación
Funding agency acronym AEI
Program Programa Estatal de I+D+i Orientado a los Retos de la Sociedad
Call Proyectos de I+D+i (Generación de Conocimiento y Retos Investigación)
Call year 2019
State Plan 2017-2020 Plan Estatal de Investigación Científica y Técnica y de Innovación
Beneficiary centre UNIVERSIDADE DA CORUÑA

Resultados relacionados

Resultados totales (Incluyendo duplicados): 11
Encontrada(s) 1 página(s)

RUC. Repositorio da Universidade da Coruña
Publication Artículo científico (JournalArticle)

Evaluating Explanatory Capabilities of Machine Learning Models in Medical Diagnostics: A Human-in-the-Loop Approach

  • Bobes-Bascarán, José
  • Mosqueira-Rey, Eduardo
  • Fernández-Leal, Ángel
  • Alonso Ríos, David
  • Figueirido Arnoso, Israel
  • Vidal-Ínsua, Yolanda
The results published here are in whole or part based upon data generated by the TCGA Research Network: https://www.cancer.gov/tcga, accessed on 5 January 2024., [Abstract]: This paper presents a broad study on the evaluation of explanatory capabilities of machine learning models, with a focus on Decision Trees, Random Forest, and XGBoost using a pancreatic cancer data set. We use Human-in-the-Loop-related techniques and medical guidelines as a source of domain knowledge to establish the importance of the different features that are relevant to select a pancreatic cancer treatment. These features are not only used as a dimensionality reduction approach for the machine learning models but also as a way to evaluate the explainability capabilities of the different models using agnostic and non-agnostic explainability techniques. To facilitate the interpretation of explanatory results, we propose the use of similarity measures such as the Weighted Jaccard Similarity coefficient. The goal is to select not only the best performing model but also the one that can best explain its conclusions and better aligns with human domain knowledge., This work has been supported by the State Research Agency of the Spanish Government (Grant PID2019-107194GB-I00/AEI/10.13039/501100011033 and Grant PID2023-147422OB-I00) and by the Xunta de Galicia (Grant ED431C 2022/44), supported by the EU European Regional Development Fund (ERDF). We wish to acknowledge support received from the Centro de Investigación de Galicia CITIC, funded by the Xunta de Galicia and ERDF (Grant ED431G 2023/01)., Xunta de Galicia; ED431C 2022/44, Xunta de Galicia; ED431G 2023/01
RUC. Repositorio da Universidade da Coruña
Publication Contribución de congreso (ConferenceOutput)

Evaluating Curriculum Learning Strategies for Pancreatic Cancer Prediction

  • Vázquez Lema, David
  • Hernández-Pereira, Elena
  • Mosqueira-Rey, Eduardo
Presented at: ESANN 2023 - 31st European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning, Bruges, Belgium, 04 - 06 October, 2023, [Abstract]: In this work we applied Curriculum Learning (CL) to evaluate the performance of a machine learning (ML) model for pancreatic cancer prediction. As the dataset required it, we applied missing value imputation and data augmentation techniques. We compare different curriculum configurations in terms of pacing functions and we perform different experiments concluding that CL helps to train the ML model. Nevertheless, not all the configurations behave in the same way, and the best results were obtained when organising the curriculum in increasing levels of difficulty following exponential pacing., This work has been supported by the State Research Agency of the Spanish Government, grant (PID2019-107194GB-I00/AEI/10.13039/501100011033). The authors wish to thank the funding received by Xunta de Galicia (grants ED431C 2022/44) and by CITIC (grant ED431G 2019/01 with European Regional Development Funds- ERDF)., Xunta de Galicia; ED431G 2019/01, Xunta de Galicia; ED431C 2022/44
RUC. Repositorio da Universidade da Coruña
Publication Artículo científico (JournalArticle)

Segmentation, classification and interpretation of breast cancer medical images using human-in-the-loop machine learning

  • Vázquez Lema, David
  • Mosqueira-Rey, Eduardo
  • Hernández-Pereira, Elena
  • Fernández-Lozano, Carlos
  • Seara-Romera, Fernando
  • Pombo-Otero, Jorge
Data availability: The dataset analyzed during the current study is available in the TCGA repository, URL: https://portal.gdc.cancer.gov/projects/ TCGA-BRCA. The images analyzed during the current study is available in the TCIA repository, URL: https://www.cancerimagingarchive.net/collection/ tcga-brca/., This is an Accepted Manuscript. This version of the article has been accepted for publication, after peer review and is subject to Springer Nature’s AM terms of use (https://www.springernature.com/gp/open-science/policies/accepted-manuscript-terms), but is not the Version of Record and does not reflect post-acceptance improvements, or any corrections. The Version of Record is available online at: https://doi.org/10.1007/s00521-024-10799-7 ., [Abstract]: This paper explores the application of Human-in-the-Loop (HITL) strategies in the training of machine learning models in the medical domain. In this case, a doctor-in-the-loop approach is proposed to leverage human expertise in dealing with large and complex data. Specifically, the paper deals with the use of Whole Slide Imaging (WSI) for the analysis and prediction of the genomic subtype of breast cancer. Three different tasks were developed: segmentation of histopathological images, classification of these images regarding the genomic subtype of the cancer, and finally, interpretation of the machine learning results. The involvement of a pathologist helped us to develop a better segmentation model trying to group areas to make it more useful for further diagnosis. Because the classification models underperformed due to the complexity of the problem and insufficient data for certain cancer types, we focus our efforts in using the feedback from the pathologist to enhance model interpretability through a HITL hyperparameter optimization process., Xunta de Galicia; ED431C 2022/44, This work has been supported by the State Research Agency of the Spanish Government (Grants PID2019-107194GBI00/ AEI/10.13039/501100011033 and Project PID2023-147422OB-I00) and by the Xunta de Galicia (Grant ED431C 2022/44), supported by the EU European Regional Development Fund (ERDF). We wish to acknowledge support received from the Centro de Investigación de Galicia CITIC, funded by the Xunta de Galicia and ERDF (Grant ED431C 2022/44). The results published here are in whole or part based upon data generated by the TCGA Research Network: https://www.cancer.gov/tcga .
RUC. Repositorio da Universidade da Coruña
Publication Tesis doctoral (DoctoralThesis)

Human-Centered Machine Learning Applied to Pancreatic Cancer Treatment Selection

  • Bobes-Bascarán, José
[Abstract] Artificial intelligence (AI) faces challenges related to data quality, model interpretability and the need for transparency, especially in critical areas such as medicine. The Human-in-the-loop machine learning (HITL-ML) approach seeks to mitigate these limitations through expert intervention in the different phases of the learning process. This research presents several key contributions. Firstly, it provides a publication comprising an updated review of the state of the art in HITL, offering definitions, classifications and analysis of existing techniques. Secondly, it evidences that human- AI interaction, through methods such as active learning, improves the efficiency of classification models. Thirdly, it proposes strategies for building effective models in contexts with scarce data, highlighting the value of expert knowledge in the training and validation of models. Finally, it evaluates techniques that improve the transparency and explainability of AI in decision making, a crucial aspect in fields such as medicine. Some of these techniques are applied to the prescription of treatment for pancreatic cancer, where the intervention of specialists made it possible to improve accuracy and reliability, highlighting the positive impact of HITL on the development of AI systems., [Resumen] La inteligencia artificial (IA) se enfrenta a desafíos relacionados con la calidad de los datos, la interpretabilidad de los modelos y la necesidad de transparencia, especialmente en ámbitos críticos como la medicina. El enfoque de aprendizaje automático con intervención humana (HITL-ML) busca mitigar estas limitaciones mediante la intervención de humanos en las distintas fases del proceso de aprendizaje. Esta investigación presenta varias contribuciones clave. En primer lugar, proporciona una publicación que comprende una revisión actualizada del estado del arte en HITL, ofreciendo definiciones, clasificaciones y análisis de técnicas existentes. En segundo lugar, evidencia que la interacción humano-IA, a través de métodos como el aprendizaje activo, mejora la eficiencia de los modelos de clasificación. En tercer lugar, propone estrategias para construir modelos efectivos en contextos con escasez de datos, destacando el valor del conocimiento experto en la formación y validación de los modelos. Finalmente, evalúa técnicas que mejoran la transparencia y explicabilidad de la IA en la toma de decisiones, aspecto crucial en ámbitos como la medicina. Se aplican algunas de estas técnicas a la prescripción de tratamiento en cáncer pancreático, donde la intervención de especialistas permitió mejorar la precisión y confiabilidad, resaltando el impacto positivo de HITL en el desarrollo de sistemas de IA., [Resumo] A intelixencia artificial (IA) enfróntase a desafíos relacionados coa calidade dos datos, a interpretabilidad dos modelos e a necesidade de transparencia, especialmente en ámbitos críticos como a medicina. O enfoque de aprendizaxe automática (ML) con intervención humana (HITL) busca mitigar estas limitacións mediante a intervención experta nas distintas fases do proceso de aprendizaxe. Esta investigación presenta varias contribucións crave. En primeiro lugar, proporciona unha revisión actualizada da estado da arte en HITL, ofrecendo definicións, clasificacións e análises de técnicas existentes. En segundo lugar, demostra que a interacción humano-IA, a través de métodos como a aprendizaxe activa, mellora a eficiencia dos modelos de clasificación. En terceiro lugar, propón estratexias para construír modelos efectivos en contextos con escaseza de datos, destacando o valor do coñecemento experto na formación e validación dos modelos. Finalmente, avalía técnicas que melloran a transparencia e explicabilidad da IA na toma de decisións, aspecto crucial en ámbitos como a medicina. Aplícanse algunhas destas técnicas á prescrición de tratamento en cancro pancreático, onde a intervención de especialistas permitiu mellorar a precisión e confiabilidad, resaltando o impacto positivo de HITL no desenvolvemento de sistemas de IA., This work has been partially supported by the State Research Agency of the Spanish Government (Grant PID2019-107194GB-I00/AEI/10.13039/501100011033 and Grant PID2023-147422OB-I00) and by the Xunta de Galicia (Grant ED431C 2022/44), supported by the EU European Regional Development Fund (ERDF). CITIC, as a center accredited for excellence within the Galician University System and a member of the CIGUS Network, receives subsidies from the Department of Education, Science, Universities, and Vocational Training of the Xunta de Galicia. Additionally, it is co-financed by the EU through the FEDER Galicia 2021-27 operational program (Ref. ED431G 2023/01) The results published here are in whole or part based upon data generated by the TCGA Research Network: https://www.cancer.gov/tcga, Xunta de Galicia; ED431C 2022/44, Xunta de Galicia; ED431G 2023/01
RUC. Repositorio da Universidade da Coruña
Publication Contribución de congreso (ConferenceOutput)

Understanding Machine Learning Explainability Models in the context of Pancreatic Cancer Treatment

  • Bobes-Bascarán, José
  • Fernández-Leal, Ángel
  • Mosqueira-Rey, Eduardo
  • Alonso Ríos, David
  • Hernández-Pereira, Elena
  • Moret-Bonillo, Vicente
Cursos e Congresos, C-155, [Abstract] The increasing adoption of artificial intelligent systems at sensitive domains where humans are particularly, such as medicine, has provided the context to deeply explore ways of making machine learning models (ML) understandable for their final users. The success of such systems require the trust of their users, and thus there is a need to design and provide methods to understand the decisions made by such systems. We start from a public Pancreatic Cancer dataset and experiment with different ML models on a diagnosis scenario with the goal to decide whether a patient should be prescribed with a chemotherapy treatment. To validate the diagnosis results we explore different explainability approaches: Decision Tree, Random Forest, and model agnostic ad-hoc models, and compare them against a standard Pancreatic Cancer treatment set of rules. The increasing adoption of artificial intelligent systems at sensitive domains where humans are particularly, such as medicine, has provided the context to deeply explore ways of making machine learning models (ML) understandable for their final users. The success of such systems require the trust of their users, and thus there is a need to design and provide methods to understand the decisions made by such systems. We start from a public Pancreatic Cancer dataset and experiment with different ML models. To validate the diagnostic results we explore different explainability approaches: Decision Tree based approach, Random Forest based approach, and different model agnostic ad-hoc approaches, and we compare them against a standard Pancreatic Cancer treatment set of rules, Xunta de Galcia; ED431G 2019/01, Xunta de Galcia; ED431C 2022/44, This work has been supported by the State Research Agency of the Spanish Government (grant PID2019-107194GB-I00/AEI/10.13039/501100011033) and by the Xunta de Galicia (grant ED431C2022/44), supported in turn by the EU European Regional Development Fund. We wish to acknowledge support received from the Centro de Investigaci ´on de Galicia CITIC, funded by the Xunta de Galicia and the European Regional Development Fund (Galicia 2014-2020 Program; grant ED431G 2019/01)
RUC. Repositorio da Universidade da Coruña
Publication Contribución de congreso (ConferenceOutput)

Human-in-the-Loop Machine Learning for the Treatment of Pancreatic Cancer

  • Mosqueira-Rey, Eduardo
  • Pérez-Sánchez, Alberto
  • Hernández-Pereira, Elena
  • Alonso Ríos, David
  • Bobes-Bascarán, José
  • Fernández-Leal, Ángel
  • Moret-Bonillo, Vicente
  • Vidal-Ínsua, Yolanda
  • Vázquez-Rivera, Francisca
The congress was held in Queensland, Australia. June 18 - 23, 2023, [Abstract]: Human-in-the-Loop Machine Learning (HITL-ML) is a set of techniques that attempt to actively introduce experts into the learning loop of machine learning (ML) models to improve the learning process. In this paper we present a HITLML strategy for the treatment of pancreatic cancer in which a classifier should decide whether a chemotherapy treatment is suitable or not for the patient. The contribution of this work is, first, to demonstrate that involving human experts in the learning process improves the learning capacity of the model; second, to develop a relatively novel Interactive Machine Learning (IML) approach in which unstructured feedback obtained from the experts is used to optimize the synthetic cases generator implemented by a Generative Adversarial Network (GAN). This GAN is used to augment the dataset and to improve the generalization capabilities of the model. Finally, the inclusion of humans in the learning process also poses new challenges, e.g., aspects related to Human-Computer Interaction (HCI), normally irrelevant in ML systems, are now of great importance and can condition the success of a HITL approach. This paper also discusses the approach taken to address these challenges., This work has been supported by the State Research Agency of the Spanish Government, grant (PID2019-107194GB-I00 / AEI / 10.13039/501100011033) and by the Xunta de Galicia, grant (ED431C 2022/44) with the European Union ERDF funds. We wish to acknowledge the support received from the Centro de Investigaci´on de Galicia “CITIC”, funded by Xunta de Galicia and the European Union (European Regional Development Fund - Galicia 2014-2020 Program), by grant ED431G 2019/01., Xunta de Galicia; ED431C 2022/44, Xunta de Galicia; ED431G 2019/01
RUC. Repositorio da Universidade da Coruña
Publication Contribución de congreso (ConferenceOutput)

Gamifying Machine Teaching: Human-in-the-Loop Approach for Diphthong and Hiatus Identification in Spanish Language

  • Mosqueira-Rey, Eduardo
  • Fernández Castaño, Sergio
  • Alonso Ríos, David
  • Vázquez-Cano, Esteban
  • López-Meneses, Eloy
Presented at 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems, KES 2023, Athens 6 - 8 September 2023, [Abstract]: Human-in-the-Loop Machine Learning (HITL-ML) is a set of techniques that attempt to actively involve experts into the learning loop of machine learning (ML) models. One of these techniques is Machine Teaching (MT) which tries to apply techniques that come from the world of didactics within machine learning (ML), such as sorting the dataset according to its difficulty and presenting the cases to the model in incremental levels of complexity. In this work we propose a new twist to MT: since its foundation is to bring didactic techniques to ML, why not use this technique as a didactic method itself? In this case we propose the creation of an ML model for the identification of diphthongs and hiatuses in the Spanish language. The first step is to develop a deep learning model to identify diphthongs and hiatuses using Curriculum Learning (CL) and a sorted dataset that identifies simple and complex cases. The accuracy of this model identifies the upper limit of efficiency that we can obtain by training the model. The next step is to reset the weights of the model but retain its architecture and offer the model to the students for its training. The idea is that students use MT techniques to make the model learn again, but the ultimate goal is that students learn by teaching in an informal and gamified learning environment. The results show how a HITL strategy can make a model learn iteratively to identify diphthongs and hiatuses and a workflow is proposed to include this technique in the classroom., This work has been supported by the Spanish Government (PID2019-107194GB-I00/AEI/10.13039/501100011033) the Xunta de Galicia (ED431C 2022/44), the Centro de Investigación “CITIC” (ED431G 2019/01) with European Union ERDF funds and the “Cátedra de Educación y Tecnologías Emergentes, Gamificación e IA (EduEmer)” of the Universidad Pablo de Olavide., Xunta de Galicia; ED431C 2022/44, Xunta de Galicia; ED431G 2019/01
RUC. Repositorio da Universidade da Coruña
Publication Artículo científico (JournalArticle)

Integrating Iterative Machine Teaching and Active Learning into the Machine Learning Loop

  • Mosqueira-Rey, Eduardo
  • Alonso Ríos, David
  • Baamonde-Lozano, Andrés
[Abstract] Scholars and practitioners are defining new types of interactions between humans and machine learning algorithms that we can group under the umbrella term of Human-in-the-Loop Machine Learning (HITL-ML). This paper is focused on implementing two approaches to this topic—Iterative Machine Teaching (iMT) and Active Learning (AL)—and analyzing how to integrate them in the learning loop. iMT is a variation of Machine Teaching in which a machine acts as a teacher that tries to transfer knowledge to a machine learning model. The focus of the problem in iMT is how to obtain the optimal training set given a machine learning algorithm and a target model. The idea is to learn a target concept with a minimal number of iterations with the smallest dataset. Active Learning, in contrast, is a specialized type of supervised learning in which humans are incorporated in the loop to act as oracles that analyze unlabeled data. AL allows us to achieve greater accuracy with less data and less training. Our proposal to incorporate iMT and AL into the machine learning loop is to use iMT as a technique to obtain the “Minimum Viable Data (MVD)” for training a learning model, that is, a dataset that allows us to increase speed and reduce complexity in the learning process by allowing to build early prototypes. Next, we will use AL to refine this first prototype by adding new data in an iterative and incremental way. We carried out several experiments to test the feasibility of our proposed approach. They show that the algorithms trained with the teachers converge faster than those that have been trained in a conventional way. Also, AL helps the model to avoid getting stuck and to keep improving after the first few iterations. The two approaches investigated in this paper can be considered complementary, as they correspond to different stages in the learning process., This work has been supported by the State Research Agency of the Spanish Government (grant PID2019-107194GB-I00 / AEI / 10.13039/501100011033) and by the Xunta de Galicia (grant ED431C 2018/34) with the European Union ERDF funds. We wish to acknowledge the support received from the Centro de Investigaci ́on de Galicia “CITIC”, funded by Xunta de Galicia and the European Union (European Regional Development Fund- Galicia 2014-2020 Program, grant ED431G 2019/01), Xunta de Galicia; ED431C 2018/34, Xunta de Galicia; ED431G 2019/01
RUC. Repositorio da Universidade da Coruña
Publication Contribución de congreso (ConferenceOutput)

Improving Medical Data Annotation Including Humans in the Machine Learning Loop

  • Bobes-Bascarán, José
  • Mosqueira-Rey, Eduardo
  • Alonso Ríos, David
Presented at the 4th XoveTIC Conference, A Coruña, Spain, 7–8 October 2021, [Abstract] At present, the great majority of Artificial Intelligence (AI) systems require the participation of humans in their development, tuning, and maintenance. Particularly, Machine Learning (ML) systems could greatly benefit from their expertise or knowledge. Thus, there is an increasing interest around how humans interact with those systems to obtain the best performance for both the AI system and the humans involved. Several approaches have been studied and proposed in the literature that can be gathered under the umbrella term of Human-in-the-Loop Machine Learning. The application of those techniques to the health informatics environment could provide a great value on prognosis and diagnosis tasks contributing to develop a better health service for Cancer related diseases., This work has been supported by the State Research Agency of the Spanish Government, grant (PID2019-107194GB-I00/AEI/10.13039/501100011033) and by the Xunta de Galicia, grant (ED431C 2018/34) with the European Union ERDF funds. We wish to acknowledge the support received from the Centro de Investigación de Galicia “CITIC”, funded by Xunta de Galicia and the European Union (European Regional Development Fund- Galicia 2014-2020 Program), by grant ED431G 2019/01, Xunta de Galicia; ED431C 2018/34, Xunta de Galicia; ED431G 2019/01
RUC. Repositorio da Universidade da Coruña
Publication Artículo científico (JournalArticle)

Human-in-the-loop machine learning: a state of the art

  • Mosqueira-Rey, Eduardo
  • Hernández-Pereira, Elena
  • Alonso Ríos, David
  • Bobes-Bascarán, José
  • Fernández-Leal, Ángel
[Abstract]: Researchers are defining new types of interactions between humans and machine learning algorithms generically called human-in-the-loop machine learning. Depending on who is in control of the learning process, we can identify: active learning, in which the system remains in control; interactive machine learning, in which there is a closer interaction between users and learning systems; and machine teaching, where human domain experts have control over the learning process. Aside from control, humans can also be involved in the learning process in other ways. In curriculum learning human domain experts try to impose some structure on the examples presented to improve the learning; in explainable AI the focus is on the ability of the model to explain to humans why a given solution was chosen. This collaboration between AI models and humans should not be limited only to the learning process; if we go further, we can see other terms that arise such as Usable and Useful AI. In this paper we review the state of the art of the techniques involved in the new forms of relationship between humans and ML algorithms. Our contribution is not merely listing the different approaches, but to provide definitions clarifying confusing, varied and sometimes contradictory terms; to elucidate and determine the boundaries between the different methods; and to correlate all the techniques searching for the connections and influences between them., This work has been supported by the State Research Agency of the Spanish Government, Grant (PID2019-107194GB-I00/AEI/10.13039/501100011033) and by the Xunta de Galicia, Grant (ED431C 2022/44) with the European Union ERDF funds. We wish to acknowledge the support received from the Centro de Investigación de Galicia “CITIC”, funded by Xunta de Galicia and the European Union (European Regional Development Fund- Galicia 2014-2020 Program), by Grant ED431G 2019/01., Xunta de Galicia; ED431C 2022/44, Xunta de Galicia; ED431G 2019/01
RUC. Repositorio da Universidade da Coruña
Publication Artículo científico (JournalArticle)

Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach

  • Mosqueira-Rey, Eduardo
  • Hernández-Pereira, Elena
  • Bobes-Bascarán, José
  • Alonso Ríos, David
  • Pérez-Sánchez, Alberto
  • Fernández-Leal, Ángel
  • Moret-Bonillo, Vicente
  • Vidal-Ínsua, Yolanda
  • Vázquez-Rivera, Francisca
[Abstract]: Any machine learning (ML) model is highly dependent on the data it uses for learning, and this is even more important in the case of deep learning models. The problem is a data bottleneck, i.e. the difficulty in obtaining an adequate number of cases and quality data. Another issue is improving the learning process, which can be done by actively introducing experts into the learning loop, in what is known as human-in-the-loop (HITL) ML. We describe an ML model based on a neural network in which HITL techniques were used to resolve the data bottleneck problem for the treatment of pancreatic cancer. We first augmented the dataset using synthetic cases created by a generative adversarial network. We then launched an active learning (AL) process involving human experts as oracles to label both new cases and cases by the network found to be suspect. This AL process was carried out simultaneously with an interactive ML process in which feedback was obtained from humans in order to develop better synthetic cases for each iteration of training. We discuss the challenges involved in including humans in the learning process, especially in relation to human–computer interaction, which is acquiring great importance in building ML models and can condition the success of a HITL approach. This paper also discusses the methodological approach adopted to address these challenges., This work has been supported by the State Research Agency of the Spanish Government (Grant PID2019-107194GB-I00/AEI/10.13039/501100011033) and by the Xunta de Galicia (Grant ED431C 2022/44), supported in turn by the EU European Regional Development Fund. We wish to acknowledge support received from the Centro de Investigación de Galicia CITIC, funded by the Xunta de Galicia and the European Regional Development Fund (Galicia 2014–2020 Program; Grant ED431G 2019/01)., Xunta de Galicia; ED431C 2022/44, Xunta de Galicia; ED431G 2019/01