Skip to main content

BUSQUEDA, SELECCION Y ORGANIZACION DE CONTENIDOS PARA NECESIDADES DE INFORMACION RELACIONADAS CON LA SALUD - CONSTRUCCION DE RECURSOS Y PERSONALIZACION

PID2022-137061OB-C21

•
Funding agency name Agencia Estatal de Investigación
Funding agency acronym AEI
Program Programa Estatal para Impulsar la Investigación Científico-Técnica y su Transferencia
Subprogram Subprograma Estatal de Generación de Conocimiento
Call Proyectos de I+D+i (Generación de Conocimiento y Retos Investigación)
Call year 2022
State Plan 2021-2023 Plan Estatal de Investigación Científica y Técnica y de Innovación
Beneficiary centre UNIVERSIDADE DA CORUÑA

Resultados relacionados

Resultados totales (Incluyendo duplicados): 18
Encontrada(s) 1 página(s)

Minerva. Repositorio Institucional de la Universidad de Santiago de Compostela
Publication Artículo científico (JournalArticle)

DepreSym: A Depression Symptom Annotated Corpus and the Role of Large Language Models as Assessors of Psychological Markers

  • Pérez, Anxo
  • Fernández Pichel, Marcos
  • Parapar, Javier
  • Losada Carril, David Enrique
Computational methods for depression detection aim to mine traces of depression from online publications posted by Internet users. However, solutions trained on existing collections exhibit limited generalisation and interpretability. To tackle these issues, recent studies have shown that identifying specific depressive symptoms can lead to more robust and effective models. The eRisk initiative fosters research on this area and has recently proposed a new ranking task focused on developing search methods to find sentences related to depressive symptoms. This search challenge relies on the symptoms specified by the Beck Depression Inventory-II (BDI-II), a questionnaire widely used in clinical practice. It includes symptoms such as sadness, irritability or lack of sleep. Given the input submitted by systems participating in eRisk, we first apply top-k pooling over the systems’ relevance rankings, obtaining a diverse set of sentences. These sentences are judged for relevance, leading to DepreSym, a dataset consisting of 21,580 sentences annotated according to their relevance to the 21 BDI-II symptoms. This dataset serves as a valuable resource for advancing the development of models that monitor depression markers. Due to the complex nature of this relevance annotation, we designed a robust assessment methodology carried out by three expert assessors, including a trained psychologist. As part of this study, we explore the potential of recent Large Language Models (ChatGPT, GPT4 and Vicuna) as assessors in this complex task. We undertake a comprehensive examination of the LLMs’ performance, studying their main limitations and analysing their role as a complement or replacement for human annotators. Finally, we incorporate our dataset into the Benchmarking Information Retrieval (BEIR) framework for a thorough search evaluation. We use state-of-the-art retrieval systems, including lexical, sparse, dense and re-ranking architectures, to gain insights about the dataset’s complexity and identify potential avenues for improvement.
Minerva. Repositorio Institucional de la Universidad de Santiago de Compostela
Publication Artículo científico (JournalArticle)

A study of word embedding models for measuring topic coherence

  • Couto Pintos, Manuel
  • Parapar López, Javier
  • Losada Carril, David Enrique
Topic modeling has emerged as a crucial tool in the field of natural language processing, enabling the automatic discovery of latent structures in large textual corpora. However, determining the quality of the topics remains a significant challenge, particularly in measuring the coherence of the top words of the extracted topics. Early efforts relied on human judgments, but these approaches are resource-intensive. Automated coherence metrics have since been developed. For example, some measures exploit word co-occurrence, while other methods are grounded in distributional semantics (e.g., employing word embeddings). In this study, we thoroughly explore the application of embedded representations to evaluate the quality of topics. While a number of isolated studies have analyzed the role of specific word representation techniques for measuring topic coherence, a complete picture of their effectiveness is still lacking. This work brings together different embedding-based approaches, including Word2Vec, FastText, GloVe, and BERT, which had been studied separately, and extends prior research by incorporating additional models, such as RoBERTa, ALBERT and MPNET. Topic coherence is measured by computing similarity scores between word embeddings, thus obtaining rich semantic associations that traditional measures may overlook. Our analysis demonstrates that these methods are as effective as, and often surpass, classical coherence measures. Our results contribute to a growing body of research advocating for advanced semantic representations as robust alternatives to traditional approaches in evaluating topic model coherence
Minerva. Repositorio Institucional de la Universidad de Santiago de Compostela
Publication Artículo científico (JournalArticle)

Temporal Word Embeddings for Early Detection of Psychological Disorders on Social Media

  • Couto, Manuel
  • Pérez, Anxo
  • Parapar, Javier
  • Losada Carril, David Enrique
Mental health disorders represent a public health challenge, where early detection is critical to mitigating adverse outcomes for individuals and society. The study of language and behavior is a pivotal component in mental health research, and the content from social media platforms serves as a valuable tool for identifying signs of mental health risks. This paper presents a novel framework leveraging temporal word embeddings to capture linguistic changes over time. We specifically aim at at identifying emerging psychological concerns on social media. By adapting temporal word representations, our approach quantifies shifts in language use that may signal mental health risks. To that end, we implement two alternative temporal word embedding models to detect linguistic variations and exploit these variations to train early detection classifiers. Our experiments, conducted on 18 datasets from the eRisk initiative (covering signs of conditions such as depression, anorexia, and self-harm), show that simple models focusing exclusively on temporal word usage patterns achieve competitive performance compared to state-of-the-art systems. Additionally, we perform a word-level analysis to understand the evolution of key terms among positive and control users. These findings underscore the potential of time-sensitive word models in this domain, being a promising avenue for future research in mental health surveillance.
RUC. Repositorio da Universidade da Coruña
Publication Artículo científico (JournalArticle)

How does depression talk on social media? Modeling depression language with relevance-based statistical language models

  • Bao, Eliseo
  • Pérez, Anxo
  • Otero, David
  • Parapar, Javier
[Abstract]: Many individuals with mental health problems turn to the internet and social media for information and support. The text generated on these platforms serves as a valuable resource for identifying mental health risks, driving interdisciplinary research to develop models for mental health analysis and prediction. In this paper, we model depression-related language using relevance-based statistical language models to create lexicons that characterize linguistic patterns associated with depression. We also propose a ranking method that leverages these lexicons to prioritize users exhibiting stronger signs of depressive language on social media. Our models integrate clinical markers from established depression questionnaires, particularly the Beck Depression Inventory-II (BDI-II), enhancing explainability, generalization, and performance. Experiments across multiple social media datasets show that incorporating clinical knowledge improves user ranking and generalizes effectively across platforms. Additionally, we refine existing depression lexicons by applying weights estimated from our models, achieving better performance in generating depression-related queries. A comparative analysis of our models highlights differences in language use between control users and those with depression, aligning with prior psycholinguistic findings. This work advances the understanding of depression-related language through statistical modeling, paving the way for scalable social media interventions to identify at-risk individuals, This work has received support from projects: PLEC2021-007662 (MCIN/AEI/10.13039/501100011033 Ministerio de Ciencia e Innovación, European Union NextGeneration) and PID2022-137061OB-C21 (MCIN/AEI/10.13039/501100011033/, Ministerio de Ciencia e Innovación, by the European Union); Consellería de Educación, Universidade e Formación Profesional, Spain (grant number ED481A-2024-079 and accreditation 2019–2022 ED431G/01 and GRC ED431C 2025/49) and the European Regional Development Fund , which acknowledges the CITIC Research Center, Xunta de Galicia; ED481A-2024-079, Xunta de Galicia; ED431G/01, Xunta de Galicia; GRC ED431C 2025/49
RUC. Repositorio da Universidade da Coruña
Publication Artículo científico (JournalArticle)

Enhancing Automatic Keyphrase Labelling with Text-to-Text Transfer Transformer (T5) Architecture: A Framework for Keyphrase Generation and Filtering

  • Gabín Brenlla, Jorge Juan
  • Ares Brea, Manuel Eduardo
  • Parapar, Javier
[Abstract]: Automatic keyphrase labelling stands for the ability of models to retrieve words or short phrases that adequately describe documents’ content. Previous work has put much effort into exploring extractive techniques to address this task; however, most of these methods cannot produce keyphrases not found in the text. Given this limitation, keyphrase generation approaches have arisen lately. This paper presents a keyphrase generation model based on the Text-to-Text Transfer Transformer (T5) architecture. Having a document’s title and abstract as input, we learn a T5 model to generate keyphrases which adequately define its content. We name this model docT5keywords. We not only perform the classic inference approach, where the output sequence is directly selected as the predicted values, but we also report results from a majority voting approach. In this approach, multiple sequences are generated, and the keyphrases are ranked based on their frequency of occurrence across these sequences. Along with this model, we present a novel keyphrase filtering technique based on the T5 architecture. We train a T5 model to learn whether a given keyphrase is relevant to a document. We devise two evaluation methodologies to prove our model’s capability to filter inadequate keyphrases. First, we perform a binary evaluation where our model has to predict if a keyphrase is relevant for a given document. Second, we filter the predicted keyphrases by keyphrase generation models and check if the evaluation scores are improved. Experimental results show that our best docT5keywords variant yields relative improvements over the strongest baselines ranging from −7.5% to +60.5% across datasets and evaluation metrics; by evaluation type the gains are: present exact-match +23.9% to +60.5%, absent exact-match −7.5% to +60.4%, and partial-match (absent) −0.9% to +10.4%. The proposed filtering technique demonstrates strong performance in eliminating false positives across all datasets, even though identifying true keyphrases remains more challenging., First and third authors acknowledge funding from the Ministry of Science, Innovation and Universities of the Government of Spain (projects PID2022-137061OB-C21, MCIN/AEI/10.13039/501100011033 and PID2025-167749OB-C22), as well as from the Department of Education, Science, Universities, and Vocational Training of the Xunta de Galicia (grant GRC ED431C 2025/49). CITIC, as a center accredited for excellence within the Galician University System and a member of the CIGUS Network, receives subsidies from the Department of Education, Science, Universities, and Vocational Training of the Xunta de Galicia. Additionally, it is co-financed by the EU through the FEDER Galicia 2021-27 operational program (Ref. ED431G 2023/01). Financial support was received from: Project PLEC2021-007662 (MCIN/AEI/10.13039/501100011033, Ministerio de Ciencia e Innovación, European Union NextGenerationEU/PRTR). Project PID2022-137061OB-C21 (MCIN/AEI/10.13039/501100011033, Ministerio de Ciencia e Innovación, ERDF “A way of making Europe” by the European Union). Consellería de Educación, Universidade e Formación Profesional, Spain (accreditation GRC ED431C 2025/49). Scholarship DIN2020-011582 financed by the MCIN/AEI/10.13039/501100011033 (acknowledged by the first author)., Xunta de Galicia; GRC ED431C 2025/49, Xunta de Galicia; ED431G 2023/01
RUC. Repositorio da Universidade da Coruña
Publication Artículo científico (JournalArticle)

Improving Clinical Reliability of LLM Reasoning for Depression Assessment via Structured Generation and GRPO

  • Bao, Eliseo
  • Pérez, Anxo
  • Parapar, Javier
The ReDSM5 dataset used in this study is publicly available at https://huggingface.co/datasets/irlab-udc/redsm5. Code for model training and evaluation is available at https://github.com/IRLab-UDC/grpo-dsm5/., [Abstract]: Objective. Digital mental health screening is increasingly explored through the use of AI systems. Yet, most models provide limited insight into how predictions are derived, restricting clinical trust and patient-centered adoption. We investigate whether Large Language Models (LLMs) can generate structured DSM-5-aligned rationales for depression assessment when trained with Group Relative Policy Optimization (GRPO), a reinforcement learning method that encourages outputs aligned with DSM-5 diagnostic criteria. Methods. We fine-tuned LLMs (1B–27B parameters) on the ReDSM5 dataset, covering 1,484 Reddit posts annotated by a licensed psychologist for DSM-5 depressive symptoms and accompanied by expert rationales. We compared standard Supervised Fine-Tuning (SFT) with GRPO-based optimization using a composite reward integrating symptom classification accuracy and the quality of generated reasoning judged against clinical rationales. Results. GRPO consistently improved symptom detection over SFT, with relative gains exceeding 10% for mid-sized models and weighted F1 scores above 0.60. Models trained to generate structured DSM-5-aligned rationales exhibited additional performance boosts (0.09–0.39 F1), particularly for complex symptoms requiring complex contextual interpretation. Qualitative analysis shows that GRPO encourages models to reference symptom-relevant evidence rather than relying on superficial cues. Conclusions. GRPO enables LLMs to produce clinically grounded explanations while improving classification accuracy, representing a promising direction for interpretable AI in social media-based mental health screening, with potential to support patient-centered applications pending clinical validation., The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This workwas supported by the Department of Education, Science, Universities, and Vocational Training of the Xunta de Galicia (grant ED481A-2024-079); by the Ministry of Science, Innovation and Universities of the Government of Spain (project PID2022-137061OB-C21, MCIN/AEI/10.13039/501100011033); by the Department of Education, Science, Universities, and Vocational Training of the Xunta de Galicia (grant GRC ED431C 2025/49); and by the Xunta de Galicia and the European Union through the FEDER Galicia 2021–2027 OperationalProgramme (Ref. ED431G 2023/01)., Xunta de Galicia; ED481A-2024-079, Xunta de Galicia; ED431C 2025/49, Xunta de Galicia; ED431G 2023/01
RUC. Repositorio da Universidade da Coruña
Publication Artículo científico (JournalArticle)

LabChain: Enabling reproducible and modular scientific experiments in Python

  • Couto Pintos, Manuel
  • Parapar, Javier
  • Losada, David E.
Data availability: The LabChain framework is publicly available at https://github.com/manucouto1/LabChain. The reference implementation for this article is v1.2.1. The mental health detection case study uses publicly available datasets from the eRisk shared tasks: Depression (2017, 2018, 2022), Anorexia (2018, 2019), Self-harm (2020, 2021), and Gambling (2022, 2023). These datasets can be requested from the eRisk organizers at https://erisk.irlab.org/. The complete implementation of the case study, including all pipeline configurations and preprocessing code, is available at https://github.com/manucouto1/Temporal-Word-Embeddings-for-Early-Detection.... No new data were generated or analyzed in support of this research., [Abstract]: Python’s flexibility accelerates research prototyping but frequently results in unmaintainable code and duplicated computational effort. The absence of software engineering practices in academic development leads to fragile experiments where even minor modifications require rerunning expensive computations from scratch. LabChain addresses this through a pipeline-and-filter architecture with hash-based caching that automatically identifies and reuses intermediate results. When evaluating multiple classifiers on the same embeddings, the framework computes embeddings once—regardless of how many classifiers are tested. This automatic reuse extends across research teams: if another researcher applies different models to the same preprocessed data, LabChain detects existing results and eliminates redundant computation. Beyond efficiency, the framework’s modular structure reduces technical debt that obscures experimental logic. Pipelines serialize to JSON for reproducibility and distributed execution across computational clusters. A mental health detection case study demonstrates dual impact: computational savings exceeding 12 hours per task with reduced CO2 emissions, alongside substantial scientific improvements—performance gains up to 192.3% in some tasks. These improvements emerged from clearer experimental organization that exposed a critical preprocessing bug hidden in the original monolithic implementation. LabChain proves that software engineering discipline amplifies scientific discovery., MC and DEL thank the financial support provided by MICIU/AEI/10.13039/501100011033 (PID2022-137061OB-C22, supported by ERDF) and Xunta de Galicia-Consellería de Cultura, Educación, Formación Profesional e Universidades (ED431G 2023/04, ED431C 2022/19, supported by ERDF). JP has received support from project PID2022-137061OB-C21 (MCIU/AEI/10.13039/5011000 11033, Ministerio de Ciencia e Innovación). He also thanks the financial support provided by the Consellería de Educación, Universidade e Formación Profesional, Spain (grant number ED481A-2024–079 and GRC ED431C 2025/49); and the European Regional Development Fund, which supports the CITIC Research Center., Xunta de Galicia; ED481A-2024–079, Xunta de Galicia; ED431C 2025/49, Xunta de Galicia; ED431G 2023/04, Xunta de Galicia; ED431C 2022/19, https://github.com/manucouto1/LabChain, https://erisk.irlab.org/, https://github.com/manucouto1/Temporal-Word-Embeddings-for-Early-Detection-of-Psychological-Disorders-on-Social-Media
RUC. Repositorio da Universidade da Coruña
Publication Artículo científico (JournalArticle)

Comparison of Clustering Algorithms for Knowledge Discovery in Social Media Publications: A Case Study of Mental Health Analysis, Comparación de algoritmos de agrupamiento para el descubrimiento de conocimiento en publicaciones de redes sociales: un caso de estudio en salud mental

  • Couto, Manuel
  • Parapar, Javier
  • Losada, David E.
[Abstract]: In the age of social media, user-generated content is critical for detecting early signs of mental disorders. In this study, we use thematic clustering to analyze the content of the social media platform Reddit. Our primary goal is to use clustering techniques for comprehensive topic discovery, with a focus on identifying common themes among user groups suffering from mental illnesses such as depression, anorexia, gambling addiction, and self-harm. Our findings show that certain clusters are more cohesive, e.g., with a higher proportion of texts indicating depression. Furthermore, we discovered subreddits that are strongly linked to texts from the depressed user group. These findings shed light on how online interactions and subreddit themes may impact users’ mental health, paving the way for future research and more targeted interventions in the field of online mental health., [Resumen]: En la era de las redes sociales, el contenido generado por los usuarios es fundamental para detectar los primeros signos de trastornos mentales. En este estudio utilizamos el agrupamiento de publicaciones por tópicos para analizar el contenido de la plataforma Reddit. Nuestro objetivo primordial es utilizar técnicas de agrupamiento para descubrir temas centrales, con un enfoque en la identificación de temas comunes entre los grupos de usuarios que sufren enfermedades mentales como la depresión, la anorexia, la adicción a los juegos de azar y las autolesiones. Nuestros hallazgos muestran que ciertos clusters son más cohesivos, por ejemplo mostrando una mayor proporción de textos de personas con depresión. Además, hemos descubierto subreddits que estan fuertemente vinculados a textos escritos por usuarios deprimidos. Estos hallazgos arrojan luz sobre cómo las interacciones en línea y los temas que se tratan en los subreddits reflejan aspectos de salud mental, abriendo el camino para futuras investigaciones e intervenciones dirigidas a la prevención de trastornos., This work was supported by project PLEC2021-007662 (MCIN/AEI/10.13039/501100011033, Ministerio de Ciencia e Innovación, Agencia Estatal de Investigación, Plan de Recuperación, Transformación y Resiliencia, Unión Europea-Next Generation EU). The first and third author thank the financial support supplied by the Xunta de Galicia-Consellería de Cultura, Educación, Formación Profesional e Universidade (ED431G 2023/04, ED431C 2022/19) and the European Regional Development Fund, which acknowledges the CiTIUS-Research Center in Intelligent Technologies of the University of Santiago de Compostela as a Research Center of the Galician University System. David E. Losada also thanks the financial support obtained from project SUBV23/00002 (Ministerio de Consumo, Subdirección General de Regulación del Juego) and project PID2022-137061OB-C22 (Ministerio de Ciencia e Innovación, Agencia Estatal de Investigación, Proyectos de Generación de Conocimiento; supported by the European Regional Development Fund). The second author thanks the financial support supplied by the CITIC, as a center accredited for excellence within the Galician University System and a member of the CIGUS Network, funded by the Xunta de Galicia and the EU through the ERDF Galicia 2021-27 operational program (ref. ED431G 2023/01) and project PID2022-137061OB-C21 (MCIN/AEI/ 10.13039/501100011033, Ministerio de Ciencia e Innovación, Agencia Estatal de Investigación, Proyectos de Generación de Conocimiento; supported by “ERDF A way of making Europe”, by the “European Union”)., Xunta de Galicia; ED431G 2023/04, Xunta de Galicia; ED431C 2022/19, Xunta de Galicia; ED431G 2023/01
RUC. Repositorio da Universidade da Coruña
Publication Artículo científico (JournalArticle)

DepreSym: A Depression Symptom Annotated Corpus and the Role of Large Language Models as Assessors of Psychological Markers

  • Pérez, Anxo
  • Fernández-Pichel; Marcos
  • Parapar, Javier
  • Losada, David E.
DepreSym can be obtained upon request, and we have adhered to strict ethical standards, particularly those about the ethical development of AI. Financiado para publicación en acceso aberto: CRUE/CSIC con Springer, [Abstract]: Computational methods for depression detection aim to mine traces of depression from online publications posted by Internet users. However, solutions trained on existing collections exhibit limited generalisation and interpretability. To tackle these issues, recent studies have shown that identifying specific depressive symptoms can lead to more robust and effective models. The eRisk initiative fosters research on this area and has recently proposed a new ranking task focused on developing search methods to find sentences related to depressive symptoms. This search challenge relies on the symptoms specified by the Beck Depression Inventory-II (BDI-II), a questionnaire widely used in clinical practice. It includes symptoms such as sadness, irritability or lack of sleep. Given the input submitted by systems participating in eRisk, we first apply top-k pooling over the systems’ relevance rankings, obtaining a diverse set of sentences. These sentences are judged for relevance, leading to DepreSym, a dataset consisting of 21,580 sentences annotated according to their relevance to the 21 BDI-II symptoms. This dataset serves as a valuable resource for advancing the development of models that monitor depression markers. Due to the complex nature of this relevance annotation, we designed a robust assessment methodology carried out by three expert assessors, including a trained psychologist. As part of this study, we explore the potential of recent Large Language Models (ChatGPT, GPT4 and Vicuna) as assessors in this complex task. We undertake a comprehensive examination of the LLMs’ performance, studying their main limitations and analysing their role as a complement or replacement for human annotators. Finally, we incorporate our dataset into the Benchmarking Information Retrieval (BEIR) framework for a thorough search evaluation. We use state-of-the-art retrieval systems, including lexical, sparse, dense and re-ranking architectures, to gain insights about the dataset’s complexity and identify potential avenues for improvement., Open Access funding provided thanks to the CRUE-CSIC agreement with Springer Nature. The first and third authors has received support from projects: PLEC2021-007662 (MCIN/AEI/10.13039/501100011033 Ministerio de Ciencia e Innovación, European Union Next Generation EU/PRTR) and also PID2022-137061OB-C21 (MCIN/AEI/10.13039/501100011033, Ministerio de Ciencia e Innovación). We also thank the financial support supplied by the Consellería de Cultura, Educación, Formación Profesional e Universidades (accreditation 2019-2022 ED431G/01, ED431B 2022/33) and the European Regional Development Fund, which acknowledges the CITIC Research Center. The second and fourth authors thank the financial support supplied by MICIU/AEI/10.13039/501100011033 (PID2022-137061OB-C22, supported by ERDF) and Xunta de Galicia-Consellería de Cultura, Educación, Formación Profesional e Universidades (ED431G 2023/04, ED431C 2022/19, supported by ERDF)., Xunta de Galicia; ED431G/01 2019/2022, Xunta de Galicia; ED431B 2022/33, Xunta de Galicia; ED431G 2023/04, Xunta de Galicia; ED431C 2022/19
RUC. Repositorio da Universidade da Coruña
Publication Artículo científico (JournalArticle)

CounselingQA: Scaling Professional Mental-Health Guidance via Dense Retrieval and Large Language Models

  • Pérez, Anxo
  • Piñeiro, César
  • Parapar, Javier
Financiado para publicación en acceso aberto: CRUE-CSIC/Springer Nature Data: The collections are available under the appropriate research agreement: https://erisk.irlab.org/ScalingMentalSupport.html, [Abstract] In the digital era, the internet and social media have emerged as essential platforms for individuals facing mental health issues, often used for seeking information and community support. Despite the resources of informal advice available on social media, the complexity of these issues frequently exceeds non-expert knowledge. Specialized sites such as CounselChat and 7Cups offer professional guidance, yet many at-risk individuals still rely on unmoderated sources and general web search. We address this gap by investigating ranking strategies that match pre-existing expert advice to incoming mental-health questions. We introduce CounselingQA, a collection built from two specialized websites, pairing user questions with verified expert responses. We address the task as answer retrieval (AR): given a question, rank expert answers by relevance. We evaluate dense retrieval with SentenceBERT and MentalBERT, and propose a second stage that improves the initial ranking via transformer-based models and large language models (LLMs), used for filtering non-relevant candidates and for reordering. Beyond retrieval, we analyze linguistic style and affective attributes across topics, questions, and responses. Results show that dense retrieval provides strong candidates and that transformer/LLM-driven reranking further elevates relevant, on-topic advice to the top positions. We further conduct qualitative error analyses, including human evaluation to study the benefits and limitations of our approaches. Taken together, these findings indicate that retrieval-first pipelines can help scale access to professional guidance., All authors affiliated with IRLab and CITIC acknowledge funding from the Ministry of Science, Innovation and Universities of the Government of Spain (project PID2022-137061OB-C21, MCIN/AEI/10.13039/501100011033), as well as from the Department of Education, Science, Universities of the Xunta de Galicia (grant GRC ED431C 2025/49). CITIC, as a center accredited for excellence within the Galician University System and a member of the CIGUS Network, receives subsidies from the Xunta de Galicia. Additionally, it is co-financed by the EU through the FEDER Galicia 2021–27 operational program (Ref. ED431G 2023/01)., Xunta de Galicia; ED431C 2025/49, Xunta de Galicia; ED431G 2023/01, https://erisk.irlab.org/ScalingMentalSupport.html
RUC. Repositorio da Universidade da Coruña
Publication Artículo científico (JournalArticle)

A Study of Word Embedding Models for Measuring Topic Coherence

  • Couto, Manuel
  • Parapar, Javier
  • Losada, David E.
Financiado para publicación en acceso aberto: CRUE-CSIC Data: The datasets (20NG, NYT, Geno) used in this study are publicly available. The code to reproduce the experiments is available at Github: https://github.com/manucouto1/A-Study-of-Word-Embedding-Models-for-Measuring-Topic-Coherence, [Abstract]: Topic modeling has emerged as a crucial tool in the field of natural language processing, enabling the automatic discovery of latent structures in large textual corpora. However, determining the quality of the topics remains a significant challenge, particularly in measuring the coherence of the top words of the extracted topics. Early efforts relied on human judgments, but these approaches are resource-intensive. Automated coherence metrics have since been developed. For example, some measures exploit word co-occurrence, while other methods are grounded in distributional semantics (e.g., employing word embeddings). In this study, we thoroughly explore the application of embedded representations to evaluate the quality of topics. While a number of isolated studies have analyzed the role of specific word representation techniques for measuring topic coherence, a complete picture of their effectiveness is still lacking. This work brings together different embedding-based approaches, including Word2Vec, FastText, GloVe, and BERT, which had been studied separately, and extends prior research by incorporating additional models, such as RoBERTa, ALBERT and MPNET. Topic coherence is measured by computing similarity scores between word embeddings, thus obtaining rich semantic associations that traditional measures may overlook. Our analysis demonstrates that these methods are as effective as, and often surpass, classical coherence measures. Our results contribute to a growing body of research advocating for advanced semantic representations as robust alternatives to traditional approaches in evaluating topic model coherence., Open Access funding provided thanks to the CRUE-CSIC agreement with Springer Nature. The first and third authors thank the financial support supplied by the Agencia Estatal de Investigación (Spain) (PID2022-137061OB-C22; MCIN/AEI/10.13039/501100011033, Plan de Recuperación, Transformación y Resiliencia, Unión Europea-Next Generation EU), Consellería de Cultura, Educación, Formación Profesional e Universidades (Centro de investigación de Galicia accreditation 2024-2027 ED431G-2023/04 and Reference Competitive Group accreditation 2022-2025, ED431C 2022/19) and the European Union (European Regional Development Fund - ERDF). The second author thanks the financial support supplied from projects: PID2022-137061OB-C21 (MCIN/AEI/10.13039/501100011033/, Ministerio de Ciencia e Innovación, ERDF A way of making Europe, by the European Union); Consellería de Educación, Universidade e Formación Profesional, Spain (accreditations 2019–2022 ED431G/01 and GPC ED431B 2022/33) and the European Regional Development Fund, which acknowledges the CITIC Research Center., Xunta de Galicia; ED431G-2023/04, Xunta de Galicia; ED431C 2022/19, Xunta de Galicia; 2019–2022 ED431G/01, Xunta de Galicia; ED431B 2022/33
RUC. Repositorio da Universidade da Coruña
Publication Artículo científico (JournalArticle)

Bridging gaps in hate speech detection: meta-collections and benchmarks for low-resource Iberian languages

  • Piot, Paloma
  • Pichel Campos, José Ramom
  • Parapar, Javier
Data is available upon request, and we have adhered to strict ethical standards, particularly those about the ethical development of AI (https://huggingface.co/datasets/irlab-udc/MetaHateES). The code to replicate the meta-collection construction, translation, analysis and running the baselines is available here https://github.com/palomapiot/mlhate., [Abstract]: Hate speech poses a serious threat to social cohesion and individual well-being, particularly on social media, where it spreads rapidly. While research on hate speech detection has progressed, it remains largely focused on English, resulting in limited resources and benchmarks for low-resource languages. Moreover, many of these languages have multiple linguistic varieties, a factor often overlooked in current approaches. At the same time, large language models require substantial amounts of data to perform reliably, a requirement that low-resource languages often cannot meet. In this work, we address these gaps by compiling a meta-collection of hate speech datasets for European Spanish, standardised with unified labels and metadata. This collection is based on a systematic analysis and integration of existing resources, aiming to bridge the data gap and support more consistent and scalable hate speech detection. We extended this collection by translating it into European Portuguese and into a Galician standard that is more convergent with Spanish and another Galician variant that is more convergent with Portuguese, creating aligned multilingual corpora. Using these resources, we establish new benchmarks for hate speech detection in Iberian languages. We evaluate state-of-the-art large language models in zero-shot, few-shot, and fine-tuning settings, providing baseline results for future research. Moreover, we perform a cross-lingual analysis with our target languages. Our findings underscore the importance of multilingual and variety-aware approaches in hate speech detection and offer a foundation for improved benchmarking in underrepresented European languages., Open Access funding provided thanks to the CRUE-CSIC agreement with Springer Nature. The authors thank the funding from the Horizon Europe research and innovation programme under the Marie Skłodowska-Curie Grant Agreement No. 101073351. The first and third authors also thank the financial support supplied by the grant PID2022-137061OB-C21 funded by MI-CIU/AEI/10.13039/501100011033 and by “ERDF/EU”. The authors also thank the funding supplied by the Consellería de Cultura, Educación, Formación Profesional e Universidades (accreditations ED431G 2023/01 and ED431C 2025/49) and the European Regional Development Fund, which acknowledges the CITIC, as a center accredited for excellence within the Galician University System and a member of the CIGUS Network, receives subsidies from the Department of Education, Science, Universities, and Vocational Training of the Xunta de Galicia. Additionally, it is co-financed by the EU through the FEDER Galicia 2021-27 operational program (Ref. ED431G 2023/01)., Xunta de Galicia; ED431G 2023/01, Xunta de Galicia; ED431C 2025/49
RUC. Repositorio da Universidade da Coruña
Publication Artículo científico (JournalArticle)

WATCHED: A Web AI Agent Tool for Combating Hate speech by Expanding Data

  • Piot, Paloma
  • Sánchez, Diego
  • Parapar, Javier
Original software publication Permanent link to code/repository used for this code version: https://github.com/ElsevierSoftwareX/SOFTX-D-25-00589 Permanent link to Reproducible Capsule: https://github.com/nulldiego/watched, [Abstract]: Online harms are a growing problem in digital spaces, putting user safety at risk and reducing trust in social media platforms. One of the most persistent forms of harm is hate speech. To address this, we need tools that combine the speed and scale of automated systems with the judgement and insight of human moderators. These tools should not only find harmful content but also explain their decisions clearly, helping to build trust and understanding. In this paper, we present WATCHED a chatbot designed to support content moderators in tackling hate speech. The chatbot is built as an Artificial Intelligence Agent system that uses Large Language Models along with several specialised tools. It compares new posts with real examples of hate speech and neutral content, uses a BERT-based classifier to help flag harmful messages, looks up slang and informal language using sources like Urban Dictionary, generates chain-of-thought reasoning, and checks platform guidelines to explain and support its decisions. This combination allows the chatbot not only to detect hate speech but to explain why content is considered harmful, grounded in both precedent and policy. Experimental results show that our proposed method surpasses existing state-of-the-art methods, reaching a macro F1 score of 0.91. Designed for moderators, safety teams, and researchers, the tool helps reduce online harms by supporting collaboration between AI and human oversight., The authors thank the funding from the Horizon Europe research and innovation programme under the Marie Skłodowska-Curie Grant Agreement No. 101073351. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Executive Agency (REA). Neither the European Union nor the granting authority can be held responsible for them. The authors thank the financial support supplied by the grant PID2022-137061OB-C21 funded by MI-CIU/AEI/10.13039/501100011033 and by “ERDF/EU”. The authors also thank the funding supplied by the Consellería de Cultura, Educación, Formación Profesional e Universidades (accreditations ED431G 2023/01 and ED431C 2025/49) and the European Regional Development Fund, which acknowledges the CITIC, as a centre accredited for excellence within the Galician University System and a member of the CIGUS Network, receives subsidies from the Department of Education, Science, Universities, and Vocational Training of the Xunta de Galicia. Additionally, it is co-financed by the EU through the FEDER Galicia 2021-27 operational programme (Ref. ED431G 2023/01)., Xunta de Galicia; ED431G 2023/01, Xunta de Galicia; ED431C 2025/49
RUC. Repositorio da Universidade da Coruña
Publication Artículo científico (JournalArticle)

Beyond questions: Leveraging ColBERT for keyphrase search

  • Gabín, Jorge
  • Parapar, Javier
  • Macdonald, Craig
This study’s code and generated resources are available at https://github.com/JorgeGabin/ColBERTKP., [Abstract]: While question-like queries are gaining popularity, keyphrase search is still the cornerstone of web search and other specialised domains such as academic and professional search. However, current dense retrieval models often fail with keyphrase-like queries, primarily because they are mostly trained on question-like ones. This paper introduces a novel model that employs the ColBERT architecture to enhance document ranking for keyphrase queries. For that, given the lack of large keyphrase-based retrieval datasets, we first explore how Large Language Models can convert question-like queries into keyphrase format. Then, using those keyphrases, we train a keyphrase-based ColBERT ranker (ColBERTKP ) to improve the performance when working with keyphrase queries. Furthermore, to make the model more flexible, allowing the use of both the question and keyphrase encoders depending on the query type, we investigate the feasibility of training only a keyphrase query encoder while keeping the document encoder weights static (ColBERTKP). We assess our proposals’ ranking performance using both automatically generated and manually annotated keyphrases. Our results reveal the potential of the late interaction architecture when working under the keyphrase search scenario., This work has received support from projects: PID2022-137061OB-C21 (MCIN/AEI/10 .13039/501100011033/, Ministerio de Ciencia e Innovación, ERDF A way of making Europe, by the European Union); Consellería de Educación, Universidade e Formación Profesional, Spain (ED431G 2023/01 and ED431C 2025/49 GRC) and the European Regional Development Fund, which acknowledges the CITIC Research Center. The first author also acknowledges the support of grant DIN2020-011582 financed by the MCIN/AEI/10.13039/501100011033., Xunta de Galicia; ED431G 2023/01, Xunta de Galicia; ED431C 2025/49
RUC. Repositorio da Universidade da Coruña
Publication Artículo científico (JournalArticle)

Exploiting Topic Analysis Models to Explore Psychological Dimensions in Social Media Data

  • Couto Pintos, Manuel
  • Parapar, Javier
  • Losada, David E.
Data availability: The dataset consisting of extracted topics and human ratings is publicly available at Zenodo: https://doi.org/10.5281/zenodo.15081947. The system built for topic assessment (web application developed specifically for storing topics and assessments in a database with a customized design) is available at https://github.com/manucouto1/Topic-Quality-Assessment-Tool. The eRisk datasets used in this study are publicly available (eRisk website). Specifically, the eRisk 2017 and 2018 datasets can be obtained at https://tec.citius.usc.es/ir/code/eRisk.html (note that 2017’s data was 2018’s training split), while the eRisk 2019 dataset is available at https://erisk.irlab.org/2019/eRisk2019.html., [Abstract]: Automatic topic generation is a fundamental tool in unstructured text analysis, yet its application to noisy web-based collections for extracting psychological patterns remains underexplored. This work compares three representative topic models from different families: Latent Dirichlet Allocation (classical probabilistic), BERTopic (embedding-based), and TopClus (deep neural network), evaluating their performance on mental health data from the eRisk initiative. Using posts from individuals with depressive disorders and control groups, we assess topic quality through both automatic coherence metrics and rigorous human evaluation by expert reviewers. This dual approach addresses the limitations of purely automatic evaluation in complex social media datasets where thematic content does not always reveal psychological cues. Our results demonstrate that BERTopic significantly outperforms other models in perceived coherence, identifying clearer and more specific themes, including depression-related topics such as mental health struggles and self-harm. Thematic analysis across user groups revealed that certain topics contained higher proportions of posts from individuals with depression, providing actionable insights for psychological screening. This work underscores the potential of advanced topic models for mental health analysis in noisy social media data and highlights the importance of human evaluation in validating topic quality for sensitive applications., The first and third authors thank the financial support supplied by the Agencia Estatal de Investigación (Spain) (PID2022-137061OB-C22; MCIN/AEI/10.13039/501100011033, Plan de Recuperación, Transformación y Resiliencia, Unión Europea-Next Generation EU), Consellería de Cultura, Educación, Formación Profesional e Universidades (Centro de investigación de Galicia accreditation 2024-2027 ED431G-2023/04 and Reference Competitive Group accreditation 2022-2025, ED431C 2022/19) and the European Union (European Regional Development Fund - ERDF). These authors also acknowledge the project “Cátedra de IA aplicada a la Medicina Personalizada de Precisión” (Cátedras ENIA, TSI-100932-2023-3); Cátedras ENIA is funded by the Ministerio de Transformación Digital y Función Pública (Secretaría de Estado de Digitalización e Inteligencia Artificial); and by the NextGeneration EU-fund. The second author thanks the financial support supplied from projects: PID2022-137061OB-C21 (MCIN/AEI/10.13039/501100011033/, Ministerio de Ciencia e Innovación, ERDF A way of making Europe, by the European Union); Consellería de Educación, Universidade e Formación Profesional, Spain (accreditations 2019–2022 ED431G/01 and GPC ED431B 2022/33) and the European Regional Development Fund, which acknowledges the CITIC Research Center., Xunta de Galicia; ED431G/01, Xunta de Galicia; ED431B 2022/33, Xunta de Galicia; ED431G-2023/04, Xunta de Galicia; ED431C 2022/19, https://doi.org/10.5281/zenodo.15081947, https://github.com/manucouto1/Topic-Quality-Assessment-Tool, https://tec.citius.usc.es/ir/code/eRisk.html, https://erisk.irlab.org/2019/eRisk2019.html
RUC. Repositorio da Universidade da Coruña
Publication Artículo científico (JournalArticle)

Relevance feedback for building pooled test collections

  • Otero, David
  • Parapar, Javier
  • Barreiro, Álvaro
This manuscript version of the article: Otero, D., Parapar, J., & Barreiro, Á. (2023). ‘Relevance feedback for building pooled test collections’ has been accepted for publication in Journal of Information Science, 2023, pp-1-18. DOI: https://doi.org/10.1177/0165551515598926., [Abstract]: Offline evaluation of information retrieval systems depends on test collections. These datasets provide the researchers with a corpus of documents, topics and relevance judgements indicating which documents are relevant for each topic. Gathering the latter is costly, requiring human assessors to judge the documents. Therefore, experts usually judge only a portion of the corpus. The most common approach for selecting that subset is pooling. By intelligently choosing which documents to assess, it is possible to optimise the number of positive labels for a given budget. For this reason, much work has focused on developing techniques to better select which documents from the corpus merit human assessments. In this article, we propose using relevance feedback to prioritise the documents when building new pooled test collections. We explore several state-of-the-art statistical feedback methods for prioritising the documents the algorithm presents to the assessors. A thorough comparison on eight Text Retrieval Conference (TREC) datasets against strong baselines shows that, among other results, our proposals improve in retrieving relevant documents with lower assessment effort than other state-of-the-art adjudicating methods without harming the reliability, fairness and reusability., The author(s) disclosed receipt of the following financial support for the research, authorship and/or publication of this article: this work has received support from: (1) project PLEC2021-007662 (grant no. MCIN/AEI/10.13039/501100011033, Ministerio de Ciencia e Innovación (MCIN), Agencia Estatal de Investigación, Plan de Recuperación, Transformación y Resiliencia, Unión Europea-NextGenerationEU), (2) Programa de Ayudas para la Formación de Profesorado Universitario, grant number FPU20/02659 (Ministerio de Universidades), (3) project PID2022-137061OB-C21 (Proyectos de Generación de Conocimiento, MCIN), (4) Consellería de Educación, Universidade e Formación Profesional (accreditation 2019-2022 ED431G 2019/01) and the European Regional Development Fund, which acknowledges the Centro de Investigación en Tecnologías de la Información y la Comunicación (CITIC) Research Centre in Information and Communications Technology (ICT) of the University of A Coruña as a Research Centre of the Galician University System and (5) project ED431-B 2022/33 (Xunta de Galicia/European Regional Development Fund (ERDF)., Xunta de Galicia; ED431-B 2022/33, Xunta de Galicia; ED431G 2019/01
RUC. Repositorio da Universidade da Coruña
Publication Artículo científico (JournalArticle)

Explainable Depression Symptom Detection in Social Media

  • Bao, Eliseo
  • Pérez, Anxo
  • Parapar, Javier
Financiado para publicación en acceso aberto: Universidade da Coruña/CISUG, [Abstract] Users of social platforms often perceive these sites as supportive spaces to post about their mental health issues. Those conversations contain important traces about individuals’ health risks. Recently, researchers have exploited this online information to construct mental health detection models, which aim to identify users at risk on platforms like Twitter, Reddit or Facebook. Most of these models are focused on achieving good classification results, ignoring the explainability and interpretability of the decisions. Recent research has pointed out the importance of using clinical markers, such as the use of symptoms, to improve trust in the computational models by health professionals. In this paper, we introduce transformer-based architectures designed to detect and explain the appearance of depressive symptom markers in user-generated content from social media. We present two approaches: (i) train a model to classify, and another one to explain the classifier’s decision separately and (ii) unify the two tasks simultaneously within a single model. Additionally, for this latter manner, we also investigated the performance of recent conversational Large Language Models (LLMs) utilizing both in-context learning and finetuning. Our models provide natural language explanations, aligning with validated symptoms, thus enabling clinicians to interpret the decisions more effectively. We evaluate our approaches using recent symptom-focused datasets, using both offline metrics and expert-in-the-loop evaluations to assess the quality of our models’ explanations. Our findings demonstrate that it is possible to achieve good classification results while generating interpretable symptom-based explanations., This work has received support from projects: PLEC2021-007662 (MCIN/AEI/10.13039/501100011033, Ministerio de Ciencia e Innovación, European Union NextGenerationEU/PRTR) and PID2022-137061OB-C21 (MCIN/AEI/10.13039/501100011033/, Ministerio de Ciencia e Innovación, ERDF A way of making Europe, by the European Union); Consellería de Educación, Universidade e Formación Profesional, Spain (grant number ED481A-2024-079 and accreditations 2019–2022 ED431G/01 and GPC ED431B 2022/33) and the European Regional Development Fund, which acknowledges the CITIC Research Center. Funding for open access charge: Universidade da Coruña/CISUG. We would also like to thank Desireé Pombar, Silvia López-Larrosa and Laura Hermo for their efforts in evaluating the results. Their work allowed us to assess the practical utility and clinical relevance of the explanations generated., Xunta de Galicia; ED481A-2024-079, Xunta de Galicia; ED431G/01, Xunta de Galicia; ED431B 2022/33
RUC. Repositorio da Universidade da Coruña
Publication Artículo científico (JournalArticle)

Temporal Word Embeddings for Early Detection of Psychological Disorders on Social Media

  • Couto Pintos, Manuel
  • Pérez, Anxo
  • Parapar, Javier
  • Losada, David E.
[Abstract]: Mental health disorders represent a public health challenge, where early detection is critical to mitigating adverse outcomes for individuals and society. The study of language and behavior is a pivotal component in mental health research, and the content from social media platforms serves as a valuable tool for identifying signs of mental health risks. This paper presents a novel framework leveraging temporal word embeddings to capture linguistic changes over time. We specifically aim at at identifying emerging psychological concerns on social media. By adapting temporal word representations, our approach quantifies shifts in language use that may signal mental health risks. To that end, we implement two alternative temporal word embedding models to detect linguistic variations and exploit these variations to train early detection classifiers. Our experiments, conducted on 18 datasets from the eRisk initiative (covering signs of conditions such as depression, anorexia, and self-harm), show that simple models focusing exclusively on temporal word usage patterns achieve competitive performance compared to state-of-the-art systems. Additionally, we perform a word-level analysis to understand the evolution of key terms among positive and control users. These findings underscore the potential of time-sensitive word models in this domain, being a promising avenue for future research in mental health surveillance., The second and third authors thank the financial support supplied from projects: PLEC2021-007662 (MCIN/AEI/10.13039/501100011033 Ministerio de Ciencia e Innovación, European Union NextGenerationEU/PRTR) and PID2022-137061OB-C21 (MCIN/AEI/10.13039/501100011033/, Ministerio de Ciencia e Innovación, ERDF A way of making Europe, by the European Union); Consellería de Educación, Universidade e Formación Profesional, Spain (accreditations 2019-2022 ED431G/01 and GPC ED431B 2022/33) and the European Regional Development Fund, which acknowledges the CITIC Research Center. The first and fourth authors thank the financial support supplied by the Agencia Estatal de Investigación (Spain) (PID2022-137061OB-C22; PLEC2021-007662 MCIN/AEI/10.13039/501100011033, Plan de Recuperación, Transformación y Resiliencia, Unión Europea-Next Generation EU), Consellería de Cultura, Educación, Formación Profesional e Universidades (Centro de investigación de Galicia accreditation 2024-2027 ED431G-2023/04 and Reference Competitive Group accreditation 2022-2025, ED431C 2022/19) and the European Union (European Regional Development Fund - ERDF). The fourth author thanks the support obtained from project SUBV23/00002 (Ministerio de Consumo, Subdirección General de Regulación del Juego). Open Access funding provided thanks to the CRUE-CSIC agreement with Springer Nature., Xunta de Galicia; ED431G/01, Xunta de Galicia; ED431B 2022/33, Xunta de Galicia; ED431G-2023/04, Xunta de Galicia; ED431C 2022/19