Pasar al contenido principal

BUSQUEDA, SELECCION Y ORGANIZACION DE CONTENIDOS PARA NECESIDADES DE INFORMACION RELACIONADAS CON LA SALUD - CONSTRUCCION DE RECURSOS Y PERSONALIZACION

PID2022-137061OB-C21

•
Nombre agencia financiadora Agencia Estatal de Investigación
Acrónimo agencia financiadora AEI
Programa Programa Estatal para Impulsar la Investigación Científico-Técnica y su Transferencia
Subprograma Subprograma Estatal de Generación de Conocimiento
Convocatoria Proyectos de I+D+i (Generación de Conocimiento y Retos Investigación)
Año convocatoria 2022
Plan Estatal 2021-2023 Plan Estatal de Investigación Científica y Técnica y de Innovación
Centro beneficiario UNIVERSIDADE DA CORUÑA

Resultados relacionados

Resultados totales (Incluyendo duplicados): 19
Encontrada(s) 1 página(s)

RUC. Repositorio da Universidade da Coruña
Publicación Contribución de congreso (ConferenceOutput)

PartisanLens: A Multilingual Dataset of Hyperpartisan and Conspiratorial Immigration Narratives in European Media

  • Maggini, Michele Joshua
  • Piot, Paloma
  • Pérez, Anxo
  • Marino, Erik Bran
  • Santamaría Montesinos, Lúa
  • Lisboa Cotovio, Ana
  • Vázquez Abuín, Marta
  • Parapar, Javier
  • Gamallo, Pablo
Presented at: 19th Conference of the European Chapter of the Association for Computational Linguistics, March 24–29, 2026, Rabat, Morocco Dataset and code are available here: https://github.com/MichJoM/PartisanLens, [Abstract]: Detecting hyperpartisan narratives and Population Replacement Conspiracy Theories (PRCT) is essential to addressing the spread of misinformation. These complex narratives pose a significant threat, as hyperpartisanship drives political polarisation and institutional distrust, while PRCTs directly motivate real-world extremist violence, making their identification critical for social cohesion and public safety. However, existing resources are scarce, predominantly English-centric, and often analyse hyperpartisanship, stance, and rhetorical bias in isolation rather than as interrelated aspects of political discourse. To bridge this gap, we introduce PartisanLens, the first multilingual dataset of 1617 hyperpartisan news headlines in Spanish, Italian, and Portuguese, annotated in multiple political discourse aspects. We first evaluate the classification performance of widely used Large Language Models (LLMs) on this dataset, establishing robust baselines for the classification of hyperpartisan and PRCT narratives. In addition, we assess the viability of using LLMs as automatic annotators for this task, analysing their ability to approximate human annotation. Results highlight both their potential and current limitations. Next, moving beyond standard judgments, we explore whether LLMs can emulate human annotation patterns by conditioning them on socio-economic and ideological profiles that simulate annotator perspectives. At last, we provide our resources and evaluation; PartisanLens supports future research on detecting partisan and conspiratorial narratives in European contexts., The authors thank the funding from the Horizon Europe research and innovation programme under the Marie Skłodowska-Curie Grant Agreement No. 101073351. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Executive Agency (REA). Neither the European Union nor the granting authority can be held responsible for them. The authors thank the financial support supplied by the grant PID2022-137061OB-C21 funded by MICIU/AEI/10.13039/501100011033 and by “ERDF/EU”. The authors also thank the funding supplied by the Consellería de Cultura, Educación, Formación Profesional e Universidades (accreditations ED431G 2023/01 and ED431C 2025/49) and the European Regional Development Fund, which acknowledges the CITIC, as a center accredited for excellence within the Galician University System and a member of the CIGUS Network, receives subsidies from the Department of Education, Science, Universities, and Vocational Training of the Xunta de Galicia. Additionally, it is co-financed by the EU through the FEDER Galicia 2021-27 operational program (Ref. ED431G 2023/01)., Xunta de Galicia; ED431G 2023/01, Xunta de Galicia; ED431C 2025/49
RUC. Repositorio da Universidade da Coruña
Publicación Contribución de congreso (ConferenceOutput)

Limitations of Automatic Relevance Assessments with Large Language Models for Fair and Reliable Retrieval Evaluation

  • Otero, David
  • Parapar, Javier
  • Barreiro, Álvaro
Presented at: SIGIR '25: The 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, July 13 - 18, 2025, Padua, Italy © Owner/Author | ACM 2025 This is the author's version of the work. It is posted here for your personal use. Not for redistribution. The definitive Version of Record was published in SIGIR ’25, https://doi.org/10.1145/3726302.3730221, [Abstract]: Offline evaluation of search systems depends on test collections. These benchmarks provide the researchers with a corpus of documents, topics and relevance judgements indicating which documents are relevant for each topic. While test collections are an integral part of Information Retrieval (IR) research, their creation involves significant efforts in manual annotation. Large language models (LLMs) are gaining much attention as tools for automatic relevance assessment. Recent research has shown that LLM-based assessments yield high systems ranking correlation with human-made judgements. These correlations are helpful in large-scale experiments but less informative if we want to focus on top-performing systems. Moreover, these correlations ignore whether and how LLM-based judgements impact the statistically significant differences among systems with respect to human assessments. In this work, we look at how LLM-generated judgements preserve ranking differences among top-performing systems and also how they preserve pairwise significance evaluation as human judgements. Our results show that LLM-based judgements are unfair at ranking top-performing systems. Moreover, we observe an exceedingly high rate of false positives regarding statistical differences., The authors thank the financial support supplied by the grant PID2022-137061OB-C21 funded by MICIU/AEI/10.13039/501100011033 and by “ERDF/EU”. The authors also thank the funding supplied by the Consellería de Cultura, Educación, Formación Profesional e Universidades (accreditations ED431G 2023/01 and ED431C 2025/49) and the European Regional Development Fund, which acknowledges the CITIC Research Center as a Center of Excellence and recognizes it as a Member of the CIGUS Network for the period 2024-2027.
RUC. Repositorio da Universidade da Coruña
Publicación Contribución de congreso (ConferenceOutput)

Can LLMs Evaluate What They Cannot Annotate? Revisiting LLM Reliability in Hate Speech Detection

  • Piot, Paloma
  • Otero, David
  • Martín-Rodilla, Patricia
  • Parapar, Javier
Presented at: The Fifteenth Language Resources and Evaluation Conference (LREC 2026), 11 - 16 May 2026, Palma, Mallorca, Spain Material complementario: https://f003.backblazeb2.com/file/lrec-media/lrec2026/posters/385.pdf (poster) e https://f003.backblazeb2.com/file/lrec-media/lrec2026/videos/385.mp4 (video) Code is available at https://github.com/palomapiot/hate-eval-agreement/, [Abstract]: Hate speech spreads widely online and harms both individuals and communities, making automatic detection essential for large-scale moderation. However, accurately detecting hate speech remains a difficult task. Part of the challenge lies in subjectivity: what one person flags as hate speech, another may see as benign. Traditional annotation agreement metrics, such as Cohen’s k, oversimplify this disagreement, treating it as an error rather than meaningful diversity. Meanwhile, Large Language Models (LLMs) promise scalable annotation, but prior studies demonstrate that they cannot fully replace human judgement, especially in subjective tasks. In this work, we reexamine LLM reliability using a subjectivity-aware framework, cross-Replication Reliability (xRR), revealing that even under fairer lens, LLMs still diverge from humans. Yet this limitation opens an opportunity: we find that LLM-generated annotations can reliably reflect performance trends across classification models, correlating with human evaluations. We test this by examining whether LLM-generated annotations preserve the relative ordering of model performance derived from human evaluation (i.e. whether models ranked as more reliable by human annotators preserve the same order when evaluated with LLM-generated labels). Our results show that, although LLMs differ from humans at the instance level, they reproduce similar ranking and classification patterns, suggesting their potential as proxy evaluators. While not a substitute for human annotators, they might serve as a scalable proxy for evaluation in subjective NLP tasks., The authors thank the funding from the Horizon Europe research and innovation programme under the Marie Skłodowska-Curie Grant Agreement No. 101073351. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Executive Agency (REA). Neither the European Union nor the granting authority can be held responsible for them. The authors thank the financial support supplied by the grant PID2022-137061OB-C21 funded by MI-CIU/AEI/10.13039/501100011033 and by “ERDF/EU”. The authors also thank the funding supplied by the Consellería de Cultura, Educación, Formación Profesional e Universidades (accreditations ED431G 2023/01 and ED431C 2025/49) and the European Regional Development Fund, which acknowledges the CITIC, as a center accredited for excellence within the Galician University System andamember of the CIGUS Network,receives subsidies from the Department of Education, Science, Universities, and Vocational Training of the Xunta de Galicia. Additionally, it is co-financed by the EU through the FEDER Galicia 2021-27 operational program (Ref. ED431G 2023/01)., Xunta de Galicia; ED431G 2023/01, Xunta de Galicia; ED431C 2025/49, https://github.com/palomapiot/hate-eval-agreement/, https://f003.backblazeb2.com/file/lrec-media/lrec2026/videos/385.mp4
RUC. Repositorio da Universidade da Coruña
Publicación Contribución de congreso (ConferenceOutput)

Learning Evidence of Depression Symptoms via Prompt Induction

  • Bao, Eliseo
  • Pérez, Anxo
  • Otero, David
  • Parapar, Javier
Presented at: SIGIR '26: Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval, July 20–24, 2026, Melbourne, Australia Code available at https://github.com/IRLab-UDC/depression-prompt-induction., [Abstract]: Depression places substantial pressure on mental health services, and many people describe their experiences outside clinical settings in high-volume user-generated text (e.g., online forums and social media). Automatically identifying clinical symptom evidence in such text can therefore complement limited clinical capacity and scale to large populations. We address this need through sentence-level classification of 21 depression symptoms from the BDI-II questionnaire, using BDI-Sen, a dataset annotated for symptom relevance. This task is fine-grained and highly imbalanced, and we find that common LLM approaches (zero-shot, in-context learning, and fine-tuning) struggle to apply consistent relevance criteria for most symptoms. We propose Symptom Induction (SI), a novel approach which compresses labeled examples into short, interpretable guidelines that specify what counts as evidence for each symptom and uses these guidelines to condition classification. Across four LLM families and eight models, SI achieves the best overall weighted F1 on BDI-Sen, with especially large gains for infrequent symptoms. Cross-domain evaluation on an external dataset further shows that induced guidelines generalize across other diseases shared symptomatology (bipolar and eating disorders)., The first author acknowledges the support of the Department of Education, Science, Universities, and Vocational Training of the Xunta de Galicia (grant ED481A-2024-079). All authors affiliated with IRLab and CITIC acknowledge funding from the Ministry of Science, Innovation and Universities of the Government of Spain (project PID2022-137061OB-C21, MCIN/AEI/10.13039/501100011033), as well aas from the Department of Education, Science, Universities, and Vocational Training of the Xunta de Galicia (grant GRC ED431C 2025/49). CITIC, as a center accredited for excellence within the Galician University System and a member of the CIGUS Network, receives subsidies from the Department of Education, Science, Universities, and Vocational Training of the Xunta de Galicia. Additionally, it is co-financed by the EU through the FEDER Galicia 2021-27 operational program (Ref. ED431G 2023/01)., Xunta de Galicia; ED481A-2024-079, Xunta de Galicia; GRC ED431C 2025/49, Xunta de Galicia; ED431G 2023/01
RUC. Repositorio da Universidade da Coruña
Publicación Contribución de congreso (ConferenceOutput)

Decoding Hate: Exploring Language Models' Reactions to Hate Speech

  • Piot, Paloma
  • Parapar, Javier
[Abstract]: Hate speech is a harmful form of online expression, often manifesting as derogatory posts. It is a significant risk in digital environments. With the rise of Large Language Models (LLMs), there is concern about their potential to replicate hate speech patterns, given their training on vast amounts of unmoderated internet data. Understanding how LLMs respond to hate speech is crucial for their responsible deployment. However, the behaviour of LLMs towards hate speech has been limited compared. This paper investigates the reactions of seven state-of-the-art LLMs (LLaMA 2, Vicuna, LLaMA 3, Mistral, GPT-3.5, GPT-4, and Gemini Pro) to hate speech. Through qualitative analysis, we aim to reveal the spectrum of responses these models produce, highlighting their capacity to handle hate speech inputs. We also discuss strategies to mitigate hate speech generation by LLMs, particularly through fine-tuning and guideline guardrailing. Finally, we explore the models’ responses to hate speech framed in politically correct language., The authors thank the funding from the Horizon Europe research and innovation programme under the Marie Skłodowska-Curie Grant Agreement No. 101073351. The authors also thank the financial support supplied by the Consellería de Cultura, Educación, Formación Profesional e Universidades (accreditation 2019-2022 ED431G/01, ED431B 2022/33) and the European Regional Development Fund, which acknowledges the CITIC Research Center in ICT as a Research Center of the Galician University System and the project PID2022-137061OB-C21 (Ministerio de Ciencia e Innovación supported by the European Regional Development Fund). The authors also thank the funding of project PLEC2021-007662 (MCIN/AEI/10.13039/501100011033, Ministerio de Ciencia e Innovación, Agencia Estatal de Investigación, Plan de Recuperación, Transformación y Resiliencia, Unión Europea-Next Generation EU)., Xunta de Galicia; 2019-2022 ED431G/01, Xunta de Galicia; ED431B 2022/33
RUC. Repositorio da Universidade da Coruña
Publicación Contribución de congreso (ConferenceOutput)

Beyond Top-e: Simulation-Based Interactive Evaluation for Query Suggestions

  • Gabín, Jorge
  • Parapar, Javier
  • Wang, Xi
Presented at: SIGIR '26: Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval, July 20–24, 2026, Melbourne, Australia, [Abstract]: Evaluating query suggestion systems in a manner that reflects real-world query formulation remains a persistent challenge. Most offline methodologies adopt static assumptions, such as users accepting all or the top-e suggestions, ignoring the inherently selective and intent-driven nature of interactive search. While online experiments provide realistic behavioural signals, they are costly, difficult to scale, and often irreproducible. To bridge this gap, we introduce SIQSE (Simulation-based Interactive Query Suggestion Evaluation), a framework that models query reformulation as an interactive selection task performed by a simulated user. In SIQSE, a Large Language Model (LLM) acts as a surrogate user that progressively selects suggestions according to contextual relevance and explicit search intent. Unlike static offline protocols, this simulation captures the iterative and selective dynamics of real query formulation. Our contributions are twofold. First, we develop and validate an LLM-based selection model, systematically analysing how varying levels of intent information and selection strategies affect its ability to approximate human selection behaviour. Second, we employ this selector to benchmark multiple query suggestion systems across diverse datasets under interactive conditions. Importantly, while the selector is LLM-based, the final evaluation is computed exclusively through ranking-based effectiveness metrics over the rankings produced by selected expansions, ensuring that system performance reflects retrieval quality rather than alignment with the surrogate user model. By modelling round-based interaction while maintaining metric independence, SIQSE offers a scalable, reproducible evaluation paradigm that brings offline assessment closer to the complexity of real-world search behaviour. To facilitate adoption and reproducibility, we release SIQSE as an open-source Python library., First and second authors acknowledge funding from the Ministry of Science, Innovation and Universities of the Government of Spain (project PID2022-137061OB-C21, MCIN/AEI/10.13039/501100011033), as well as from the Department of Education, Science, Universities, and Vocational Training of the Xunta de Galicia (grant GRC ED431C 2025/49). CITIC, as a center accredited for excellence within the Galician University System and a member of the CIGUS Network, receives subsidies from the Department of Education, Science, Universities, and Vocational Training of the Xunta de Galicia. Additionally, it is co-financed by the EU through the FEDER Galicia 2021-27 operational program (Ref. ED431G 2023/01)., Xunta de Galicia; GRC ED431C 2025/49, Xunta de Galicia; ED431G 2023/01
RUC. Repositorio da Universidade da Coruña
Publicación Contribución de congreso (ConferenceOutput)

Towards Reliable Testing for Multiple Information Retrieval System Comparisons

  • Otero, David
  • Parapar, Javier
  • Barreiro, Álvaro
Presented at: Advances in Information Retrieval: 47th European Conference on Information Retrieval, ECIR 2025, Lucca, Italy, April 6–10, 2025, This version of the article has been accepted for publication, after peer review and is subject to Springer Nature’s AM terms of use, but is not the Version of Record and does not reflect post-acceptance improvements, or any corrections. The Version of Record is available online at: http://dx.doi.org/10.1007/978-3-031-88711-6_27, [Abstract]: Null Hypothesis Significance Testing is the de facto tool for assessing effectiveness differences between Information Retrieval systems. Researchers use statistical tests to check whether those differences will generalise to online settings or are just due to the samples observed in the laboratory. Much work has been devoted to studying which test is the most reliable when comparing a pair of systems, but most of the IR real-world experiments involve more than two. In the multiple comparisons scenario, testing several systems simultaneously may inflate the errors committed by the tests. In this paper, we use a new approach to assess the reliability of multiple comparison procedures using simulated and real TREC data. Experiments show that Wilcoxon plus the Benjamini-Hochberg correction yields Type I error rates according to the significance level for typical sample sizes while being the best test in terms of statistical power., The authors thank the financial support supplied by the Consellería de Cultura, Educación, Formación Profesional e Universidades (accreditation 2019-2022 ED431G/01, ED431B 2022/33) and the European Regional Development Fund, which acknowledges the CITIC Research Center in ICT as a Research Center of the Galician University System and the project PID2022-137061OB-C21 (Ministerio de Ciencia e Innovación supported by the European Regional Development Fund). The authors also thank the funding of project PLEC2021-007662 (MCIN/AEI/10.13039/501100011033, Ministerio de Ciencia e Innovación, Agencia Estatal de Investigación, Plan de Recuperación, Transformación y Resiliencia, Unión Europea-Next Generation EU)., Xunta de Galicia; ED431G/01, Xunta de Galicia; ED431B 2022/33
RUC. Repositorio da Universidade da Coruña
Publicación Contribución de congreso (ConferenceOutput)

Overview of eRisk 2024: Early Risk Prediction on the Internet (Extended Overview)

  • Parapar, Javier
  • Martín-Rodilla, Patricia
  • Losada, David E.
  • Crestani, Fabio
Included in: Working Notes of the Conference and Labs of the Evaluation Forum (CLEF 2024) Grenoble, France, 9-12 September, 2024., [Abstract]: This paper presents eRisk 2024, the eighth edition of the CLEF conference’s lab dedicated to early risk detection. Since its inception, the lab has been at the forefront of developing and refining evaluation methodologies, effectiveness metrics, and processes for early risk detection across various domains. These early alerting models hold significant value, particularly in sectors focused on health and safety, where timely intervention can be crucial. eRisk 2024 featured three main tasks designed to push the boundaries of early risk detection techniques. The first task challenged participants to rank sentences based on their relevance to standardized depression symptoms, a crucial step in identifying early signs of depression from textual data. The second task focused on the early detection of anorexia indicators, aiming to develop models that can recognize the subtle cues of this eating disorder before it becomes critical. The third task was centered around estimating responses to an eating disorders questionnaire by analyzing users’ social media posts. Participants had to leverage the rich, real-world textual data available on social media to gauge potential mental health risks. Through these tasks, eRisk 2024 continues to advance the field of early risk detection, fostering innovations that could lead to significant improvements in public health interventions., This work was supported by project PLEC2021-007662 (MCIN/AEI/10.13039/ 501100011033, Ministerio de Ciencia e Innovación, Agencia Estatal de Investigación, Plan de Recuperación, Transformación y Resiliencia, Unión Europea-Next Generation EU). The first and second authors thank the financial support supplied by the Xunta de Galicia-Consellería de Cultura, Educación, Formación Profesional e Universidade (GPC ED431B 2022/33) and the European Regional Development Fund and project PID2022- 137061OB-C21 (MCIN/AEI/ 10.13039/501100011033, Ministerio de Ciencia e Innovación, Agencia Estatal de Investigación, Proyectos de Generación de Conocimiento; supported by “ERDF A way of making Europe”, by the “European Union”). The CITIC, as a center accredited for excellence within the Galician University System and a member of the CIGUS Network, receives subsidies from the Department of Education, Science, Universities, and Vocational Training of the Xunta de Galicia. Additionally, CITIC is co-financed by the EU through the FEDER Galicia 2021-27 operational program (Ref. ED431G 2023/01). The third author thanks the financial support supplied by the Xunta de Galicia-Consellería de Cultura, Educación, Formación Profesional e Universidade (accreditation 2019-2022 ED431G-2019/04, ED431C 2022/19) and the European Regional Development Fund, which acknowledges the CiTIUS-Research Center in Intelligent Technologies of the University of Santiago de Compostela as a Research Center of the Galician University System. David E. Losada also thanks the financial support obtained from project SUBV23/00002 (Ministerio de Consumo, Subdirección General de Regulación del Juego) and project PID2022-137061OB-C22 (Ministerio de Ciencia e Innovación, Agencia Estatal de Investigación, Proyectos de Generación de Conocimiento; supported by the European Regional Development Fund)., Xunta de Galicia; ED431B 2022/33, Xunta de Galicia; ED431G 2023/01, Xunta de Galicia; ED431C 2022/19, Xunta de Galicia; ED431G 2019/04
RUC. Repositorio da Universidade da Coruña
Publicación Contribución de congreso (ConferenceOutput)

MindWell: A Conversational Agent for Professional Depression Screening on Social Media

  • Bao, Eliseo
  • Pérez, Anxo
  • Parapar, Javier
Presented at: ECIR 2025, 47th European Conference on Information Retrieval. April 6–10, 2025, Lucca, Italy This version of the article has been accepted for publication, after peer review (when applicable) and is subject to Springer Nature’s AM terms of use, but is not the Version of Record and does not reflect post-acceptance improvements, or any corrections. The Version of Record is available online at: https://doi.org/10.1007/978-3-031-88720-8_9 Part of the book series: Lecture Notes in Computer Science (LNCS,volume 15576), [Abstract]: Depression is among the most prevalent mental health conditions, with an early and accurate diagnosis being essential for mitigating its effects. Yet, stigma often prevents individuals from seeking professional help. In this context, social media offers a unique resource for depression screening, as users frequently share, comment, and disclose their daily struggles, providing key insights into their mental health through online activity. However, the immense volume of data generated on these platforms presents a significant challenge, requiring substantial time and effort for mental health professionals to analyze. This demo paper introduces MindWell, an open-source conversational agent designed to support clinicians in identifying symptoms and emotions relevant to clinical assessments. MindWell uses a Retrieval-Augmented Generation (RAG) framework, incorporating a Large Language Model (LLM) based on Llama 3.1 and fine-tuned specifically for depression screening based on clinical symptom criteria, particularly the Beck Depression Inventory-II (BDI-II). By leveraging users’ social media history as informed and reliable context, MindWell is designed to answer questions formulated by clinicians, facilitating the review process. We collaborated with a professional psychologist to assess MindWell’s responses in a clinical setting, finding that the system effectively captures users’ depressive signs and shows promise for mental health support applications., This work has received support from projects: PLEC2021-007662 (MCIN/AEI/10.13039/501100011033 Ministerio de Ciencia e Innovación, European Union NextGenerationEU/PRTR) and PID2022-137061OB-C21 (MCIN/AEI/10.13039 /501100011033/, Ministerio de Ciencia e Innovación, ERDF A way of making Europe, by the European Union); Consellería de Educación, Universidade e Formación Profesional, Spain (grant number ED481A-2024-079 and accreditations 2019–2022 ED431G/ 01 and GPC ED431B 2022/33) and the European Regional Development Fund, which acknowledges the CITIC Research Center., Xunta de Galicia; ED481A-2024-079, Xunta de Galicia; ED431G/ 01, Xunta de Galicia; ED431B 2022/33
RUC. Repositorio da Universidade da Coruña
Publicación Contribución de congreso (ConferenceOutput)

How Discriminative Are Your Qrels? How To Study the Statistical Significance of Document Adjudication Methods

  • Otero, David
  • Parapar, Javier
  • Ferro, Nicola
[Abstract]: Creating test collections for offline retrieval evaluation requires human effort to judge documents' relevance. This expensive activity motivated much work in developing methods for constructing benchmarks with fewer assessment costs. In this respect, adjudication methods actively decide both which documents and the order in which experts review them, in order to better exploit the assessment budget or to lower it. Researchers evaluate the quality of those methods by measuring the correlation between the known gold ranking of systems under the full collection and the observed ranking of systems under the lower-cost one. This traditional analysis ignores whether and how the low-cost judgements impact on the statistically significant differences among systems with respect to the full collection. We fill this void by proposing a novel methodology to evaluate how the low-cost adjudication methods preserve the pairwise significant differences between systems as the full collection. In other terms, while traditional approaches look for stability in answering the question "is system A better than system B?", our proposed approach looks for stability in answering the question "is system A significantly better than system B?", which is the ultimate questions researchers need to answer to guarantee the generalisability of their results. Among other results, we found that the best methods in terms of ranking of systems correlation do not always match those preserving statistical significance., This work has received support from: (i) project PLEC2021-007662 (MCIN/AEI/10.13039/501100011033, Ministerio de Ciencia e Inno-vación, Agencia Estatal de Investigación, Plan de Recuperación, Transformación y Resiliencia, Unión Europea-Next GenerationEU); (ii) Programa de Ayudas para la Formación de Profesorado Universitario, grant number FPU20/02659 (Ministerio de Universidades); (iii) project PID2022-137061OB-C21 (Proyectos de Generación de Conocimiento, MCIN); (iv) project ED431-B 2022/33 (Xunta de Galicia/ERDF); (v) CAMEO, PRIN 2022 n. 2022ZLL7MW., Xunta de Galicia; ED431-B 2022/33
RUC. Repositorio da Universidade da Coruña
Publicación Contribución de congreso (ConferenceOutput)

Overview of eRisk 2025: Early Risk Prediction on the Internet (Extended Overview)

  • Parapar, Javier
  • Pérez, Anxo
  • Crestani, Fabio
  • Wang, Xi
[Abstract]: This paper presents an extended overview of eRisk 2025, the ninth edition of the CLEF lab on early risk detection. Since its beginnings, eRisk has served as a benchmark for assessing methodologies, evaluation metrics, and challenges in the early identification of personal risks, particularly within health and safety domains. The 2025 edition marks an important evolution, amplifying the lab’s scope toward problems that require richer contextual and conversational understanding. The first task, the only one preserved from last year, asks systems to rank sentences by their relevance to the BDI-II depression symptoms, enabling fine-grained retrieval of depressive cues. The second task reformulates early detection as a contextual decision problem. In this task, the full conversational thread, including the user’s posts and all the interactions from the rest of the people involved, is revealed incrementally. At each step, the models must decide whether sufficient evidence exists to predict depression for the user, thereby rewarding both accuracy and timeliness. Finally, the pilot task pioneers an interactive scenario: fine-tuned large language models engage participants in dialogue and must infer depressive signals from the evolving conversations, probing the feasibility and safety of conversational screening agents. Together, these three tasks continue to advance the field of early risk detection, open new research avenues and align the evaluation framework more closely with real-world conversational settings., The authors thank the financial support supplied by the grant PID2022-137061OB-C21 funded by MICIU/AEI/10.13039/501100011033 and by “ERDF/EU”. The authors also thank the funding supplied by the Consellería de Cultura, Educación, Formación Profesional e Universidades (accreditations ED431G 2023/01 and ED431C 2025/49) and the European Regional Development Fund, which acknowledges the CITIC, as a center accredited for excellence within the Galician University System and a member of the CIGUS Network, receives subsidies from the Department of Education, Science, Universities, and Vocational Training of the Xunta de Galicia. Additionally, it is co-financed by the EU through the FEDER Galicia 2021-27 operational program., Xunta de Galicia; ED431G 2023/01, Xunta de Galicia; ED431C 2025/49
RUC. Repositorio da Universidade da Coruña
Publicación Contribución de congreso (ConferenceOutput)

ReDSM5: A Reddit Dataset for DSM-5 Depression Detection

  • Bao, Eliseo
  • Pérez, Anxo
  • Parapar, Javier
Presented at: CIKM '25: The 34th ACM International Conference on Information and Knowledge Management, Seoul, Republic of Korea, November 10 - 14, 2025 © Autores | ACM 2025. This is the author's version of the work. It is posted here for your personal use. Not for redistribution. The definitive Version of Record was published in CIKM '25: Proceedings of the 34th ACM International Conference on Information and Knowledge Management, https://doi.org/10.1145/3746252.3761610, [Abstract]: Depression is a pervasive mental health condition that affects hundreds of millions of individuals worldwide, yet many cases remain undiagnosed due to barriers in traditional clinical access and pervasive stigma. Social media platforms, and Reddit in particular, offer rich, user-generated narratives that can reveal early signs of depressive symptomatology. However, existing computational approaches often label entire posts simply as depressed or not depressed, without linking language to specific criteria from the DSM-5, the standard clinical framework for diagnosing depression. This limits both clinical relevance and interpretability. To address this gap, we introduce ReDSM5, a novel Reddit corpus comprising 1484 long-form posts, each exhaustively annotated at the sentence level by a licensed psychologist for the nine DSM-5 depression symptoms. For each label, the annotator also provides a concise clinical rationale grounded in DSM-5 methodology. We conduct an exploratory analysis of the collection, examining lexical, syntactic, and emotional patterns that characterize symptom expression in social media narratives. Compared to prior resources, ReDSM5 uniquely combines symptom-specific supervision with expert explanations, facilitating the development of models that not only detect depression but also generate human-interpretable reasoning. We establish baseline benchmarks for both multi-label symptom classification and explanation generation, providing reference results for future research on detection and interpretability., This work was supported by the project PID2022-137061OB-C21 (MCIN/AEI/10.13039/501100011033, Ministerio de Ciencia e Innovación, ERDF, A way of making Europe by the European Union); the Consellería de Educación, Universidade e Formación Profesional, Spain (grant number ED481A-2024-079 and accreditations 2019- 2022 ED431G/01 and GRC ED431C 2025/49); and the European Regional Development Fund, which supports the CITIC Research Center., Xunta de Galicia; ED481A-2024-079, Xunta de Galicia; 2019-2022 ED431G/01, Xunta de Galicia; GRC ED431C 2025/49
RUC. Repositorio da Universidade da Coruña
Publicación Contribución de congreso (ConferenceOutput)

Hybrid Pooling with LLMs via Relevance Context Learning

  • Otero, David
  • Parapar, Javier
Presented at: SIGIR '26, 49th International ACM SIGIR Conference on Research and Development in Information Retrieval, July 20–24, 2026, Melbourne, Australia, [Abstract]: High-quality relevance judgements over large query sets are essential for evaluating Information Retrieval (IR) systems, yet manual annotation remains costly and time-consuming. Large Language Models (LLMs) have recently shown promise as automatic relevance assessors, but their reliability is still limited. Most existing approaches rely on zero-shot prompting or in-context learning (ICL) with a small number of labelled examples. However, standard ICL treats examples as independent instances and fails to explicitly capture the underlying relevance criteria of a topic, restricting its ability to generalise to unseen query-document pairs. To address this limitation, we introduce Relevance Context Learning (RCL), a novel framework that leverages human relevance judgements to explicitly model topic-specific relevance criteria. Rather than directly using labelled examples for in-context prediction, RCL first prompts an LLM (Instructor LLM) to analyse sets of judged query-document pairs and generate explicit narratives that describe what constitutes relevance for a given topic. These relevance narratives are then used as structured prompts to guide a second LLM (Assessor LLM) in producing relevance judgements. To evaluate RCL in a realistic data collection setting, we propose a hybrid pooling strategy in which a shallow depth-k pool from participating systems is judged by human assessors, while the remaining documents are labelled by LLMs. Experimental results demonstrate that RCL substantially outperforms zero-shot prompting and consistently improves over standard ICL. Overall, our findings indicate that transforming relevance examples into explicit, context-aware relevance narratives is a more effective way of exploiting human judgements for LLM-based IR dataset construction., All authors acknowledge funding from the Ministry of Science, Innovation and Universities of the Government of Spain (project PID2022-137061OB-C21, MCIN/AEI/10.13039/501100011033), as well as from the Department of Education, Science, Universities, andVocational Training of the Xunta de Galicia(grantGRCED431C 2025/49). CITIC, as a center accredited for excellence within the Galician University System and a member of the CIGUS Network, receives subsidies from the Department of Education, Science, Universities, and Vocational Training of the Xunta de Galicia. Additionally, it is co-financed by the EU through the FEDER Galicia 2021-27 operational program (Ref. ED431G 2023/01)., Xunta de Galicia; GRC ED431C 2025/49, Xunta de Galicia; ED431G 2023/01
RUC. Repositorio da Universidade da Coruña
Publicación Contribución de congreso (ConferenceOutput)

Lost in the Evidence? Reproducing Document Position and Context Size Effects in RAG

  • Gabín, Jorge
  • Pérez, Anxo
  • Parapar, Javier
Presented at: SIGIR '26: Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval, July 20–24, 2026, Melbourne, Australia, [Abstract]: Retrieval-Augmented Generation (RAG) systems rely on retrieved documents being concatenated into a model's input context, making both document ordering and context size critical yet controversial design choices. Prior work reports position-based effects such as lost in the middle and related long-context phenomena. However, empirical findings remain inconsistent and hard to reproduce across models, datasets, and evaluation protocols. In this paper, we present a systematic reproducibility study that revisits these claims and examines how they evolve with contemporary LLMs under a controlled evaluation framework. We first show that topic sampling is a major source of variance: small topic sets can mask or exaggerate ordering effects. Based on repeated subset sampling across multiple topic budgets, we provide a practical calibration procedure that identifies topic counts yielding stable trends at feasible cost. Using these fixed topic sets, we then reproduce and extend results on position sensitivity, re-evaluating lost in the middle and positional biases in modern LLMs. Then, we also study a more realistic RAG scenario in which relevance is mediated by a retriever rather than oracle access to ground-truth documents. In this setting, we re-examine a recent industry study and identify discrepancies to evaluation choices such as limited topic coverage and reliance on LLM-based judges. Finally, we conduct an analysis of how retrieval order and context size affect downstream LLM performance under imperfect retrieval. Our results demonstrate that both factors interact strongly with retrieval quality and model choice, and that conclusions drawn from idealised setups do not always transfer to real-world RAG pipelines. We release all code and configurations to support reproducibility and future work on robust RAG evaluation., All authors acknowledge funding from the Ministry of Science, Innovation and Universities of the Government of Spain (project PID2022-137061OB-C21, MCIN/AEI/10.13039/501100011033), as well as from the Department of Education, Science, Universities, and Vocational Training of the Xunta de Galicia (grant GRC ED431C 2025/49). CITIC, as a center accredited for excellence within the Galician University System and a member of the CIGUS Network, receives subsidies from the Department of Education, Science, Universities, and Vocational Training of the Xunta de Galicia. Additionally, it is co-financed by the EU through the FEDER Galicia 2021-27 operational program (Ref. ED431G 2023/01)., Xunta de Galicia; GRC ED431C 2025/49, Xunta de Galicia; ED431G 2023/01
RUC. Repositorio da Universidade da Coruña
Publicación Contribución de congreso (ConferenceOutput)

eRisk 2026: Tasks on Symptoms Ranking, Contextual and Conversational Approaches for Early Mental Health Detection

  • Pérez, Anxo
  • Parapar, Javier
  • Wang, Xi
  • Crestani, Fabio
Presented at: ECIR 2026, Advances in Information Retrieval. 29 March - 2 April 2026, Delft, The Netherlands This version of the article has been accepted for publication, after peer review (when applicable) and is subject to Springer Nature’s AM terms of use, but is not the Version of Record and does not reflect post-acceptance improvements, or any corrections. The Version of Record is available online at: http://dx.doi.org/10.1007/978-3-032-21321-1_33 Part of the book series: Lecture Notes in Computer Science (LNCS, volume 16486), [Abstract]: Since its foundation in 2017, the eRisk CLEF Lab has pioneered research in early risk detection on the Internet, focusing on mental health challenges such as depression, anorexia, and pathological gambling. Over the years, participants have contributed to the development of detection models and exploited the datasets we constructed to advance this critical area. In 2026, which marks the tenth edition of the lab, we continue this trajectory with three tasks that emphasize conversational and contextual modeling as well as symptom-oriented retrieval. The first task, Conversational Depression Detection, introduces the challenge of identifying depression through interactions with fine-tuned Large Language Models (LLMs) personas. The second task, Contextualised Early Detection of Depression, focuses on user-level classification by analyzing full conversational contexts, with participants engaging iteratively in natural interactions. Finally, the third task, ADHD Symptom Sentence Ranking, expands our scope beyond depression by requiring systems to rank sentences according to their relevance to the symptoms defined in the Adult ADHD Self-Report Scale. This paper outlines the progress of the lab to date, introduces the three tasks of eRisk 2026, and discusses our innovative plans for promoting research on mental health challenges., This work was supported by the project PID2022-137061OB-C21 (MCIN/AEI/10.13039/501100011033, Ministerio de Ciencia e Innovación, ERDF, A way of making Europe by the European Union); the Consellería de Educación, Universidade e Formación Profesional, Spain (accreditations 2019–2022 ED431G/01 and GRC ED431C 2025/49); and the European Regional Development Fund, which supports the CITIC Research Center., Xunta de Galicia; ED431G/01, Xunta de Galicia; ED431C 2025/49
RUC. Repositorio da Universidade da Coruña
Publicación Contribución de congreso (ConferenceOutput)

TalkDep: Clinically Grounded LLM Personas for Conversation-Centric Depression Screening

  • Wang, Xi
  • Pérez, Anxo
  • Parapar, Javier
  • Crestani, Fabio
Presented at: CIKM '25: The 34th ACM International Conference on Information and Knowledge Management, Seoul Republic of Korea, November 10 - 14, 2025., [Abstract]: The increasing demand for mental health services has outpaced the availability of real training data to develop clinical professionals, leading to limited support for the diagnosis of depression. This shortage has motivated the development of simulated or virtual patients to assist in training and evaluation, but existing approaches often fail to generate clinically valid, natural, and diverse symptom presentations. In this work, we embrace the recent advanced language models as the backbone and propose a novel clinician-in-the-loop patient simulation pipeline, TalkDep, with access to diversified patient profiles to develop simulated patients. By conditioning the model on psychiatric diagnostic criteria, symptom severity scales, and contextual factors, our goal is to create authentic patient responses that can better support diagnostic model training and evaluation. We verify the reliability of these simulated patients with thorough assessments conducted by clinical professionals. The availability of validated simulated patients offers a scalable and adaptable resource for improving the robustness and generalisability of automatic depression diagnosis systems., This work was supported by the project PID2022-137061OB-C21 (MCIN/AEI/10.13039/501100011033, Ministerio de Ciencia e Inno- vación, ERDF, A way of making Europe by the European Union); the Consellería de Educación, Universidade e Formación Profesional, Spain (accreditations 2019-2022 ED431G/01 and GRC ED431C 2025/49); University of Alberta and University of Sheffield Seed Grant (X/016279- 14); and the European Regional Development Fund, which supports the CITIC Research Center., Xunta de Galicia; ED431G/01, Xunta de Galicia; ED431C 2025/49
RUC. Repositorio da Universidade da Coruña
Publicación Contribución de congreso (ConferenceOutput)

LLM-Assisted Pseudo-Relevance Feedback

  • Otero, David
  • Parapar, Javier
The conference was held in Delft, The Netherlands, from 29 March to 2 April 2026, [Abstract]: Query expansion is a long-standing technique to mitigate vocabulary mismatch in ad hoc Information Retrieval. Pseudo-relevance feedback methods, such as RM3, estimate an expanded query model from the top-ranked documents, but remain vulnerable to topic drift when early results include noisy or tangential content. Recent approaches instead prompt Large Language Models to generate synthetic expansions or query variants. While effective, these methods risk hallucinations and misalignment with collection-specific terminology. We propose a hybrid alternative that preserves the robustness and interpretability of classical PRF while leveraging LLM semantic judgement. Our method inserts an LLM-based filtering stage prior to RM3 estimation: the LLM judges the documents in the initial top-k ranking, and RM3 is computed only over those accepted as relevant. This simple intervention improves over blind PRF and a strong baseline across several datasets and metrics., The authors thank the financial support supplied by the grant PID2022-137061OB-C21 funded by MI-CIU/AEI/10.13039/501100011033 and by “ERDF/EU”. The authors also thank the funding supplied by the Consellería de Cultura, Educación, Formación Profesional e Universidades (accreditations ED431G 2023/01 and ED431C 2025/49) and the European Regional Development Fund, which acknowledges the CITIC, as a center accredited for excellence within the Galician University System and a member of the CIGUS Network, receives subsidies from the Department of Education, Science, Universities, and Vocational Training of the Xunta de Galicia. Additionally, it is co-financed by the EU through the FEDER Galicia 2021-27 operational program (Ref. ED431G 2023/01)., Xunta de Galicia; ED431G 2023/01, Xunta de Galicia; ED431C 2025/49, Xunta de Galicia; ED431G 2023/01
RUC. Repositorio da Universidade da Coruña
Publicación Contribución de congreso (ConferenceOutput)

Delving into the Depths: Evaluating Depression Severity through BDI-biased Summaries

  • Aragón, Mario Ezra
  • Parapar, Javier
  • Losada, David E.
[Abstract]: Depression is a global concern suffered by millions of people, significantly impacting their thoughts and behavior. Over the years, heightened awareness, spurred by health campaigns and other initiatives, has driven the study of this disorder using data collected from social media platforms. In our research, we aim to gauge the severity of symptoms related to depression among social media users. The ultimate goal is to estimate the user’s responses to a well-known standardized psychological questionnaire, the Beck Depression Inventory-II (BDI). This is a 21-question multiple-choice self-report inventory that covers multiple topics about how the subject has been feeling. Mining users’ social media interactions and understanding psychological states represents a challenging goal. To that end, we present here an approach based on search and summarization that extracts multiple BDI-biased summaries from the thread of users’ publications. We also leverage a robust large language model to estimate the potential answer for each BDI item. Our method involves several steps. First, we employ a search strategy based on sentence similarity to obtain pertinent extracts related to each topic in the BDI questionnaire. Next, we compile summaries of the content of these groups of extracts. Last, we exploit chatGPT to respond to the 21 BDI questions, using the summaries as contextual information in the prompt. Our model has undergone rigorous evaluation across various depression datasets, yielding encouraging results. The experimental report includes a comparison against an assessment done by expert humans and competes favorably with state-of-the-art methods., The second author thanks the financial support supplied by the Consellería de Cultura, Educación, Formación Profesional e Universidades (accreditation 2019-2022 ED431G/01, ED431B 2022/33) and the European Regional Development Fund, which acknowledges the CITIC Research Center in ICT of the University of A Coruña as a Research Center of the Galician University System and the project PID2022-137061OB-C21 (Ministerio de Ciencia e Innovación, Agencia Estatal de Investigación, Proyectos de Generación de Conocimiento; supported by the European Regional Development Fund). The first and third authors thank: i) the financial support supplied by the Consellería de Cultura, Educación, Formación Profesional e Universidades (accreditation 2019-2022 ED431G-2019/04, ED431C 2022/19) and the European Regional Development Fund, which acknowledges the CiTIUS- Research Center in Intelligent Technologies of the University of Santiago de Compostela as a Research Center of the Galician University System, and ii) the financial support supplied by project PID2022-137061OB-C22 (Ministerio de Ciencia e Innovación, Agencia Estatal de Investigación, Proyectos de Generación de Conocimiento; supported by the European Regional Development Fund). The third author thanks the financial support obtained from project SUBV23/00002 (Ministerio de Consumo, Subdirección General de Regulación del Juego). The authors also thank the funding of project PLEC2021-007662 (MCIN/AEI/10.13039/501100011033, Ministerio de Ciencia e Innovación, Agencia Estatal de Investigación, Plan de Recuperación, Transformación y Resiliencia, Unión Europea-Next Generation EU)., Xunta de Galicia; ED431G/01, Xunta de Galicia; ED431B 2022/33, Xunta de Galicia; ED431G-2019/04, Xunta de Galicia; ED431C 2022/19
RUC. Repositorio da Universidade da Coruña
Publicación Contribución de congreso (ConferenceOutput)

Semantic Similarity Models for Depression Severity Estimation

  • Pérez, Anxo
  • Warikoo, Neha
  • Wang, Kexin
  • Parapar, Javier
  • Gurevych, Iryna
Presented at: EMNLP 2023 Conference on Empirical Methods in Natural Language Processing, Singapore, December 6th to December 10th., [Abstract]: Depressive disorders constitute a severe public health issue worldwide. However, public health systems have limited capacity for case detection and diagnosis. In this regard, the widespread use of social media has opened up a way to access public information on a large scale. Computational methods can serve as support tools for rapid screening by exploiting this user-generated social media content. This paper presents an efficient semantic pipeline to study depression severity in individuals based on their social media writings. We select test user sentences for producing semantic rankings over an index of representative training sentences corresponding to depressive symptoms and severity levels. Then, we use the sentences from those results as evidence for predicting symptoms severity. For that, we explore different aggregation methods to answer one of four Beck Depression Inventory (BDI-II) options per symptom. We evaluate our methods on two Reddit-based benchmarks, achieving improvement over state of the art in terms of measuring depression level., This work has received support from projects: PLEC2021-007662, PID2022-137061OB-C21 (MCIN/AEI/10.13039/501100011033, Ministerio de Ciencia e Innovación, Agencia Estatal de Investigación, Proyectos de Generación de Conocimiento; suppported by the European Regional Development Fund); Consellería de Educación, Universidade e Formación Profesional, Spain (accreditation 2019–2022 ED431G/01 and GPC ED431B 2022/33) and the European Regional Development Fund, which acknowledges the CITIC Research Center, an ICT of the University of A Coruña as a Research Center of the Galician University System. The first author also acknowledges the predoctoral grant contract ref. ED481 A 2021/034 funded by Xunta de Galicia and the European Social Fund (ESF). From UKP Lab, this work has been funded by the LOEWE Distinguished Chair “Ubiquitous Knowledge Processing” (LOEWE initiative, Hesse, Germany)., Xunta de Galicia; ED481 A 2021/034, Xunta de Galicia; ED431G/01, Xunta de Galicia; ED431B 2022/33