Pasar al contenido principal

MODELOS DE LENGUAJE PROBABILISTICOS PARA RANKINGS PERSONALIZADOS EN SISTEMAS DE ACCESO A LA INFORMACION

TIN2015-64282-R

•
Nombre agencia financiadora Ministerio de Economía y Competitividad
Acrónimo agencia financiadora MINECO
Programa Programa Estatal de I+D+i Orientado a los Retos de la Sociedad
Convocatoria Proyectos I+D+i Retos de Investigación
Año convocatoria 2015
Plan Estatal 2013-2016 Plan Estatal de Investigación Científica y Técnica y de Innovación
Unidad de gestión Dirección General de Investigación Científica y Técnica
Centro beneficiario UNIVERSIDADE DA CORUÑA
Identificador persistente http://dx.doi.org/10.13039/501100003329

Resultados relacionados

Resultados totales (Incluyendo duplicados): 5
Encontrada(s) 1 página(s)

Minerva. Repositorio Institucional de la Universidad de Santiago de Compostela
Publicación Artículo científico (JournalArticle)

When to stop making relevance judgments? A study of stopping methods for building information retrieval test collections

  • Losada Carril, David Enrique
  • Parapar, Javier
  • Barreiro, Álvaro
In information retrieval evaluation, pooling is a well‐known technique to extract a sample of documents to be assessed for relevance. Given the pooled documents, a number of studies have proposed different prioritization methods to adjudicate documents for judgment. These methods follow different strategies to reduce the assessment effort. However, there is no clear guidance on how many relevance judgments are required for creating a reliable test collection. In this article we investigate and further develop methods to determine when to stop making relevance judgments. We propose a highly diversified set of stopping methods and provide a comprehensive analysis of the usefulness of the resulting test collections. Some of the stopping methods introduced here combine innovative estimates of recall with time series models used in Financial Trading. Experimental results on several representative collections show that some stopping methods can reduce up to 95% of the assessment effort and still produce a robust test collection. We demonstrate that the reduced set of judgments can be reliably employed to compare search systems using disparate effectiveness metrics such as Average Precision, NDCG, P@100, and Rank Biased Precision. With all these measures, the correlations found between full pool rankings and reduced pool rankings is very high
Investigo. Repositorio Institucional de la Universidade de Vigo
Publicación Artículo científico (JournalArticle)

ALBAYZIN 2018 spoken term detection evaluation: a multi-domain international evaluation in Spanish

  • Tejedor Noguerales, Javier
  • Toledano, Doroteo T.
  • López Otero, Paula
  • Docío Fernández, Laura
  • Montalvo, Ana R.
  • Ramírez, José M.
  • Peñagarikano, Mikel
  • Rodríguez Fuentes, Luis Javier
Search on speech (SoS) is a challenging area due to the huge amount of information stored in audio and video repositories. Spoken term detection (STD) is an SoS-related task aiming to retrieve data from a speech repository given a textual representation of a search term (which can include one or more words). This paper presents a multi-domain internationally open evaluation for STD in Spanish. The evaluation has been designed carefully so that several analyses of the main results can be carried out. The evaluation task aims at retrieving the speech files that contain the terms, providing their start and end times, and a score that reflects the confidence given to the detection. Three different Spanish speech databases that encompass different domains have been employed in the evaluation: the MAVIR database, which comprises a set of talks from workshops; the RTVE database, which includes broadcast news programs; and the COREMAH database, which contains 2-people spontaneous speech conversations about different topics. We present the evaluation itself, the three databases, the evaluation metric, the systems submitted to the evaluation, the results, and detailed post-evaluation analyses based on some term properties (within-vocabulary/out-of-vocabulary terms, single-word/multi-word terms, and native/foreign terms). Fusion results of the primary systems submitted to the evaluation are also presented. Three different research groups took part in the evaluation, and 11 different systems were submitted. The obtained results suggest that the STD task is still in progress and performance is highly sensitive to changes in the data domain., Xunta de Galicia | Ref. ED431B 2016/035, Xunta de Galicia | Ref. ED431B 2019/003, Xunta de Galicia | Ref. 2014/024, Xunta de Galicia | Ref. ED431G/01, Xunta de Galicia | Ref. ED431G/04, Ministerio de Economía y Competitividad | Ref. TEC2015-68172-C2-1-P, Ministerio de Economía y Competitividad | Ref. TEC2015-65345-P, Ministerio de Economía y Competitivad | Ref. TIN2015-64282-R, Agencia Estatal de Investigación | Ref. RTI2018-093336-B-C22
Investigo. Repositorio Institucional de la Universidade de Vigo
Publicación Artículo científico (JournalArticle)

ALBAYZIN Query-by-example spoken term detection 2016 evaluation

  • Tejedor Noguerales, Javier
  • Toledano, Doroteo T
  • López Otero, Paula
  • Docío Fernández, Laura
  • Proença, Jorge
  • Perdigão, Fernando
  • García Granada, Fernando
  • Sanchís Arnal, Emilio
  • Pompili, Anna
  • Abad Gareta, Alberto
Query-by-example Spoken Term Detection (QbE STD) aims to retrieve data from a speech repository given an acoustic (spoken) query containing the term of interest as the input. This paper presents the systems submitted to the ALBAYZIN QbE STD 2016 Evaluation held as a part of the ALBAYZIN 2016 Evaluation Campaign at the IberSPEECH 2016 conference. Special attention was given to the evaluation design so that a thorough post-analysis of the main results could be carried out. Two different Spanish speech databases, which cover different acoustic and language domains, were used in the evaluation: the MAVIR database, which consists of a set of talks from workshops, and the EPIC database, which consists of a set of European Parliament sessions in Spanish. We present the evaluation design, both databases, the evaluation metric, the systems submitted to the evaluation, the results, and a thorough analysis and discussion. Four different research groups participated in the evaluation, and a total of eight template matching-based systems were submitted. We compare the systems submitted to the evaluation and make an in-depth analysis based on some properties of the spoken queries, such as query length, single-word/multi-word queries, and in-language/out-of-language queries., Fundação para a Ciência e Tecnologia | Ref. UID/EEA/50008/2013, Fundação para a Ciência e Tecnologia | Ref. UID/CEC/50021/2013, Fundação para a Ciência e Tecnologia | Ref. SFRH/BD/97187/2013, Fundação para a Ciência e Tecnologia | Ref. SFRH/BD/97204/2013, Ministerio de Economía y Competitividad | Ref. TEC2015-68172-C2-1-P, Ministerio de Economía y Competitividad | Ref. TIN2015-64282-R, Ministerio de Economía y Competitividad | Ref. TEC2015-65345-P, Xunta de Galicia | Ref. ED431G/04, Xunta de Galicia | Ref. ED431G/01, Xunta de Galicia | Ref. 2014/024
RUC. Repositorio da Universidade da Coruña
Publicación Artículo científico (JournalArticle)

ALBAYZIN Query-by-example Spoken Term Detection 2016 evaluation

  • Tejedor, Javier
  • Toledano, Doroteo T.
  • López-Otero, Paula
  • Docío-Fernández, Laura
  • Proença, Jorge
  • Perdigão, Fernando
  • García-Granada, Fernando
  • Sanchis, Emilio
  • Pompili, Anna
  • Abad, Alberto
[Abstract]: Query-by-example Spoken Term Detection (QbE STD) aims to retrieve data from a speech repository given an acoustic (spoken) query containing the term of interest as the input. This paper presents the systems submitted to the ALBAYZIN QbE STD 2016 Evaluation held as a part of the ALBAYZIN 2016 Evaluation Campaign at the IberSPEECH 2016 conference. Special attention was given to the evaluation design so that a thorough post-analysis of the main results could be carried out. Two different Spanish speech databases, which cover different acoustic and language domains, were used in the evaluation: the MAVIR database, which consists of a set of talks from workshops, and the EPIC database, which consists of a set of European Parliament sessions in Spanish. We present the evaluation design, both databases, the evaluation metric, the systems submitted to the evaluation, the results, and a thorough analysis and discussion. Four different research groups participated in the evaluation, and a total of eight template matching-based systems were submitted. We compare the systems submitted to the evaluation and make an in-depth analysis based on some properties of the spoken queries, such as query length, single-word/multi-word queries, and in-language/out-of-language queries., This work was partially supported by Fundação para a Ciência e Tecnologia (FCT) under the projects UID/EEA/50008/2013 (pluriannual funding in the scope of the LETSREAD project) and UID/CEC/50021/2013, and Grant SFRH/BD/97187/2013. Jorge Proença is supported by the SFRH/BD/97204/2013 FCT Grant. This work was also supported by the Galician Government (‘Centro singular de investigación de Galicia’ accreditation 2016-2019 ED431G/01 and the research contract GRC2014/024 (Modalidade: Grupos de Referencia Competitiva 2014)), the European Regional Development Fund (ERDF), the projects “DSSL: Redes Profundas y Modelos de Subespacios para Detección y Seguimiento de Locutor, Idioma y Enfermedades Degenerativas a partir de la Voz” (TEC2015-68172-C2-1-P) and the TIN2015-64282-R funded by Ministerio de Economía y Competitividad in Spain, the Spanish Government through the project "TraceThem" (TEC2015-65345-P), and AtlantTIC ED431G/04., Portugal. Fundação para a Ciência e a Tecnologia; UID/EEA/50008/2013, Portugal. Fundação para a Ciência e a Tecnologia; UID/CEC/50021/2013, Portugal. Fundação para a Ciência e a Tecnologia; SFRH/BD/97187/2013, Portugal. Fundação para a Ciência e a Tecnologia; SFRH/BD/97204/2013, Xunta de Galicia; ED431G/01, Xunta de Galicia; GRC2014/024, Xunta de Galicia; ED431G/04
RUC. Repositorio da Universidade da Coruña
Publicación Artículo científico (JournalArticle)

Using Score Distributions to Compare Statistical Significance Tests for Information Retrieval Evaluation, Compare statistical significance tests for information retrieval evaluation

  • Parapar, Javier
  • Losada, David E.
  • Presedo-Quindimil, Manuel-Antonio
  • Barreiro, Álvaro
This is the peer reviewed version of the following article: Parapar, J., Losada, D.E., Presedo-Quindimil, M.A. and Barreiro, A. (2020), 'Using Score Distributions to Compare Statistical Significance Tests for Information Retrieval Evaluation', Journal of the Association for Information Science and Technology, 71: 98-113, which has been published in final form at https://doi.org/10.1002/asi.24203. This article may be used for non-commercial purposes in accordance with Wiley Terms and Conditions for Use of Self-Archived Versions. This article may not be enhanced, enriched or otherwise transformed into a derivative work, without express permission from Wiley or by statutory rights under applicable legislation. Copyright notices must not be removed, obscured or modified. The article must be linked to Wiley’s version of record on Wiley Online Library and any embedding, framing or otherwise making available the article or pages thereof by third parties from platforms, services and websites other than Wiley Online Library must be prohibited., [Abstract] Statistical significance tests can provide evidence that the observed difference in performance between two methods is not due to chance. In Information Retrieval, some studies have examined the validity and suitability of such tests for comparing search systems.We argue here that current methods for assessing the reliability of statistical tests suffer from some methodological weaknesses, and we propose a novel way to study significance tests for retrieval evaluation. Using Score Distributions, we model the output of multiple search systems, produce simulated search results from such models, and compare them using various significance tests. A key strength of this approach is that we assess statistical tests under perfect knowledge about the truth or falseness of the null hypothesis. This new method for studying the power of significance tests in Information Retrieval evaluation is formal and innovative. Following this type of analysis, we found that both the sign test and Wilcoxon signed test have more power than the permutation test and the t-test. The sign test and Wilcoxon signed test also have a good behavior in terms of type I errors. The bootstrap test shows few type I errors, but it has less power than the other methods tested., Xunta de Galicia; GPC 2016/035, Xunta de Galicia; ED431G/01, Xunta de Galicia; ED431G/08, This work has received financial support from the "Ministerio de Economía y Competitividad" of the Government of Spain and FEDER Funds under the research project TIN2015-64282-R, Xunta de Galicia (project GPC 2016/035), and Xunta de Galicia – "Consellería de Cultura, Educación e Ordenación Universitaria" and the European Regional Development Fund (ERDF) through the following 2016-2019 accreditations: ED431G/01 ("Centro singular de investigación de Galicia") and ED431G/08.