RUC. Repositorio da Universidade da Coruña
Publicación
Contribución de congreso (ConferenceOutput)
Beaver: Efficiently Building Test Collections for Novel Tasks
Proceedings of the Joint Conference of the Information Retrieval Communities in Europe (CIRCLE 2020) Samatan, Gers, France, July 6-9, 2020, published at http://ceur-ws.org., [Abstract]: Evaluation is a mandatory task for Information Retrieval research.
Under the Cranfield paradigm, this evaluation needs test collections.
The creation of these is a time and resource-consuming process. At
the same time, new tasks and models are continuously appearing.
These tasks demand the building of new test collections. Typically,
the researchers organize TREC-like competitions for building these
evaluation benchmarks. This is very expensive, both for the organizers and for the participants. In this paper, we present a platform
to easily and cheaply build datasets for Information Retrieval evaluation without the need of organizing expensive campaigns. In
particular, we propose the simulation of participant systems and
the use of pooling strategies to make the most of the assessor’s
work. Our platform is aimed to cover the whole process of building
the test collection, from document gathering to judgment creation., This work was supported by project RTI2018-093336-B-C22 (MCIU
& ERDF), project GPC ED431B 2019/03 (Xunta de Galicia & ERDF),
and accreditation ED431G 2019/01 (Xunta de Galicia & ERDF)., Xunta de Galicia; ED431B 2019/03, Xunta de Galicia; ED431G 2019/01