Skip to main content

EVOLUCION CONTINUA Y EFICIENTE DE SISTEMAS DE ML: UN ENFOQUE DIRIGIDO POR EL ECOSISTEMA

PID2024-156019OB-I00

•
Funding agency name Agencia Estatal de Investigación
Funding agency acronym AEI
Program Programa Estatal para la Investigación y el Desarrollo Experimental
Subprogram Subprograma Estatal de Generación de Conocimiento Científico-Técnico y Desarrollo Experimental
Call Proyectos de I+D+i (Generación de Conocimiento y Retos Investigación)
Call year 2024
State Plan 2024-2027 Plan Estatal de Investigación Científica y Técnica y de Innovación
Beneficiary centre UNIVERSITAT POLITECNICA DE CATALUNYA

Resultados relacionados

Resultados totales (Incluyendo duplicados): 2
Encontrada(s) 1 página(s)

Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

What’s in a benchmark? The case of SWE-bench in automated program repair

  • Martínez Martínez, Matías-Sebastián
  • Franch Gutiérrez, Javier
The rapid progress in Automated Program Repair (APR) has been fueled by advances in AI, particularly large language models (LLMs) and agent-based systems. SWE-Bench is a benchmark designed to evaluate repair systems using real issues mined from popular open-source Python repositories. Its public leaderboards—SWE-Bench Lite and Verified—have become central platforms for tracking progress and comparing solutions. In this paper, we present the first comprehensive study of these two leaderboards, examining who is submitting solutions, the products behind the submissions, the LLMs employed, and the openness of the approaches. We analyze 79 entries submitted to Lite leaderboard and 133 to Verified. Our results show that most entries on both leaderboards originate from industry, particularly small companies and large publicly traded companies. These submissions often achieve top results, although academic contributions—typically open source—also remain competitive. We also find a clear dominance of proprietary LLMs, especially Claude family, with state-of-the-art results on both leaderboards currently achieved by Claude 4 Sonnet. These findings offer insights into the SWE-Bench ecosystem that can guide greater transparency and diversity in future benchmark-driven research., This paper has been funded by the “Ramon y Cajal” Fellowship (RYC2021-031523-I), by grant PID2024-156019OB-I00 funded by MICIU/AEI/10.13039/501100011033 and by ERDF, EU., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

A tool for automatically cataloguing and selecting pre-trained models and datasets for software engineering

  • González Álvarez, Alexandra
  • Cerezo Consuegra, Óscar
  • Franch Gutiérrez, Javier
  • Martínez Fernández, Silverio Juan
The rapid growth of machine learning assets has made it increasingly difficult for software engineers to identify models and datasets that match their specific needs. Browsing large registries, such as Hugging Face, is time-consuming, error-prone, and rarely tailored to Software Engineering (SE) tasks. We present MLAssetSelection, a web application that automatically extracts SE assets and supports four key functionalities: (i) a configurable leaderboard for ranking models across multiple benchmarks and metrics; (ii) requirementsbased selection of models and datasets; (iii) real-time automated updates through scheduled jobs that keep asset information current; and (iv) user-centric features including login, personalized asset lists, and configurable alert notifications. A demonstration video is available at https://youtu.be/t6CJ6P9asV4., This work was supported by the FI-STEP predoctoral fellowship (grant no. 2025 STEP 00407) awarded by AGAUR (Government of Catalonia) and co-funded by the European Social Fund Plus (FSE+) of Catalonia 2021–2027, and by Grant PID2024-156019OB-I00 funded by MICIU/AEI/10.13039/501100011033 and by ERDF, EU., Peer Reviewed, Postprint (published version)