Skip to main content

EVOLUCION CONTINUA Y EFICIENTE DE SISTEMAS DE ML: UN ENFOQUE DIRIGIDO POR EL ECOSISTEMA

PID2024-156019OB-I00

•
Funding agency name Agencia Estatal de Investigación
Funding agency acronym AEI
Program Programa Estatal para la Investigación y el Desarrollo Experimental
Subprogram Subprograma Estatal de Generación de Conocimiento Científico-Técnico y Desarrollo Experimental
Call Proyectos de I+D+i (Generación de Conocimiento y Retos Investigación)
Call year 2024
State Plan 2024-2027 Plan Estatal de Investigación Científica y Técnica y de Innovación
Beneficiary centre UNIVERSITAT POLITECNICA DE CATALUNYA

Resultados relacionados

Resultados totales (Incluyendo duplicados): 3
Encontrada(s) 1 página(s)

Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

What’s in a benchmark? The case of SWE-bench in automated program repair

  • Martínez Martínez, Matías-Sebastián
  • Franch Gutiérrez, Javier
The rapid progress in Automated Program Repair (APR) has been fueled by advances in AI, particularly large language models (LLMs) and agent-based systems. SWE-Bench is a benchmark designed to evaluate repair systems using real issues mined from popular open-source Python repositories. Its public leaderboards—SWE-Bench Lite and Verified—have become central platforms for tracking progress and comparing solutions. In this paper, we present the first comprehensive study of these two leaderboards, examining who is submitting solutions, the products behind the submissions, the LLMs employed, and the openness of the approaches. We analyze 79 entries submitted to Lite leaderboard and 133 to Verified. Our results show that most entries on both leaderboards originate from industry, particularly small companies and large publicly traded companies. These submissions often achieve top results, although academic contributions—typically open source—also remain competitive. We also find a clear dominance of proprietary LLMs, especially Claude family, with state-of-the-art results on both leaderboards currently achieved by Claude 4 Sonnet. These findings offer insights into the SWE-Bench ecosystem that can guide greater transparency and diversity in future benchmark-driven research., This paper has been funded by the “Ramon y Cajal” Fellowship (RYC2021-031523-I), by grant PID2024-156019OB-I00 funded by MICIU/AEI/10.13039/501100011033 and by ERDF, EU., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Contribución de congreso (ConferenceOutput)

A tool for automatically cataloguing and selecting pre-trained models and datasets for software engineering

  • González Álvarez, Alexandra
  • Cerezo Consuegra, Óscar
  • Franch Gutiérrez, Javier
  • Martínez Fernández, Silverio Juan
The rapid growth of machine learning assets has made it increasingly difficult for software engineers to identify models and datasets that match their specific needs. Browsing large registries, such as Hugging Face, is time-consuming, error-prone, and rarely tailored to Software Engineering (SE) tasks. We present MLAssetSelection, a web application that automatically extracts SE assets and supports four key functionalities: (i) a configurable leaderboard for ranking models across multiple benchmarks and metrics; (ii) requirementsbased selection of models and datasets; (iii) real-time automated updates through scheduled jobs that keep asset information current; and (iv) user-centric features including login, personalized asset lists, and configurable alert notifications. A demonstration video is available at https://youtu.be/t6CJ6P9asV4., This work was supported by the FI-STEP predoctoral fellowship (grant no. 2025 STEP 00407) awarded by AGAUR (Government of Catalonia) and co-funded by the European Social Fund Plus (FSE+) of Catalonia 2021–2027, and by Grant PID2024-156019OB-I00 funded by MICIU/AEI/10.13039/501100011033 and by ERDF, EU., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publication Artículo científico (JournalArticle)

MLSToolbox Code Generator: A tool for generating quality ML pipelines for ML systems

  • Gómez Seoane, Cristina
  • López Cuesta, Lidia
  • Ayala Martínez, Claudia Patricia
  • Lopez Cuesta, Miguel
Machine learning-based systems play a critical and increasingly pervasive role in various aspects of daily life. Despite the growing recognition of the importance of producing high-quality code for Machine Learning (ML) pipelines to ensure proper evolution, maintenance, and reusability, actionable guidance at the design and implementation levels remains scarce. This paper introduces MLSToolbox Code Generator, a low-code tool designed to support data scientists in graphically defining ML pipelines and generating their corresponding Python code. The tool leverages core Software Engineering design principles to promote high-quality Python code. Through a detailed example, we demonstrate how data scientists can use the tool. The flexible and extensible architecture of the tool enables data scientists to customize ML pipeline generation to meet domain-specific requirements, fostering greater efficiency and adaptability in ML workflows., This work was supported by the “Spanish Ministerio de Ciencia e Innovación” under project / funding scheme PID2024-156019OB-I00., Peer Reviewed, Postprint (published version)