Skip to main content

Resultados relacionados

Resultados totales (Incluyendo duplicados): 8
Encontrada(s) 1 página(s)

RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publication Artículo científico (JournalArticle)

Delta GUI change detection using inferred models

  • Pastor-Ricós, Fernando|||0000-0002-5790-193X
  • Marín, Beatriz|||0000-0001-8025-0023
  • Vos, Tanja Ernestina|||0000-0002-6003-9113
  • Neeft, Rick
  • Aho, Pekka
[EN] Recent software development methodologies emphasize iterative and incremental evolution to align with stakeholders' needs. This perpetual and rapid software evolution demands ongoing research into verification practices and technologies that ensure swift responsiveness and effective management of software delta increments. Strategies such as code review have been widely adopted for development and verification, ensuring readability and consistency in the delta increments of software projects. However, the integration of techniques to detect and visually report delta changes within the Graphical User Interface (GUI) software applications remains an underutilized process. In this paper, we set out to achieve two objectives. First, we aim to conduct a comprehensive review of existing studies concerning GUI change detection in desktop, web, and mobile applications to recognize common practices. Second, we introduce a novel change detection tool capable of highlighting delta GUI changes for this diverse range of applications. To accomplish our first objective, we performed a systematic mapping of the literature using the Scopus database. To address the second objective, we designed and developed a GUI change detection tool. This tool simultaneously transits and compares state models inferred by a scriptless testing tool, enabling the detection and highlighting of GUI changes to detect the widgets or functionalities that have been added, removed, or modified. Our study reveals the existence of a multitude of techniques for change detection in specific GUI systems with different objectives. However, there is no widely adopted technique suitable for the diverse range of existing desktop, web, and mobile applications. Our tool and findings demonstrate the effectiveness of using inferred state models to highlight between 8 and 20 GUI changes in software delta increments containing a large number of changes over months and between 4 and 6 GUI changes in delta increments of small iterations performed over multiple weeks. Moreover, some of these changes were recognized by the software developers as GUI failures that required a fix. Finally, we expose the motivation for using this technique to help developers and testers analyze GUI changes to validate delta increments and detect potential GUI failures, thereby fostering knowledge dissemination and paving the way to standard practices., This research has been funded by the project European Innovation Alliance for Testing Education (ENACTEST) , ERASMUS+ number 101055874, 2022-2025.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publication Artículo científico (JournalArticle)

The Impact of Model Simulation on Semantic Conceptual Model Quality for Novice Modelers

  • Cammaerts, Felix
  • Snoeck, Monique
  • Marín, Beatriz|||0000-0001-8025-0023
[EN] Purpose:Conceptual modeling is crucial for information systems development, yet novice modelers often struggle with ensuring a high semantic quality, leading to flawed system designs. This study investigates the impact of Feedback-enabled Interactive Rapid Prototyping (FIRP) simulation on novice modelers' ability to identify and correct semantic errors in conceptual models.Methods:We analyzed data from 39 master-level students who used the MERODE prototyper in a FIRP simulation environment. We examined the relationship between testing effort, measured by event invocations and model coverage, and the likelihood of students identifying seeded semantic mistakes and providing correct solutions, using logistic regression.Results:Logistic regression analyses revealed a statistically significant positive correlation between testing effort and the detection of specific behavioral constraint violations that resulted in deadlocks. However, the relationship was not consistent across all mistake types.Conclusion:FIRP simulation is particularly effective in helping novice modelers identify behavioral constraint violations that disrupt system workflows, i.e., those leading to deadlocks. Active model exploration through FIRP enhances error detection, highlighting its potential as a valuable tool for conceptual modeling education. However, the effectiveness varies with the type of semantic error, suggesting the need for tailored simulation strategies., This work was funded by the ENACTEST - European innovation alliance for testing education (ERASMUS+ Project number 101055874, 2022-2025).
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publication Artículo científico (JournalArticle)

A model-driven approach to adopt good practices for agile process configuration and certification

  • Giachetti-Herrera, Giovanni Andrés
  • Marín, Beatriz|||0000-0001-8025-0023
  • de la Vara, José Luis
[EN] Context: The agile development paradigm has been extensively adopted in the industry. This adoption is highly dependent on the knowledge and good practices applied by most experienced practitioners in organizations. Hence, it would be valuable to count on appropriate support to preserve and systematically use this expert knowledge in configuring agile development processes aligned with organizational standards. Objective: This paper presents a model-driven approach for representing and selecting good practices to configure agile practices in development processes aligned with organizational development practices and quality standards. Method: We define a conceptual approach for configuring agile development processes that fulfills enterprise good practices and external quality standards. This approach was implemented in a tool suite and applied to an industrial development scenario related to ISO 9001 certification. Results: The approach was implemented in a model-driven tool that provides automatic support for identifying good practices when configuring agile development processes. The tool also verifies consistency with development methods and quality standards, such as ISO 9001. Conclusions: The results obtained from the industrial application indicate that practitioners can reuse expert knowledge to configure agile development processes aligned with quality certifications. Moreover, the approach also facilitates the tailoring of agile practices into concrete development processes that take advantage of organizational good practices., The work leading to this paper received funding from the iRel4.0 (H2020-ECSEL grant agreement no 876659; Spain's MICINN ref. PCI2020-112240), VALU3S (H2020-ECSEL grant agreement no 876852; Spain's MICINN ref. PCI2020-112001); Treasure (JCCM ref. SBPLY/19/180501/000270; EC's European Regional Development Fund) projects; The Ramon y Cajal Program (Spain's MICINN ref. RYC-2017-22836; EC's European Social Fund); ENACTEST project (ERASMUS+ Project Number 101055874-ENACTEST-ERASMUS-EDU-2021-PI-ALL-INNO); The Valencian Innovation Agency and Innovation through the OGMIOS project (INNEST/2021/57); The Generalitat Valenciana through the CoMoDiD project (CIPROM/2021/023); The Spanish State Research Agency through the DELFOS (PDC2021-121243-I00) and SREC (PID2021-123824OB-I00) projects; MICIN/AEI/10.13039/501 100011033; and co-financed with ERDF and the European Union Next Generation EU/PRTR; SIAM (Chile's CORFO-INNOVA ref. 16COTE-60218), and from NISUM Technologies Chile.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publication Artículo científico (JournalArticle)

Teachers&apos, Perspective on Software Testing Education

  • Fasolino, Anna Rita
  • Mendes, Alexandra
  • Paiva, Ana C. R.
  • Cammaerts, Felix
  • Snoeck, Monique
  • Saadatmand, Mehrdad
  • Tramontana, Porfirio
  • Vos, Tanja Ernestina|||0000-0002-6003-9113
  • Marín, Beatriz|||0000-0001-8025-0023
[EN] Context. Software testing is a critical aspect of the software development lifecycle, yet it remains underrepresented in academic curricula. Despite advances in pedagogical practices and increased attention from the academic community, challenges persist in effectively teaching software testing. Understanding these challenges from the teachers' perspective is crucial to aligning education with industry needs. Objective. To analyze the characteristics, practices, tools, and challenges of software testing courses in higher education, from the perspective of educators, and to assess the integration of recent pedagogical approaches in software testing education. Method. A structured survey consisting of 52 questions was distributed to 143 software testing educators across Western European universities, resulting in 49 valid responses. The survey explored topics taught, course organization, teaching practices, tools and materials used, gamification approaches, and teacher satisfaction. Results. The survey revealed significant variability in course content, structure, and teaching methods. Most dedicated software testing courses are offered at the master's level and are elective, whereas testing is introduced earlier in less specialized (NST) courses. There is low adoption of formal guidelines (e.g., ACM, SWEBOK), limited integration of non-functional testing types, and a high diversity in textbooks and tools used. While modern practices like Test-Driven Development and automated assessment are increasingly adopted, gamification and active learning approaches remain underutilized. Teachers expressed a need for improved and more consistent teaching materials. Conclusion. The study highlights a mismatch between academic practices and industry expectations in software testing education. Greater integration of standardized curricula, broader adoption of modern teaching tools, and increased support for teachers through high-quality, adaptable teaching materials are needed to enhance the effectiveness of software testing education., This work has been funded by ENACTEST (European iNnovation AllianCe for TESTing education) , ERASMUS+ Project number 101055874, 2022-2025.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publication Artículo científico (JournalArticle)

Behavior Driven Development for 3D games

  • Pastor-Ricós, Fernando|||0000-0002-5790-193X
  • Marín, Beatriz|||0000-0001-8025-0023
  • Vos, Tanja Ernestina|||0000-0002-6003-9113
  • I.S.W.B. PRASETYA
  • Davidson, Joseph
  • Hovorka, Karel
[EN] Computer 3D games are complex software environments that require novel testing processes to ensure high-quality standards. The Intelligent Verification/Validation for Extended Reality Based Systems (iv4XR) framework addresses this need by enabling the implementation of autonomous agents to automate game testing scenarios. This framework facilitates the automation of regression test cases for complex 3D games like Space Engineers. Nevertheless, the technical expertise required to define test scripts using iv4XR can constrain seamless collaboration between developers and testers. This paper reports how integrating a Behavior-Driven Development (BDD) approach with the iv4XR framework allows the industrial company behind Space Engineers to automate regression testing. The success of this industrial collaboration has inspired the iv4XR team to integrate the BDD approach to improve the automation of play-testing for the experimental 3D game LabRecruits. Furthermore, the iv4XR framework has been extended with tactical programming to enable the automation of long-play test scenarios in Space Engineers. These results underscore the versatility of the iv4XR framework in supporting diverse testing approaches while showcasing how BDD empowers users to create, manage, and execute automated game tests using comprehensive and human-readable statements., This work has been partially funded by : H2020 EU iv4XR, Spain grant nr. 856716 and ENACTEST ERASMUS+, The Netherlands grant nr. 101055874.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publication Artículo científico (JournalArticle)

Towards understanding students&apos, sensemaking of test case design

  • Doorn, Niels
  • Vos, Tanja Ernestina|||0000-0002-6003-9113
  • Marín, Beatriz|||0000-0001-8025-0023
[EN] Context: Software testing is the most used technique for quality assurance in industry. However, in computer science education software testing is still treated as a second-class citizen and students are unable to test their software well enough. One reason for this is that teaching the subject of software testing is difficult as it is a complex intellectual activity for which students need to allocate multiple cognitive resources at the same time. A myriad of primary and secondary studies have tried to solve this problem in education, however still with very limited results.Objective: Before we can design interventions to improve our pedagogical approaches, we need to gain more in-depth understanding and recognition of sensemaking as it is happening when students design test cases.Method: An initial exploratory study identified four different sensemaking approaches used by students while creating test models. In this paper we present a follow-up study with 50 students from a large university in Spain. The used methodology was based on the previous study with the improvements that originated from its evaluation. We asked the participants to create a test model based on a description of a test problem using a specialized web-based tool for modeling test cases. We measured how well these models fit the test problem, the sensemaking process that students went through when creating the models, and the students' perception of the modeling task. The participants received no compensation for their efforts, and we scheduled the experiment during a regular class. Apart from the created models and their metadata, we also collected recordings of the students' computer screens made during the experiment and used a questionnaire to study their perspectives on the assignment. All the collected textual, graphical, and video data was analyzed using an iterative inductive analysis process to allow new information about the different sensemaking approaches to emerge.Results: We gained better insights into the sensemaking processes of students while modeling test cases for a problem. The results enabled us to refine our previous findings, and we identified new sensemaking approaches.Conclusions: Based on these results, we can further investigate ways to influence the sense-making process in education, the possible misconceptions that have a negative influence on it, and the desired mental model we want our students to have to design test cases., We like to thank Silvio Cacace for his input on the assignment, and for the use of the TestCompass tool, Erik Barendsen for his advice and feedback on this research and of course the participating students from the bachelor GTDM at the Universitat Politecnica de Valencia. The work leading to this paper has received funding from the European Union by the ENACTEST project (101055874) and the Erasmus + project QPeD under contract number 2020-1-NL01-KA203-064626.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publication Tesis doctoral (DoctoralThesis)

Improving the QA Testing Process of Desktop, Web, Mobile, and XR Applications through Scriptless Testing using the TESTAR Tool

  • Pastor-Ricós, Fernando|||0000-0002-5790-193X
[ES] El software es un pilar fundamental en sectores como la banca, los videojuegos y la salud. Las pruebas de software son el método más empleado para garantizar la calidad y fiabilidad de estos sistemas. Al someter los sistemas a pruebas rigurosas y recopilar evidencias, este proceso permite tomar decisiones informadas sobre la calidad del producto. Los testers humanos son fundamentales en las pruebas de software, aportando intuición, creatividad y comprensión profunda. No obstante, los esfuerzos de los testers no deben centrarse en la ejecución manual y repetitiva de interacciones con el software, ya que ello implica una carga de trabajo excesiva, frustración y fatiga, lo que genera falta de atención y limitación en el tiempo de realización de pruebas. Scripted testing es un enfoque para automatizar pruebas de software con scripts secuenciales y oráculos de prueba, reduciendo el esfuerzo manual y ayudando a mantener la calidad del software al detectar fallos de regresión. Sin embargo, la necesidad de mantener un número incontrolable de scripts, requeridos para sistemas grandes y complejos, hace que las pruebas basadas en scripts resulten insuficientes. Scriptless testing es un enfoque automatizado complementario que no depende de scripts, sino que explora el software de manera dinámica mediante acciones no secuenciales. Este enfoque introduce aleatoriedad, lo que permite descubrir estados y fallos no identificados por pruebas manuales o con scripts. TESTAR es una herramienta de automatización de pruebas sin scripts de código abierto que ha demostrado ser eficaz en la cobertura y detección de fallos al probar aplicaciones de escritorio y web industriales, a través de la Interfaz Gráfica de Usuario (GUI). La arquitectura modular de TESTAR permite seguir mejorando la herramienta para: (1) conectarse con otros sistemas software complejos, (2) incorporar algoritmos exploratorios para mejorar la efectividad y eficiencia de las pruebas, (3) integrar diverso de oráculos de prueba para cubrir una amplia variedad de características y detectar sus fallos, y (4) ser validada en proyectos industriales y de código abierto bien conocidos. En esta tesis, hemos colaborado con socios industriales y proyectos de código abierto para mejorar TESTAR, integrando marcos tecnológicos para conectar con aplicaciones de escritorio, web, móvil y de Realidad Extendida (XR); añadiendo algoritmos exploratorios para mejorar la efectividad y eficiencia de los mecanismos de selección de acciones; e integrando nuevos oráculos de prueba para la detección automatizada de fallos. Estas nuevas extensiones se han evaluado con proyectos industriales y de código abierto bien establecidos. Nuestros resultados empíricos demuestran una mejora significativa en las capacidades de TESTAR. Primero, TESTAR puede conectarse y probar aplicaciones de escritorio, web, móvil y juegos de XR. Segundo, la implementación de procesos de inferencia de modelos de estado, un enfoque de inferencia distribuida, reglas de selección de acciones basadas en gramáticas y un algoritmo interactivo para juegos, mejoran significativamente la efectividad y la eficiencia de la exploración sin scripts. Tercero, la integración de oráculos de prueba y detección de cambios delta permiten a TESTAR detectar diversos tipos de fallos en el software e identificar cambios en la GUI entre versiones delta de la misma aplicación. Cuarto, TESTAR ha demostrado ser una solución complementaria a enfoques de pruebas manuales y scripts en la industria. Finalmente, hemos realizado una colaboración implementando un enfoque de Desarrollo Guiado por Comportamiento (BDD) para automatizar pruebas de regresión de un juego XR industrial. Esta tesis concluye que usar una herramienta scriptless como TESTAR puede mejorar significativamente el proceso de aseguramiento de la calidad (QA) en aplicaciones de escritorio, web, móviles y XR, al aumentar la cobertura y detectar fallos inesperados., [CA] El programari és un pilar fonamental en sectors com la banca, els videojocs i la salut. Les proves de programari són el procés més utilitzat per assegurar la qualitat i la fiabilitat d'estos sistemes. Mitjançant proves rigoroses sobre els sistemes i la recopilació d'evidències, este procés permet prendre decisions informades sobre la qualitat d'un producte de programari. Els testers humans són fonamental en les proves de programari perquè aporten intuïció, creativitat i comprensió profunda. No obstant això, els seus esforços no haurien de destinar-se a l'execució d'interaccions manuals i repetitives amb el programari, ja que això genera càrregues de treball elevades, frustració i fatiga, cosa que comporta una falta d'atenció i de temps per realitzar les proves. Scripted testing és un enfocament per automatitzar l'execució de proves creant scripts seqüencials i oracles de prova, reduint l'esforç manual i ajudant a mantindre la qualitat del programari detectant fallades de regressió. No obstant això, la necessitat de mantindre un nombre incontrolable de scripts, per a sistemes grans i complexos, fan que la prova amb scripts siga un enfocament insuficient. Scriptless testing és un enfocament automatitzat complementari que no es basa en scripts, sinó que explora dinàmicament el programari amb accions no seqüencials. Este enfocament introduïx aleatorietat, la qual ajuda a descobrir estats i falles no identificats per les proves manuals o amb scripts. TESTAR és una eina d'automatització de proves sense scripts de codi obert que ha demostrat efectivitat en la cobertura i detecció de falles provant aplicacions industrials d'escriptori i web a través de la Interfície Gràfica d'Usuari (GUI). L'arquitectura modular de TESTAR permet a l'eina de millorar per a: (1) connectar-se amb altres sistemes de programari complexos, (2) incorporar algoritmes exploratoris per millorar l'efectivitat i l'eficiència de les proves, (3) integrar diversos oracles de prova per cobrir una àmplia varietat de característiques o descobrir els seus errors, i (4) ser validada en projectes industrials i de codi obert coneguts. En esta tesi, hem col·laborat amb socis industrials i projectes de codi obert per millorar TESTAR, integrant marcs tecnològics per connectar amb aplicacions d'escriptori, web, mòbils i de Realitat Estesa (XR); afegint algoritmes exploratoris per millorar l'efectivitat i l'eficiència dels mecanismes de selecció d'accions; i integrant nous oracles de prova per a la detecció automatitzada de fallades. Estes noves extensions s'han avaluat amb projectes industrials i de codi obert ben establits. Els nostres resultats empírics demostren una millora significativa en les capacitats de TESTAR. Primer, TESTAR pot connectar-se i provar aplicacions d'escriptori, web, mòbils i jocs de XR. En segon lloc, la implementació de processos d'inferència de models d'estat, un enfocament d'inferència distribuïda, regles de selecció d'accions basades en gramàtiques i un algoritme interactiu per a jocs milloren significativament l'efectivitat i l'eficiència de l'exploració sense scripts. En tercer lloc, la integració d'oracles de prova i la detecció de canvis delta permeten a TESTAR detectar diversos tipus de fallades en el programari i identificar canvis en la GUI entre versions delta de la mateixa aplicació. En quart lloc, TESTAR ha demostrat ser una solució complementària als enfocaments de proves manuals i amb scripts en la indústria. Finalment, hem col·laborat implementant un enfocament de Desenvolupament Guiat per Comportament (BDD) per automatitzar proves de regressió d'un joc XR industrial. Esta tesi conclou que utilitzar una eina scriptless com TESTAR pot millorar significativament el procés d'assegurament de la qualitat (QA) de les aplicacions d'escriptori, web, mòbils i XR augmentant la cobertura i detectant fallades inesperades., [EN] Software is a fundamental pillar in sectors such as banking, video games, and healthcare. Software testing is the most commonly used method to ensure the quality and reliability of these systems. By rigorously exercising software systems and collecting evidence, testing enables informed decisions about a software product's quality. Human testers play a vital role in software testing, bringing intuition, creativity, and a deep understanding of real-world contexts. However, testers' efforts should not be misdirected to execute manual and repetitive software interactions, as this leads to excessive workload, frustration, and fatigue, resulting in decreased attention and limited testing time. Scripted testing automates testing through sequential scripts and test oracles, reducing manual effort and helping maintain software quality by detecting unintended regression failures. However, the need to maintain these scripts and the unmanageable number of scripts required for large and complex systems make scripted testing an insufficient approach. Scriptless testing is a complementary automated approach that does not rely on scripts but dynamically explores software with non-sequential actions. This approach introduces randomness, which helps to uncover states and failures not identified by manual or scripted testing. TESTAR is an open-source scriptless testing tool that has demonstrated its coverage and fault detection effectiveness when testing industrial desktop and web applications through the Graphical User Interface (GUI). The modular architecture of TESTAR enables the tool for further improvement to: (1) connect with other complex software systems, (2) incorporate additional exploratory algorithms for test effectiveness and efficiency, (3) integrate a diverse set of test oracles to cover a wide variety of features or discover its faults, and (4) be validated with industrial and well-known open-source projects. In this thesis, we conducted several research collaborations with industrial partners and open-source projects to improve TESTAR. We extended its interaction capabilities by integrating frameworks to test desktop, web, mobile, and eXtended Reality (XR) applications, added exploratory algorithms to improve the test effectiveness and efficiency of action selection mechanisms, and integrated a new set of test oracles for automated software assurance and fault detection. These novel extensions were evaluated with industrial applications and well-established open-source projects. Our empirical results demonstrate a significant improvement in TESTAR capabilities. First, TESTAR can successfully connect to and test various desktop, web, mobile, and XR game applications. Second, a state model inference processes, a distributed inference approach, grammar-based action selection rules, and an interactive algorithm for navigating game environments help significantly improve the effectiveness and efficiency of scriptless exploration. Third, the integration of online test oracles and offline delta change detection oracles enabled TESTAR to detect diverse types of software failures and identify and highlight GUI changes between delta versions of the same software application. Fourth, TESTAR has proved to be complementary to manual and scripted testing approaches in industrial organizations. Finally, we additionally collaborated to implement a Behavior-Driven Development (BDD) approach for automating regression tests of an XR industrial game system. This thesis concludes that using a scriptless testing tool like TESTAR can significantly improve the Quality Assurance (QA) testing process of desktop, web, mobile, and XR applications by increasing coverage and detecting unexpected failures., This thesis has been developed at the Universitat Politècnica de València and has been funded by various European research projects: ITEA3 TESTOMAT project (TESTOMAT 16032), H2020 DEveloper COmpanion for Documented and annotatEd code Reference (DECODER 824231), H2020 Intelligent Verification/Validation for Extended Reality Based Systems (IV4XR 856716), ITEA3 Industrial-grade Verification and Validation of Evolving Systems (IVVES 18022), Erasmus+ European Innovation Alliance for Testing Education (ENACTEST 101055874), NWO OTP project Automated Unobtrusive Techniques for LINK- ing requirements and testing in agile software development (AUTOLINK 19521).
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publication Artículo científico (JournalArticle)

Identifying testing behaviour patterns in conceptual modelling education

  • Cammaerts, Felix
  • Tramontana, Porfirio
  • Flores, Nuno
  • Doorn, Niels
  • Fasolino, Anna Rita
  • Paiva, Ana C. R.
  • Snoeck, Monique
  • Vos, Tanja Ernestina|||0000-0002-6003-9113
  • Marín, Beatriz|||0000-0001-8025-0023
[EN] Context: Software testing is an often used method to ensure software quality, with developers spending a significant amount of time on it. However, software testing education is underrepresented in curricula, particularly with regard to Conceptual Modelling (CM), where model-level validation is important. As a consequence, graduate students are insufficiently prepared for industry-level testing. Objective: We want to understand students' behavioural patterns during model testing and their relationship to testing effectiveness. Methods: We use process mining to analyse 554 interaction logs from students using a conceptual model simulation tool. Results: We identify three behavioural profiles: the coverage chaser (using feedback iteratively), the problem solver (balancing feedback with requirements focus), and the disconnected tester (avoiding feedback-driven iteration). High-coverage students tended to integrate feedback into test loops; low-coverage students did not. Notably, high and average-coverage students achieved similar, generally high grades, a pattern consistent with a performance plateau where systematic testing is associated with sufficient educational performance without requiring maximum coverage. Conclusions: Our findings suggest that systematic use of coverage feedback is associated with more effective testing outcomes, suggesting that coverage-feedback use may be a promising focus for future instructional support in CM education., This work has been funded by ENACTEST-European innovation alliance for testing education-ERASMUS+ Project no. 101055874, 2022-2025.