Pasar al contenido principal

Resultados relacionados

Resultados totales (Incluyendo duplicados): 8
Encontrada(s) 1 página(s)

RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publicación Artículo científico (JournalArticle)

Scripted and scriptless GUI testing for web applications: An industrial case

  • Bons, Axel
  • Aho, Pekka
  • Marín, Beatriz|||0000-0001-8025-0023
  • Vos, Tanja Ernestina|||0000-0002-6003-9113
[EN] Context: Automation is required in the software development to reduce the high costs of producing software and to address the short release cycles of modern development processes. Lot of effort has been performed to automate testing, which is one of the most resource-consuming development phases. Automation of testing through the Graphical User Interface (GUI) has been researched to improve the system testing.Objective: We aim to evaluate the complementarity of automated GUI testing tools in a real industrial context, which refers to the capability of the tools to work usefully together.Methods: To address the objective, we conduct an exploratory case study in an IT development company from The Netherlands. We select two representative tools for automated GUI testing, one for scripted and another for scriptless testing. We measure the complementarity by measuring the effectiveness, the efficiency, and subjective satisfaction of the tools. Results: It can be observed that the scripted tool performs better in detecting process failures, and the scriptless tool performs better in detecting visible failures and also reaching higher coverage. Both tools perform in a similar way in terms of efficiency. Additionally, both tools were perceived to be useful in the survey performed for the subjective satisfaction.Conclusion: We conclude that scriptless and scripted testing approaches are complementary, and they can improve the effectiveness compared to manual testing processes performed in an industrial context by detecting different failures and reducing the effort and time to find these failures and to reproduce them., This research has been funded by iv4XR, H2020, grant number 856716 (https://iv4xr-project.eu) , and IVVES, ITEA, contract number 18022 (https://ivves.eu/) projects.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publicación Contribución de congreso (ConferenceOutput)

30 Years of Automated GUI Testing: A Bibliometric Analysis

  • Rodríguez-Valdés, Olivia
  • Aho, Pekka
  • Vos, Tanja Ernestina|||0000-0002-6003-9113
  • Marín, Beatriz|||0000-0001-8025-0023
[EN] Context: Over the last 30 years, GUIs have changed considerably, becoming everyday part of our lives through smart phones and other devices. More complex GUIs and multitude of platforms have increased the challenges when testing software through the GUI. Objective: To visualise how the field of automated GUI testing has evolved by studying the growth of the field; types of publications; influential events, papers and authors; collaboration among authors; and trends on GUI testing. Method: To conduct a bibliometric analysis of automated GUI testing by performing a systematic search of primary studies in Scopus from 1990 to 2020. Results: 744 publications were selected as primary studies. The majority of them were conference papers, the most cited paper was published on 2013, and the most published author has 53 papers. Conclusions: Automated GUI testing has continuously grown. Keywords show that testing applied to mobile interfaces will be the trend in next years, along with the integration of Artificial Intelligence and automated exploration techniques., We thank Fernando Pastor for his valuable contribution. This research has been funded by DECODER (decoder-project.eu), iv4XR (iv4xrproject.eu), and IVVES (ivves.weebly.com) projects.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publicación Artículo científico (JournalArticle)

Distributed state model inference for scriptless GUI testing

  • Pastor-Ricós, Fernando|||0000-0002-5790-193X
  • Marín, Beatriz|||0000-0001-8025-0023
  • Vos, Tanja Ernestina|||0000-0002-6003-9113
  • Arend Slomp
  • Open Universiteit
[EN] State model inference of software applications through the Graphical User Interface (GUI) is a technique that identifies GUI states and transitions, and maps them into a model. Scriptless GUI testing tools can benefit substantially from the availability of these state models, for example, to improve the exploration, or have sophisticated test oracles. However, inferring models for large systems requires a long execution time. Our goal is to improve the speed of the state model inference process. To achieve this goal, this paper presents a distributed state model inference approach with an open source scriptless GUI testing tool. Moreover, in order to be able to infer a suitable model, we design a set of strategies to deal with abstraction challenges and to distinguish GUI states and transitions in the model. To validate it, we conduct an experiment with two open source web applications that have been tested with the distributed architecture using one to six Docker containers sharing the same state model. With the obtained results, we can conclude that it is feasible to infer a model with a distributed approach and that using the distributed approach reduces the time required for inferring a state model., Thanks to Olivia Rodriguez-Valdes. This research has been funded by the following projects: H2020 EU project iv4XR (www. iv4xr-project.eu) grant nr. 856716, and ITEA3 project IVVES (www. ivves.eu) grant nr. 18022.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publicación Artículo científico (JournalArticle)

Behavior Driven Development for 3D games

  • Pastor-Ricós, Fernando|||0000-0002-5790-193X
  • Marín, Beatriz|||0000-0001-8025-0023
  • Vos, Tanja Ernestina|||0000-0002-6003-9113
  • I.S.W.B. PRASETYA
  • Davidson, Joseph
  • Hovorka, Karel
[EN] Computer 3D games are complex software environments that require novel testing processes to ensure high-quality standards. The Intelligent Verification/Validation for Extended Reality Based Systems (iv4XR) framework addresses this need by enabling the implementation of autonomous agents to automate game testing scenarios. This framework facilitates the automation of regression test cases for complex 3D games like Space Engineers. Nevertheless, the technical expertise required to define test scripts using iv4XR can constrain seamless collaboration between developers and testers. This paper reports how integrating a Behavior-Driven Development (BDD) approach with the iv4XR framework allows the industrial company behind Space Engineers to automate regression testing. The success of this industrial collaboration has inspired the iv4XR team to integrate the BDD approach to improve the automation of play-testing for the experimental 3D game LabRecruits. Furthermore, the iv4XR framework has been extended with tactical programming to enable the automation of long-play test scenarios in Space Engineers. These results underscore the versatility of the iv4XR framework in supporting diverse testing approaches while showcasing how BDD empowers users to create, manage, and execute automated game tests using comprehensive and human-readable statements., This work has been partially funded by : H2020 EU iv4XR, Spain grant nr. 856716 and ENACTEST ERASMUS+, The Netherlands grant nr. 101055874.
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publicación Contribución de congreso (ConferenceOutput)

Deploying TESTAR to Enable Remote Testing in an Industrial CI Pipeline: A Case-Based Evaluation

  • Pastor-Ricós, Fernando|||0000-0002-5790-193X
  • Vos, Tanja Ernestina|||0000-0002-6003-9113
  • Aho, Pekka
  • Torres Boigues, Ismael
  • Calás Blasco, Ernesto
  • Martínez Martínez, Hector
[EN] Companies are facing constant pressure towards shorter release cycles while still maintaining a high level of quality. Agile development, continuous integration and testing are commonly used quality assurance techniques applied in industry. Increasing the level of test automation is a key ingredient to address the short release cycles. Testing at the graphical user interface (GUI) level is challenging to automate, and therefore many companies still do this manually. To help find solutions for better GUI test automation, academics are researching scriptless GUI testing to complement the script-based approach. In order to better match industrial problems with academic results, more academia-industry collaborations for case-based evaluations are needed. This paper describes such an initiative to improve, transfer and integrate an academic scriptless GUI testing tool TESTAR into the CI pipeline of a Spanish company Prodevelop. The paper describes the steps taken, the outcome, the challenges, and some lessons learned for successful industry-academia collaboration., This work has been funded through the ITEA3 TESTOMAT project (www.testomatproject.eu), the EU H2020 DECODER project (www.decoderproject.eu), the EU H2020 iv4XR project (iv4xr-project.eu) and the ITEA3 IVVES project (ivves.weebly.com).
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publicación Tesis doctoral (DoctoralThesis)

Improving the QA Testing Process of Desktop, Web, Mobile, and XR Applications through Scriptless Testing using the TESTAR Tool

  • Pastor-Ricós, Fernando|||0000-0002-5790-193X
[ES] El software es un pilar fundamental en sectores como la banca, los videojuegos y la salud. Las pruebas de software son el método más empleado para garantizar la calidad y fiabilidad de estos sistemas. Al someter los sistemas a pruebas rigurosas y recopilar evidencias, este proceso permite tomar decisiones informadas sobre la calidad del producto. Los testers humanos son fundamentales en las pruebas de software, aportando intuición, creatividad y comprensión profunda. No obstante, los esfuerzos de los testers no deben centrarse en la ejecución manual y repetitiva de interacciones con el software, ya que ello implica una carga de trabajo excesiva, frustración y fatiga, lo que genera falta de atención y limitación en el tiempo de realización de pruebas. Scripted testing es un enfoque para automatizar pruebas de software con scripts secuenciales y oráculos de prueba, reduciendo el esfuerzo manual y ayudando a mantener la calidad del software al detectar fallos de regresión. Sin embargo, la necesidad de mantener un número incontrolable de scripts, requeridos para sistemas grandes y complejos, hace que las pruebas basadas en scripts resulten insuficientes. Scriptless testing es un enfoque automatizado complementario que no depende de scripts, sino que explora el software de manera dinámica mediante acciones no secuenciales. Este enfoque introduce aleatoriedad, lo que permite descubrir estados y fallos no identificados por pruebas manuales o con scripts. TESTAR es una herramienta de automatización de pruebas sin scripts de código abierto que ha demostrado ser eficaz en la cobertura y detección de fallos al probar aplicaciones de escritorio y web industriales, a través de la Interfaz Gráfica de Usuario (GUI). La arquitectura modular de TESTAR permite seguir mejorando la herramienta para: (1) conectarse con otros sistemas software complejos, (2) incorporar algoritmos exploratorios para mejorar la efectividad y eficiencia de las pruebas, (3) integrar diverso de oráculos de prueba para cubrir una amplia variedad de características y detectar sus fallos, y (4) ser validada en proyectos industriales y de código abierto bien conocidos. En esta tesis, hemos colaborado con socios industriales y proyectos de código abierto para mejorar TESTAR, integrando marcos tecnológicos para conectar con aplicaciones de escritorio, web, móvil y de Realidad Extendida (XR); añadiendo algoritmos exploratorios para mejorar la efectividad y eficiencia de los mecanismos de selección de acciones; e integrando nuevos oráculos de prueba para la detección automatizada de fallos. Estas nuevas extensiones se han evaluado con proyectos industriales y de código abierto bien establecidos. Nuestros resultados empíricos demuestran una mejora significativa en las capacidades de TESTAR. Primero, TESTAR puede conectarse y probar aplicaciones de escritorio, web, móvil y juegos de XR. Segundo, la implementación de procesos de inferencia de modelos de estado, un enfoque de inferencia distribuida, reglas de selección de acciones basadas en gramáticas y un algoritmo interactivo para juegos, mejoran significativamente la efectividad y la eficiencia de la exploración sin scripts. Tercero, la integración de oráculos de prueba y detección de cambios delta permiten a TESTAR detectar diversos tipos de fallos en el software e identificar cambios en la GUI entre versiones delta de la misma aplicación. Cuarto, TESTAR ha demostrado ser una solución complementaria a enfoques de pruebas manuales y scripts en la industria. Finalmente, hemos realizado una colaboración implementando un enfoque de Desarrollo Guiado por Comportamiento (BDD) para automatizar pruebas de regresión de un juego XR industrial. Esta tesis concluye que usar una herramienta scriptless como TESTAR puede mejorar significativamente el proceso de aseguramiento de la calidad (QA) en aplicaciones de escritorio, web, móviles y XR, al aumentar la cobertura y detectar fallos inesperados., [CA] El programari és un pilar fonamental en sectors com la banca, els videojocs i la salut. Les proves de programari són el procés més utilitzat per assegurar la qualitat i la fiabilitat d'estos sistemes. Mitjançant proves rigoroses sobre els sistemes i la recopilació d'evidències, este procés permet prendre decisions informades sobre la qualitat d'un producte de programari. Els testers humans són fonamental en les proves de programari perquè aporten intuïció, creativitat i comprensió profunda. No obstant això, els seus esforços no haurien de destinar-se a l'execució d'interaccions manuals i repetitives amb el programari, ja que això genera càrregues de treball elevades, frustració i fatiga, cosa que comporta una falta d'atenció i de temps per realitzar les proves. Scripted testing és un enfocament per automatitzar l'execució de proves creant scripts seqüencials i oracles de prova, reduint l'esforç manual i ajudant a mantindre la qualitat del programari detectant fallades de regressió. No obstant això, la necessitat de mantindre un nombre incontrolable de scripts, per a sistemes grans i complexos, fan que la prova amb scripts siga un enfocament insuficient. Scriptless testing és un enfocament automatitzat complementari que no es basa en scripts, sinó que explora dinàmicament el programari amb accions no seqüencials. Este enfocament introduïx aleatorietat, la qual ajuda a descobrir estats i falles no identificats per les proves manuals o amb scripts. TESTAR és una eina d'automatització de proves sense scripts de codi obert que ha demostrat efectivitat en la cobertura i detecció de falles provant aplicacions industrials d'escriptori i web a través de la Interfície Gràfica d'Usuari (GUI). L'arquitectura modular de TESTAR permet a l'eina de millorar per a: (1) connectar-se amb altres sistemes de programari complexos, (2) incorporar algoritmes exploratoris per millorar l'efectivitat i l'eficiència de les proves, (3) integrar diversos oracles de prova per cobrir una àmplia varietat de característiques o descobrir els seus errors, i (4) ser validada en projectes industrials i de codi obert coneguts. En esta tesi, hem col·laborat amb socis industrials i projectes de codi obert per millorar TESTAR, integrant marcs tecnològics per connectar amb aplicacions d'escriptori, web, mòbils i de Realitat Estesa (XR); afegint algoritmes exploratoris per millorar l'efectivitat i l'eficiència dels mecanismes de selecció d'accions; i integrant nous oracles de prova per a la detecció automatitzada de fallades. Estes noves extensions s'han avaluat amb projectes industrials i de codi obert ben establits. Els nostres resultats empírics demostren una millora significativa en les capacitats de TESTAR. Primer, TESTAR pot connectar-se i provar aplicacions d'escriptori, web, mòbils i jocs de XR. En segon lloc, la implementació de processos d'inferència de models d'estat, un enfocament d'inferència distribuïda, regles de selecció d'accions basades en gramàtiques i un algoritme interactiu per a jocs milloren significativament l'efectivitat i l'eficiència de l'exploració sense scripts. En tercer lloc, la integració d'oracles de prova i la detecció de canvis delta permeten a TESTAR detectar diversos tipus de fallades en el programari i identificar canvis en la GUI entre versions delta de la mateixa aplicació. En quart lloc, TESTAR ha demostrat ser una solució complementària als enfocaments de proves manuals i amb scripts en la indústria. Finalment, hem col·laborat implementant un enfocament de Desenvolupament Guiat per Comportament (BDD) per automatitzar proves de regressió d'un joc XR industrial. Esta tesi conclou que utilitzar una eina scriptless com TESTAR pot millorar significativament el procés d'assegurament de la qualitat (QA) de les aplicacions d'escriptori, web, mòbils i XR augmentant la cobertura i detectant fallades inesperades., [EN] Software is a fundamental pillar in sectors such as banking, video games, and healthcare. Software testing is the most commonly used method to ensure the quality and reliability of these systems. By rigorously exercising software systems and collecting evidence, testing enables informed decisions about a software product's quality. Human testers play a vital role in software testing, bringing intuition, creativity, and a deep understanding of real-world contexts. However, testers' efforts should not be misdirected to execute manual and repetitive software interactions, as this leads to excessive workload, frustration, and fatigue, resulting in decreased attention and limited testing time. Scripted testing automates testing through sequential scripts and test oracles, reducing manual effort and helping maintain software quality by detecting unintended regression failures. However, the need to maintain these scripts and the unmanageable number of scripts required for large and complex systems make scripted testing an insufficient approach. Scriptless testing is a complementary automated approach that does not rely on scripts but dynamically explores software with non-sequential actions. This approach introduces randomness, which helps to uncover states and failures not identified by manual or scripted testing. TESTAR is an open-source scriptless testing tool that has demonstrated its coverage and fault detection effectiveness when testing industrial desktop and web applications through the Graphical User Interface (GUI). The modular architecture of TESTAR enables the tool for further improvement to: (1) connect with other complex software systems, (2) incorporate additional exploratory algorithms for test effectiveness and efficiency, (3) integrate a diverse set of test oracles to cover a wide variety of features or discover its faults, and (4) be validated with industrial and well-known open-source projects. In this thesis, we conducted several research collaborations with industrial partners and open-source projects to improve TESTAR. We extended its interaction capabilities by integrating frameworks to test desktop, web, mobile, and eXtended Reality (XR) applications, added exploratory algorithms to improve the test effectiveness and efficiency of action selection mechanisms, and integrated a new set of test oracles for automated software assurance and fault detection. These novel extensions were evaluated with industrial applications and well-established open-source projects. Our empirical results demonstrate a significant improvement in TESTAR capabilities. First, TESTAR can successfully connect to and test various desktop, web, mobile, and XR game applications. Second, a state model inference processes, a distributed inference approach, grammar-based action selection rules, and an interactive algorithm for navigating game environments help significantly improve the effectiveness and efficiency of scriptless exploration. Third, the integration of online test oracles and offline delta change detection oracles enabled TESTAR to detect diverse types of software failures and identify and highlight GUI changes between delta versions of the same software application. Fourth, TESTAR has proved to be complementary to manual and scripted testing approaches in industrial organizations. Finally, we additionally collaborated to implement a Behavior-Driven Development (BDD) approach for automating regression tests of an XR industrial game system. This thesis concludes that using a scriptless testing tool like TESTAR can significantly improve the Quality Assurance (QA) testing process of desktop, web, mobile, and XR applications by increasing coverage and detecting unexpected failures., This thesis has been developed at the Universitat Politècnica de València and has been funded by various European research projects: ITEA3 TESTOMAT project (TESTOMAT 16032), H2020 DEveloper COmpanion for Documented and annotatEd code Reference (DECODER 824231), H2020 Intelligent Verification/Validation for Extended Reality Based Systems (IV4XR 856716), ITEA3 Industrial-grade Verification and Validation of Evolving Systems (IVVES 18022), Erasmus+ European Innovation Alliance for Testing Education (ENACTEST 101055874), NWO OTP project Automated Unobtrusive Techniques for LINK- ing requirements and testing in agile software development (AUTOLINK 19521).
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publicación Contribución de congreso (ConferenceOutput)

Evaluating TESTAR&apos, s effectiveness through code coverage

  • van der Brugge, Aaron
  • Aho, Pekka
  • Pastor-Ricós, Fernando|||0000-0002-5790-193X
  • Marín, Beatriz|||0000-0001-8025-0023
  • Vos, Tanja Ernestina|||0000-0002-6003-9113
[EN] Testing is of paramount importance in assuring the quality of software products. Nevertheless, it is not easy to judge which techniques or tools are the most effective. A commonly used surrogate metric to evaluate the effectiveness of testing tools is code coverage, which has been widely used for unit and integration testing. However, for GUI testing approaches, this metric has not been sufficiently investigated. To fill this gap, we run experiments with the TESTAR tool, a scriptless testing tool that automatically generates test cases at the Graphical User Interface (GUI) level. In the experiment, we analyze and compare the obtained code coverage when using four different action selection mechanisms (ASMs) in TESTAR that are used to test three SUTs, This research has been funded by the following projects: H2020 EU project DECODER (www.decoder-project.eu), H2020 EU project iv4XR (www.iv4xrproject.eu) and ITEA project IVVES (www.ivves.eu).
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publicación Artículo científico (JournalArticle)

testar - scriptless testing through graphical user interface

  • Vos, Tanja Ernestina|||0000-0002-6003-9113
  • Pastor-Ricós, Fernando|||0000-0002-5790-193X
  • Aho, Pekka
  • Rodríguez-Valdés, Olivia
  • Mulders, Ad
[EN] Covering all the possible paths of the graphical user interface (GUI) with test scripts would take too much effort and result in serious maintenance issues. We propose complementing scripted testing with scriptless test automation using the open-source TESTAR tool. This paper gives a comprehensive overview of TESTAR and its latest extensions together with the ongoing and future research. With this paper, we hope we can help and encourage other researchers to use TESTAR for their GUI testing-related research and pave the way for an international research agenda in GUI testing built upon stable and open-source infrastructure. © 2021 The Authors. Software Testing, Verification & Reliability published by John Wiley & Sons Ltd., This work has been funded by the following projects: FITTEST (FP7 Information and Communication Technologies, 257574), SHIP (EACEA/A2/UHB/CL 554187), PERTEST (TIN2013-46928-C3-1-R), TESTOMAT (ITEA3, 16032), DECODER (H2020, 824231), iv4XR (H2020, 856716), TAILOR (H2020, 952215) and IVVES (ITEA3, 18022)