In the last decade, modern data analytics technologies have enabled the creation of software analytics tools offering real-time visualization of various aspects related to software development and usage. These tools seem to be particularly attractive for companies doing agile software development. However, the information provided by the available tools is neither aggregated nor connected to higher quality goals. At the same time, assessing and improving software quality has also been a key target for the software engineering community, yielding several proposals for standards and software quality models. Integrating such quality models into software analytics tools could close the gap by providing the connection to higher quality goals. This study aims at understanding whether the integration of quality models into software analytics tools provides understandable, reliable, useful, and relevant information at the right level of detail about the quality of a process or product, and whether practitioners intend to use it. Over the course of more than one year, the four companies involved in this case study deployed such a tool to assess and improve software quality in several projects. We used standardized measurement instruments to elicit the perception of 22 practitioners regarding their use of the tool. We complemented the findings with debriefing sessions held at the companies. In addition, we discussed challenges and lessons learned with four practitioners leading the use of the tool. Quantitative and qualitative analyses provided positive results; i.e., the practitioners’ perception with regard to the tool’s understandability, reliability, usefulness, and relevance was positive. Individual statements support the statistical findings and constructive feedback can be used for future improvements. We conclude that potential for future adoption of quality models within software analytics tools definitely exists and encourage other practitioners to use the presented seven challenges and seven lessons learned and adopt them in their companies., Peer Reviewed, Postprint (published version)

Agile software development (ASD) aims to deliver high-quality software. However, software quality often receives less attention in ASD due to focus on functionality. Elicit requirements for, design, validate, implement, and evaluate an evidence-based quality-aware ASD process that organizations can use for better integration of quality management. Using design science research, we collaborated for over 1.5 years with four organizations applying ASD. Involving a total of 35 practitioners, we designed new evidence-based quality-aware ASD processes, which were validated through simulated scenarios, implemented, and evaluated by the practitioners. A general ASD process model, derived from the companyspecific models, was created; it shows activities, artifacts, roles, and development phases where evidence can be used to improve software quality. The new process model contributed to higher quality awareness, improved bottleneck identification, release readiness, and quality issues prioritization in the studied companies. We also learned that companies must be mindful of the rules that determine the use of evidence. Integrating the use of quality related evidence into the ASD process can make it quality-aware, support software quality management, and enable timely reaction through fast feedback loops. The evidence-based quality-aware ASD process is applicable in any Scrum-like ASD process. Applicability in other forms of ASD, however, requires further research., This work was supported by the Q-Rapids project though the European Union’s Horizon 2020 Research and Innovation Program under
Grant 732253., Peer Reviewed, Postprint (published version)

Context: In despite of agile and rapid software development (ARSD) being researched and applied extensively, managing quality requirements (QRs) are still challenging. As ARSD processes produce a large amount of data, measurement has become a strategy to facilitate QR management. Objective: This study aims to survey the literature related to QR management through metrics in ARSD, focusing on: bibliometrics, QR metrics, and quality-related indicators used in quality management. Method: The study design includes the definition of research questions, selection criteria, and snowballing as search strategy. Results: We selected 61 primary studies (2001-2019). Despite a large body of knowledge and standards, there is no consensus regarding QR measurement. Terminology is varying as are the measuring models. However, seemingly different measurement models do contain similarities. Conclusion: The industrial relevance of the primary studies shows that practitioners have a need to improve quality measurement. Our collection of measures and data sources can serve as a starting point for practitioners to include quality measurement into their decision-making processes. Researchers could benefit from the identified similarities to start building a common framework for quality measurement. In addition, this could help researchers identify what quality aspects need more focus, e.g., security and usability with few metrics reported., This work has been funded by the European Union’s Horizon 2020 research and innovation program through the Q-Rapids project (grant no. 732253). This research was also partially supported by the Spanish Ministerio de Economía, Industria y Competitividad through the DOGO4ML project (grant PID2020-117191RB-I00). Silverio Martínez-Fernández worked in Fraunhofer IESE before January 2020., Peer Reviewed, Postprint (published version)

Context: Agile software development has become commonplace in software development companies due to the numerous benefits it provides. However, conducting Agile projects is demanding in Small and Medium Enterprises (SMEs), because projects start and end quickly, but still have to fulfil customers' quality requirements. Objective: This paper aims at reporting a practical experience on the use of metrics related to the software development process as a means supporting SMEs in the development of software following an Agile methodology. Method: We followed Action-Research principles in a Polish small-size software development company. We developed and executed a study protocol suited to the needs of the company, using a pilot case. Results: A catalogue of Agile development process metrics practically validated in the context of a small-size software development company, adopted by the company in their Agile projects. Conclusions: Practitioners may adopt these metrics in their Agile projects, especially if working in an SME, and customise them to their own needs and tools. Academics may use the findings as a baseline for new research work, including new empirical studies., The authors would like to thank all the members of the QRapids H2020 project consortium., Peer Reviewed, Postprint (published version)

Context: In despite of agile and rapid software development (ARSD) being researched and applied extensively, managing quality requirements (QRs) are still challenging. As ARSD processes produce a large amount of data, measurement has become a strategy to facilitate QR management. Objective: This study aims to survey the literature related to QR management through metrics in ARSD, focusing on: bibliometrics, QR metrics, and quality-related indicators used in quality management. Method: The study design includes the definition of research questions, selection criteria, and snowballing as search strategy. Results: We selected 61 primary studies (2001-2019). Despite a large body of knowledge and standards, there is no consensus regarding QR measurement. Terminology is varying as are the measuring models. However, seemingly different measurement models do contain similarities. Conclusion: The industrial relevance of the primary studies shows that practitioners have a need to improve quality measurement. Our collection of measures and data sources can serve as a starting point for practitioners to include quality measurement into their decision-making processes. Researchers could benefit from the identified similarities to start building a common framework for quality measurement. In addition, this could help researchers identify what quality aspects need more focus, e.g., security and usability with few metrics reported., This work has been funded by the European Union’s Horizon 2020 research and innovation program through the Q-Rapids project (grant no. 732253). This research was also partially supported by the Spanish Ministerio de Economía, Industria y Competitividad through the DOGO4ML project (grant PID2020-117191RB-I00). Silverio Martínez-Fernández worked in Fraunhofer IESE before January 2020., Peer Reviewed

Context: Exploiting software development related data from software-development intensive organizations to support tactical and strategic decision making is a challenge. Combining data-driven approaches with expert knowledge has been highlighted as a sensible approach for leading software-development intensive organizations to rightful decision-making improvements. However, most of the existing proposals lack of important aspects that hinders their industrial uptake such as: customization guidelines to fit the proposals to other contexts and/or automatic or semi-automatic data collection support for putting them forward in a real organization. As a result, existing proposals are rarely used in the industrial context.

Objective: Support software-development intensive organizations with guidance and tools for exploiting software development related data and expert knowledge to improve their decision making.

Method: We have developed a novel method called SESSI (Specification and Estimation of Software Strategic Indicators) that was articulated from industrial experiences with Nokia, Bittium, Softeam and iTTi in the context of Q-Rapids European project following a design science approach. As part of the industrial summative evaluation, we performed the first case study focused on the application of the method.

Results: We detail the phases and steps of the SESSI method and illustrate its application in the development of ModelioNG, a software product of Modeliosoft development firm.

Conclusion: The application of the SESSI method in the context of ModelioNG case study has provided us with useful feedback to improve the method and has evidenced that applying the method was feasible in this context., This work was supported by the Spanish Ministry of Economy and Competitiveness (project GENESIS, grant agreement TIN2016-79269- R), and by the European Union’s Horizon 2020 research and innovation program (Q-Rapids project, grant agreement N◦732253)., Peer Reviewed

Context: Agile software development has become commonplace in software development companies due to the numerous benefits it provides. However, conducting Agile projects is demanding in Small and Medium Enterprises (SMEs), because projects start and end quickly, but still have to fulfil customers' quality requirements. Objective: This paper aims at reporting a practical experience on the use of metrics related to the software development process as a means supporting SMEs in the development of software following an Agile methodology. Method: We followed Action-Research principles in a Polish small-size software development company. We developed and executed a study protocol suited to the needs of the company, using a pilot case. Results: A catalogue of Agile development process metrics practically validated in the context of a small-size software development company, adopted by the company in their Agile projects. Conclusions: Practitioners may adopt these metrics in their Agile projects, especially if working in an SME, and customise them to their own needs and tools. Academics may use the findings as a baseline for new research work, including new empirical studies., The authors would like to thank all the members of the QRapids H2020 project consortium., Peer Reviewed

Context:
Quality requirements (QRs) describe the desired quality of software, and they play an important role in the success of software projects. In agile software development (ASD), QRs are often ill-defined and not well addressed due to the focus on quickly delivering functionality. Rapid software development (RSD) approaches (e.g., continuous delivery and continuous deployment), which shorten delivery times, are more prone to neglect QRs. Despite the significance of QRs in both ASD and RSD, there is limited synthesized knowledge on their management in those approaches.

Objective:
This study aims to synthesize state-of-the-art knowledge about QR management in ASD and RSD, focusing on three aspects: bibliometric, strategies, and challenges.

Research method:
Using a systematic mapping study with a snowballing search strategy, we identified and structured the literature on QR management in ASD and RSD.

Results:
We found 156 primary studies: 106 are empirical studies, 16 are experience reports, and 34 are theoretical studies. Security and performance were the most commonly reported QR types. We identified various QR management strategies: 74 practices, 43 methods, 13 models, 12 frameworks, 11 advices, 10 tools, and 7 guidelines. Additionally, we identified 18 categories and 4 non-recurring challenges of managing QRs. The limited ability of ASD to handle QRs, time constraints due to short iteration cycles, limitations regarding the testing of QRs and neglect of QRs were the top categories of challenges.

Conclusion:
Management of QRs is significant in ASD and is becoming important in RSD. This study identified research gaps, such as the need for more tools and guidelines, lightweight QR management strategies that fit short iteration cycles, investigations of the link between QRs challenges and technical debt, and extension of empirical validation of existing strategies to a wider context. It also synthesizes QR management strategies and challenges, which may be useful for practitioners., Peer Reviewed

Agile software development (ASD) aims to deliver high-quality software. However, software quality often receives less attention in ASD due to focus on functionality. Elicit requirements for, design, validate, implement, and evaluate an evidence-based quality-aware ASD process that organizations can use for better integration of quality management. Using design science research, we collaborated for over 1.5 years with four organizations applying ASD. Involving a total of 35 practitioners, we designed new evidence-based quality-aware ASD processes, which were validated through simulated scenarios, implemented, and evaluated by the practitioners. A general ASD process model, derived from the companyspecific models, was created; it shows activities, artifacts, roles, and development phases where evidence can be used to improve software quality. The new process model contributed to higher quality awareness, improved bottleneck identification, release readiness, and quality issues prioritization in the studied companies. We also learned that companies must be mindful of the rules that determine the use of evidence. Integrating the use of quality related evidence into the ASD process can make it quality-aware, support software quality management, and enable timely reaction through fast feedback loops. The evidence-based quality-aware ASD process is applicable in any Scrum-like ASD process. Applicability in other forms of ASD, however, requires further research., This work was supported by the Q-Rapids project though the European Union’s Horizon 2020 Research and Innovation Program under
Grant 732253., Peer Reviewed

In the last decade, modern data analytics technologies have enabled the creation of software analytics tools offering real-time visualization of various aspects related to software development and usage. These tools seem to be particularly attractive for companies doing agile software development. However, the information provided by the available tools is neither aggregated nor connected to higher quality goals. At the same time, assessing and improving software quality has also been a key target for the software engineering community, yielding several proposals for standards and software quality models. Integrating such quality models into software analytics tools could close the gap by providing the connection to higher quality goals. This study aims at understanding whether the integration of quality models into software analytics tools provides understandable, reliable, useful, and relevant information at the right level of detail about the quality of a process or product, and whether practitioners intend to use it. Over the course of more than one year, the four companies involved in this case study deployed such a tool to assess and improve software quality in several projects. We used standardized measurement instruments to elicit the perception of 22 practitioners regarding their use of the tool. We complemented the findings with debriefing sessions held at the companies. In addition, we discussed challenges and lessons learned with four practitioners leading the use of the tool. Quantitative and qualitative analyses provided positive results; i.e., the practitioners’ perception with regard to the tool’s understandability, reliability, usefulness, and relevance was positive. Individual statements support the statistical findings and constructive feedback can be used for future improvements. We conclude that potential for future adoption of quality models within software analytics tools definitely exists and encourage other practitioners to use the presented seven challenges and seven lessons learned and adopt them in their companies., Peer Reviewed

Software and data analytics solutions support improving development processes and the quality of the software produced in Agile Software Development (ASD). However, decision makers in software teams (e.g., product owner, project manager) are demanding powerful tools providing evidence data that support their strategic decision-making processes. In this paper, we present and provide access to QaSD, a Quality-aware Strategic Dashboard supporting decision makers in ASD. The dashboard allows decision makers to define high-level strategic indicators (e.g., customer satisfaction, process performance) related to software quality and to measure, explore, simulate and forecast the values of those indicators in order to explain and justify their decisions. Moreover, we also provide the results of a conducted evaluation of the dashboard quality in a real environment that evaluated the QaSD as usable, easy to use, with good navigation, and reliable, This work was supported by Q-Rapids (Quality-Aware Rapid Software Development) and VISDOM (Visual diagnosis for DevOps software development) projects. Q-Rapids was funded by the European Union’s Horizon 2020 research and innovation programme under grant agreement n◦ 732253. VISDOM is funded by the Research, Development and Innovation cluster programme ITEA 3 (project no 17038)., Peer Reviewed

Quality requirements (QRs) are a key artifact needed to ensure the quality and success of a software system. Despite their importance, QRs rarely get the same degree of attention as their functional counterpart in agile software development (ASD) projects. Moreover, crucial information that can be obtained from software development repositories (e.g., JIRA, GitHub) is not fully exploited, or is even neglected, in QR elicitation activities. In this work, we present a data-driven tooled approach for the semi-automatic generation and documentation of QRs in the context of ASD. The approach is based on the declaration of thresholds over quality-related issues, whose violation triggers user-defined alerts. These alerts are used to browse a catalog of QR patterns that are presented to the ASD team by means of a dashboard that implements several analysis techniques. Once selected, the patterns generate the QRs, which are documented and stored in the product backlog. The full approach is implemented via a configurable platform. Over the course of 1 year, four companies differing in size and profile followed this approach and deployed the platform in their premises to semi-automatically generate QRs in several projects. We used standardized measurement instruments to elicit the perception of 22 practitioners regarding their use of the tool. The quantitative and qualitative analyses yielded positive results; i.e., the practitioners’ perception with regard to the tool’s understandability, reliability, usefulness, and relevance was positive. We conclude that the results show potential for future adoption of data-driven elicitation of QRs in agile companies and encourage other practitioners to use the presented tool and adopt it in their companies., This work is a result of the Q-Rapids project, which has received funding from the European Union’s Horizon 2020 Research and Innovation Programme under grant agreement no. 732253., Peer Reviewed

Agile methods can deliver software that fulfills customer needs rapidly and continuously. Quality requirements (QRs) are important in this regard; however, detailed studies on how companies applying agile methods to manage QRs are limited, as are studies on the rationale for choosing specific QR management practices and related challenges. The aim of this study was to address why practitioners manage QRs as they do and what challenges they face. We also analyzed how existing practices mitigate some of the found challenges. Lastly, we connect the contextual elements of the companies with their practices and challenges. We conducted 36 interviews with practitioners from four companies of varying sizes. Since each company operates in different domains, comparing QR management strategies and related challenges in different contexts was possible. We found that the companies apply proactive, reactive, and interactive strategies to manage QRs. Additionally, our study revealed 40 challenges in six categories that companies applying agile methods may face in QR management. We also identified nine contextual elements that affect QR management practice choices and which, importantly, can explain many related challenges. Based on these findings, we constructed a theoretical model about the connection between context, QR management practices, and challenges. Practitioners in similar contexts can learn from the practices identified in this study. Our preliminary theoretical model can help other practitioners identify what challenges they can expect to face in QR management in different developmental contexts as well as which practices to apply to mitigate these challenges., This work was supported by the European Union’s Horizon 2020 Research and Innovation Programme under Grant Agreement 732253., Peer Reviewed

Agile methods can deliver software that fulfills customer needs rapidly and continuously. Quality requirements (QRs) are important in this regard; however, detailed studies on how companies applying agile methods to manage QRs are limited, as are studies on the rationale for choosing specific QR management practices and related challenges. The aim of this study was to address why practitioners manage QRs as they do and what challenges they face. We also analyzed how existing practices mitigate some of the found challenges. Lastly, we connect the contextual elements of the companies with their practices and challenges. We conducted 36 interviews with practitioners from four companies of varying sizes. Since each company operates in different domains, comparing QR management strategies and related challenges in different contexts was possible. We found that the companies apply proactive, reactive, and interactive strategies to manage QRs. Additionally, our study revealed 40 challenges in six categories that companies applying agile methods may face in QR management. We also identified nine contextual elements that affect QR management practice choices and which, importantly, can explain many related challenges. Based on these findings, we constructed a theoretical model about the connection between context, QR management practices, and challenges. Practitioners in similar contexts can learn from the practices identified in this study. Our preliminary theoretical model can help other practitioners identify what challenges they can expect to face in QR management in different developmental contexts as well as which practices to apply to mitigate these challenges., This work was supported by the European Union’s Horizon 2020 Research and Innovation Programme under Grant Agreement 732253., Peer Reviewed, Postprint (published version)

Context:
Quality requirements (QRs) describe the desired quality of software, and they play an important role in the success of software projects. In agile software development (ASD), QRs are often ill-defined and not well addressed due to the focus on quickly delivering functionality. Rapid software development (RSD) approaches (e.g., continuous delivery and continuous deployment), which shorten delivery times, are more prone to neglect QRs. Despite the significance of QRs in both ASD and RSD, there is limited synthesized knowledge on their management in those approaches.

Objective:
This study aims to synthesize state-of-the-art knowledge about QR management in ASD and RSD, focusing on three aspects: bibliometric, strategies, and challenges.

Research method:
Using a systematic mapping study with a snowballing search strategy, we identified and structured the literature on QR management in ASD and RSD.

Results:
We found 156 primary studies: 106 are empirical studies, 16 are experience reports, and 34 are theoretical studies. Security and performance were the most commonly reported QR types. We identified various QR management strategies: 74 practices, 43 methods, 13 models, 12 frameworks, 11 advices, 10 tools, and 7 guidelines. Additionally, we identified 18 categories and 4 non-recurring challenges of managing QRs. The limited ability of ASD to handle QRs, time constraints due to short iteration cycles, limitations regarding the testing of QRs and neglect of QRs were the top categories of challenges.

Conclusion:
Management of QRs is significant in ASD and is becoming important in RSD. This study identified research gaps, such as the need for more tools and guidelines, lightweight QR management strategies that fit short iteration cycles, investigations of the link between QRs challenges and technical debt, and extension of empirical validation of existing strategies to a wider context. It also synthesizes QR management strategies and challenges, which may be useful for practitioners., Peer Reviewed, Postprint (author's final draft)

Software and data analytics solutions support improving development processes and the quality of the software produced in Agile Software Development (ASD). However, decision makers in software teams (e.g., product owner, project manager) are demanding powerful tools providing evidence data that support their strategic decision-making processes. In this paper, we present and provide access to QaSD, a Quality-aware Strategic Dashboard supporting decision makers in ASD. The dashboard allows decision makers to define high-level strategic indicators (e.g., customer satisfaction, process performance) related to software quality and to measure, explore, simulate and forecast the values of those indicators in order to explain and justify their decisions. Moreover, we also provide the results of a conducted evaluation of the dashboard quality in a real environment that evaluated the QaSD as usable, easy to use, with good navigation, and reliable, This work was supported by Q-Rapids (Quality-Aware Rapid Software Development) and VISDOM (Visual diagnosis for DevOps software development) projects. Q-Rapids was funded by the European Union’s Horizon 2020 research and innovation programme under grant agreement n◦ 732253. VISDOM is funded by the Research, Development and Innovation cluster programme ITEA 3 (project no 17038)., Peer Reviewed, Postprint (author's final draft)

Context: Exploiting software development related data from software-development intensive organizations to support tactical and strategic decision making is a challenge. Combining data-driven approaches with expert knowledge has been highlighted as a sensible approach for leading software-development intensive organizations to rightful decision-making improvements. However, most of the existing proposals lack of important aspects that hinders their industrial uptake such as: customization guidelines to fit the proposals to other contexts and/or automatic or semi-automatic data collection support for putting them forward in a real organization. As a result, existing proposals are rarely used in the industrial context.

Objective: Support software-development intensive organizations with guidance and tools for exploiting software development related data and expert knowledge to improve their decision making.

Method: We have developed a novel method called SESSI (Specification and Estimation of Software Strategic Indicators) that was articulated from industrial experiences with Nokia, Bittium, Softeam and iTTi in the context of Q-Rapids European project following a design science approach. As part of the industrial summative evaluation, we performed the first case study focused on the application of the method.

Results: We detail the phases and steps of the SESSI method and illustrate its application in the development of ModelioNG, a software product of Modeliosoft development firm.

Conclusion: The application of the SESSI method in the context of ModelioNG case study has provided us with useful feedback to improve the method and has evidenced that applying the method was feasible in this context., This work was supported by the Spanish Ministry of Economy and Competitiveness (project GENESIS, grant agreement TIN2016-79269- R), and by the European Union’s Horizon 2020 research and innovation program (Q-Rapids project, grant agreement N◦732253)., Peer Reviewed, Postprint (author's final draft)

Quality requirements (QRs) are a key artifact needed to ensure the quality and success of a software system. Despite their importance, QRs rarely get the same degree of attention as their functional counterpart in agile software development (ASD) projects. Moreover, crucial information that can be obtained from software development repositories (e.g., JIRA, GitHub) is not fully exploited, or is even neglected, in QR elicitation activities. In this work, we present a data-driven tooled approach for the semi-automatic generation and documentation of QRs in the context of ASD. The approach is based on the declaration of thresholds over quality-related issues, whose violation triggers user-defined alerts. These alerts are used to browse a catalog of QR patterns that are presented to the ASD team by means of a dashboard that implements several analysis techniques. Once selected, the patterns generate the QRs, which are documented and stored in the product backlog. The full approach is implemented via a configurable platform. Over the course of 1 year, four companies differing in size and profile followed this approach and deployed the platform in their premises to semi-automatically generate QRs in several projects. We used standardized measurement instruments to elicit the perception of 22 practitioners regarding their use of the tool. The quantitative and qualitative analyses yielded positive results; i.e., the practitioners’ perception with regard to the tool’s understandability, reliability, usefulness, and relevance was positive. We conclude that the results show potential for future adoption of data-driven elicitation of QRs in agile companies and encourage other practitioners to use the presented tool and adopt it in their companies., This work is a result of the Q-Rapids project, which has received funding from the European Union’s Horizon 2020 Research and Innovation Programme under grant agreement no. 732253., Peer Reviewed, Postprint (author's final draft)