The Cloudification Perspectives of Search-based Software Testing
1 like482 views
The document discusses the role of search-based software testing (SBST) in enhancing the testing practices and validation of cloud-native applications (CNAs) amidst the growing challenges in cloud computing. It highlights the need for automated solutions to ensure effective verification of CNA properties such as elasticity and resilience, and summarizes existing literature while outlining research challenges and opportunities in this domain. The work aims to address the untapped potential of SBST in supporting cloud development and testing processes.
![Literature Review on Testing for the Cloud
Test Suite
Generation
The Cloudification Perspectives of Search-based Software Testing
Diego Martin
Zurich University of Applied Science (ZHAW)
Winterthur, Switzerland
marg@zhaw.ch
Sebastiano Panichella
Zurich University of Applied Science (ZHAW)
Winterthur, Switzerland
panc@zhaw.ch
Abstract—To promote and sustain the future of our society,
the most critical challenge of contemporary software engineer-
ing and cloud computing experts are related to the efficient
integration of emerging cloudification and DevOps practices
in the development and testing processes of modern systems.
In this context, we argue that SBST can play a critical role
in improving testing practices and automating the verification
and validation (V&V) of cloudification properties of Cloud
Native Applications (CNA). Hence, in this paper, we focus on
the untouched side of SBST in the cloud field, by discussing
(1) the testing challenges in the cloud research field and (2)
summarizing the recent contributions of SBST in supporting
development practices of CNA. Finally, we discuss the emerging
research topics characterizing the cloudification perspectives of
SBST in the cloud field.
Keywords-Cloud Native Applications, Search-based Software
Testing, Test Suite Generation
I. INTRODUCTION
The first work on Search-Based Software Testing (SBST)
appeared in 1976 [1] and in the last decade the SBST
research field reached a high maturity, with several research
work and tools [2]–[5] aimed at supporting test data gener-
ation and test suite quality assessment [6]–[9].
The most critical challenge of software engineering and
cloud computing experts is related to the efficient integration
of emerging cloudification [10] and DevOps practices in the
development and testing processes of modern systems. This
put in place the need of solutions ensuring, quantifying, and
verifying the elastic scalability (i.e., adjusting their capacity
by adding or removing resources) and resiliency (i.e., antici-
pating failures and fluctuation) [11] of the software/hardware
(micro)services composing the systems.
In the cloud field, SBST strategies have been recently
experimented for task scheduling [12] and service composi-
tion [13]. We argue that SBST can play a critical role in the
supporting the verification and validation (V&V) of cloud
native applications. Hence, in this paper, we summarize the
main challenges and opportunities on the untouched side of
the SBST in the cloud field.
II. ANALYSIS AND DISCUSSION
A. Cloud Testing Literature Insights
Cloud-native applications (CNAs) are “distributed, elas-
tic and horizontal-scalable systems composed of (mi-
cro)services which isolates states in a minimum of stateful
components” [14]. Each self-contained deployment unit
of CNAs is designed according to cloud-focused design
patterns and operated on a self-service elastic platform.
This means that CNAs are supposed to be composable,
decoupled, elastic and resilient [14].
Even thought most of these properties have been inves-
tigated/analyzed in the literature [15], [16], there are still
open problems/challenges where SBST could contribute:
1) V&V of CNA properties: Provide automated solu-
tions for V&V of CNA properties like composition, decou-
pling, elasticity and resilience.
2) CNAs Adaptations: Provide tools for V&V of cloud
migrations or migration between cloud providers.
3) Microservices Evolution: Support developers with
tools providing an architectural view of microservices evo-
lution, with V&V on the side-effects of specific changes in
microservices composition and orchestration.
4) Local v.s. Global V&V: Provide automated V&V of
microservices evolutions that verify the behaviour of each
microservice in isolation (local level) as well as the global
microservices behaviour and interactions in the system.
5) Execution Time: Execution of test suites should
be efficient and effective, and should encapsulate advanced
coverage criteria for microservices based architecture.
B. Literature review of SBST for the cloud
SBST literature in the cloud field focused on the optimization
of testing frameworks [17], [18] and the stress of basic CNA
properties (elasticity [19] and resilience [20]), with some work
analyzing the challenges of cloud migrations [21], and the possi-
bilities of using combinatorial testing [22]. However, some of the
the challenges previously identified are completely untouched.
We conducted a literature review on SBST papers pub-
lished in the last 5 years (period 2014-2019), focusing on
the works that contributed to the cloud research field.
The literature review has been performed by using DBLP
as main source of information. Specifically, in the period
analyzed, the selection of papers in DBLP was performed
by using specific sets of search keywords, according to the
following logic formula: CLOUD ^ (GENETIC _ COMBINATORIAL _
SIMULATED ANNEALING _ TABU SEARCH _ PROFILING _ SLICING _
COVERAGE _ COEVOLUTION _ MUTATION _ MORPH _ HEALING _
SELF REPAIR _ HILL CLIMBING _ SEARCH BASED).
The aforementioned filter resulted in only 6 papers rele-
vant to SBST applications in the cloud field from the 396
“…we discuss how automated testing is
a potential solution to the challenges
identified in the cloud testing field”
SBST 2019
!21](https://image.slidesharecdn.com/talk-sbst-2019-190527162444/85/The-Cloudification-Perspectives-of-Search-based-Software-Testing-21-320.jpg)
![The Cloudification Perspectives of Search-based Software Testing
Diego Martin
Zurich University of Applied Science (ZHAW)
Winterthur, Switzerland
marg@zhaw.ch
Sebastiano Panichella
Zurich University of Applied Science (ZHAW)
Winterthur, Switzerland
panc@zhaw.ch
Abstract—To promote and sustain the future of our society,
the most critical challenge of contemporary software engineer-
ing and cloud computing experts are related to the efficient
integration of emerging cloudification and DevOps practices
in the development and testing processes of modern systems.
In this context, we argue that SBST can play a critical role
in improving testing practices and automating the verification
and validation (V&V) of cloudification properties of Cloud
Native Applications (CNA). Hence, in this paper, we focus on
the untouched side of SBST in the cloud field, by discussing
(1) the testing challenges in the cloud research field and (2)
summarizing the recent contributions of SBST in supporting
development practices of CNA. Finally, we discuss the emerging
research topics characterizing the cloudification perspectives of
SBST in the cloud field.
Keywords-Cloud Native Applications, Search-based Software
Testing, Test Suite Generation
I. INTRODUCTION
The first work on Search-Based Software Testing (SBST)
appeared in 1976 [1] and in the last decade the SBST
research field reached a high maturity, with several research
work and tools [2]–[5] aimed at supporting test data gener-
ation and test suite quality assessment [6]–[9].
The most critical challenge of software engineering and
cloud computing experts is related to the efficient integration
of emerging cloudification [10] and DevOps practices in the
development and testing processes of modern systems. This
put in place the need of solutions ensuring, quantifying, and
verifying the elastic scalability (i.e., adjusting their capacity
by adding or removing resources) and resiliency (i.e., antici-
pating failures and fluctuation) [11] of the software/hardware
(micro)services composing the systems.
In the cloud field, SBST strategies have been recently
experimented for task scheduling [12] and service composi-
tion [13]. We argue that SBST can play a critical role in the
supporting the verification and validation (V&V) of cloud
native applications. Hence, in this paper, we summarize the
main challenges and opportunities on the untouched side of
the SBST in the cloud field.
II. ANALYSIS AND DISCUSSION
A. Cloud Testing Literature Insights
Cloud-native applications (CNAs) are “distributed, elas-
tic and horizontal-scalable systems composed of (mi-
cro)services which isolates states in a minimum of stateful
components” [14]. Each self-contained deployment unit
of CNAs is designed according to cloud-focused design
patterns and operated on a self-service elastic platform.
This means that CNAs are supposed to be composable,
decoupled, elastic and resilient [14].
Even thought most of these properties have been inves-
tigated/analyzed in the literature [15], [16], there are still
open problems/challenges where SBST could contribute:
1) V&V of CNA properties: Provide automated solu-
tions for V&V of CNA properties like composition, decou-
pling, elasticity and resilience.
2) CNAs Adaptations: Provide tools for V&V of cloud
migrations or migration between cloud providers.
3) Microservices Evolution: Support developers with
tools providing an architectural view of microservices evo-
lution, with V&V on the side-effects of specific changes in
microservices composition and orchestration.
4) Local v.s. Global V&V: Provide automated V&V of
microservices evolutions that verify the behaviour of each
microservice in isolation (local level) as well as the global
microservices behaviour and interactions in the system.
5) Execution Time: Execution of test suites should
be efficient and effective, and should encapsulate advanced
coverage criteria for microservices based architecture.
B. Literature review of SBST for the cloud
SBST literature in the cloud field focused on the optimization
of testing frameworks [17], [18] and the stress of basic CNA
properties (elasticity [19] and resilience [20]), with some work
analyzing the challenges of cloud migrations [21], and the possi-
bilities of using combinatorial testing [22]. However, some of the
the challenges previously identified are completely untouched.
We conducted a literature review on SBST papers pub-
lished in the last 5 years (period 2014-2019), focusing on
the works that contributed to the cloud research field.
The literature review has been performed by using DBLP
as main source of information. Specifically, in the period
analyzed, the selection of papers in DBLP was performed
by using specific sets of search keywords, according to the
following logic formula: CLOUD ^ (GENETIC _ COMBINATORIAL _
SIMULATED ANNEALING _ TABU SEARCH _ PROFILING _ SLICING _
COVERAGE _ COEVOLUTION _ MUTATION _ MORPH _ HEALING _
SELF REPAIR _ HILL CLIMBING _ SEARCH BASED).
The aforementioned filter resulted in only 6 papers rele-
vant to SBST applications in the cloud field from the 396
1) Cloud Testing Challenges
2) Literature review Automated Testing
3) Cloud future perspectives
!22
Literature Review on Testing for the Cloud](https://image.slidesharecdn.com/talk-sbst-2019-190527162444/85/The-Cloudification-Perspectives-of-Search-based-Software-Testing-22-320.jpg)
![State-of-the-art on Testing For the Cloud
Test Suite
Generation
The Cloudification Perspectives of Search-based Software Testing
Diego Martin
Zurich University of Applied Science (ZHAW)
Winterthur, Switzerland
marg@zhaw.ch
Sebastiano Panichella
Zurich University of Applied Science (ZHAW)
Winterthur, Switzerland
panc@zhaw.ch
Abstract—To promote and sustain the future of our society,
the most critical challenge of contemporary software engineer-
ing and cloud computing experts are related to the efficient
integration of emerging cloudification and DevOps practices
in the development and testing processes of modern systems.
In this context, we argue that SBST can play a critical role
in improving testing practices and automating the verification
and validation (V&V) of cloudification properties of Cloud
Native Applications (CNA). Hence, in this paper, we focus on
the untouched side of SBST in the cloud field, by discussing
(1) the testing challenges in the cloud research field and (2)
summarizing the recent contributions of SBST in supporting
development practices of CNA. Finally, we discuss the emerging
research topics characterizing the cloudification perspectives of
SBST in the cloud field.
Keywords-Cloud Native Applications, Search-based Software
Testing, Test Suite Generation
I. INTRODUCTION
The first work on Search-Based Software Testing (SBST)
appeared in 1976 [1] and in the last decade the SBST
research field reached a high maturity, with several research
work and tools [2]–[5] aimed at supporting test data gener-
ation and test suite quality assessment [6]–[9].
The most critical challenge of software engineering and
cloud computing experts is related to the efficient integration
of emerging cloudification [10] and DevOps practices in the
development and testing processes of modern systems. This
put in place the need of solutions ensuring, quantifying, and
verifying the elastic scalability (i.e., adjusting their capacity
by adding or removing resources) and resiliency (i.e., antici-
pating failures and fluctuation) [11] of the software/hardware
(micro)services composing the systems.
In the cloud field, SBST strategies have been recently
experimented for task scheduling [12] and service composi-
tion [13]. We argue that SBST can play a critical role in the
supporting the verification and validation (V&V) of cloud
native applications. Hence, in this paper, we summarize the
main challenges and opportunities on the untouched side of
the SBST in the cloud field.
II. ANALYSIS AND DISCUSSION
A. Cloud Testing Literature Insights
Cloud-native applications (CNAs) are “distributed, elas-
tic and horizontal-scalable systems composed of (mi-
cro)services which isolates states in a minimum of stateful
components” [14]. Each self-contained deployment unit
of CNAs is designed according to cloud-focused design
patterns and operated on a self-service elastic platform.
This means that CNAs are supposed to be composable,
decoupled, elastic and resilient [14].
Even thought most of these properties have been inves-
tigated/analyzed in the literature [15], [16], there are still
open problems/challenges where SBST could contribute:
1) V&V of CNA properties: Provide automated solu-
tions for V&V of CNA properties like composition, decou-
pling, elasticity and resilience.
2) CNAs Adaptations: Provide tools for V&V of cloud
migrations or migration between cloud providers.
3) Microservices Evolution: Support developers with
tools providing an architectural view of microservices evo-
lution, with V&V on the side-effects of specific changes in
microservices composition and orchestration.
4) Local v.s. Global V&V: Provide automated V&V of
microservices evolutions that verify the behaviour of each
microservice in isolation (local level) as well as the global
microservices behaviour and interactions in the system.
5) Execution Time: Execution of test suites should
be efficient and effective, and should encapsulate advanced
coverage criteria for microservices based architecture.
B. Literature review of SBST for the cloud
SBST literature in the cloud field focused on the optimization
of testing frameworks [17], [18] and the stress of basic CNA
properties (elasticity [19] and resilience [20]), with some work
analyzing the challenges of cloud migrations [21], and the possi-
bilities of using combinatorial testing [22]. However, some of the
the challenges previously identified are completely untouched.
We conducted a literature review on SBST papers pub-
lished in the last 5 years (period 2014-2019), focusing on
the works that contributed to the cloud research field.
The literature review has been performed by using DBLP
as main source of information. Specifically, in the period
analyzed, the selection of papers in DBLP was performed
by using specific sets of search keywords, according to the
following logic formula: CLOUD ^ (GENETIC _ COMBINATORIAL _
SIMULATED ANNEALING _ TABU SEARCH _ PROFILING _ SLICING _
COVERAGE _ COEVOLUTION _ MUTATION _ MORPH _ HEALING _
SELF REPAIR _ HILL CLIMBING _ SEARCH BASED).
The aforementioned filter resulted in only 6 papers rele-
vant to SBST applications in the cloud field from the 396
SBST 2019
!38
Thanks for the
Attention!
Question?](https://image.slidesharecdn.com/talk-sbst-2019-190527162444/85/The-Cloudification-Perspectives-of-Search-based-Software-Testing-38-320.jpg)
The Cloudification Perspectives of Search-based Software Testing
- 1. Diego Martin Sebastiano Panichella Zurich University of Applied Science (ZHAW) [email protected] and [email protected] @spanichella https://www.zhaw.ch/en/about-us/person/panc/ or https://spanichella.github.io/ The Cloudification Perspectives of Search-based Software Testing Institute of Applied Information Technology (InIT) &
- 2. 1) Research Context and Motivation 3) Future Directions 2) Literature review on: - Cloud Testing Challenges - Automated Testing Outline PART I PART II !2
- 3. 1) Research Context and Motivation 3) Future Directions 2) Literature review on: - Cloud Testing Challenges - Automated Testing Outline PART I PART II !3
- 4. Versioning Control Repository (VCR) Continuous Integration (CI) Server Fail Fetch Changes Commit Commit Success Stakeholders Deployment Phase Release 5 6 2 3 4 Get Feedback Release Planning Decisions 1 Feedback Analysis Mechanism 7 Relevant Feedback Release Plan 4 Simplified Development Process Research Context and Motivation
- 5. Versioning Control Repository (VCR) Continuous Integration (CI) Server Fail Fetch Changes Commit Commit Success Stakeholders Deployment Phase Release 5 6 2 3 4 Get Feedback Release Planning Decisions 1 Feedback Analysis Mechanism 7 Relevant Feedback Release Plan 5 Simplified Development Process Research Context and Motivation
- 6. Versioning Control Repository (VCR) Continuous Integration (CI) Server Fail Fetch Changes Commit Commit Success Stakeholders Deployment Phase Release 5 6 2 3 4 Get Feedback Release Planning Decisions 1 Feedback Analysis Mechanism 7 Relevant Feedback Release Plan 6 Simplified Development Process Research Context and Motivation
- 7. Versioning Control Repository (VCR) Continuous Integration (CI) Server Fail Fetch Changes Commit Commit Success Stakeholders Deployment Phase Release 5 6 2 3 4 Get Feedback Release Planning Decisions 1 Feedback Analysis Mechanism 7 Relevant Feedback Release Plan 7 Simplified Development Process Research Context and Motivation
- 8. Versioning Control Repository (VCR) Continuous Integration (CI) Server Fail Fetch Changes Commit Commit Success Stakeholders Deployment Phase Release 5 6 2 3 4 Get Feedback Release Planning Decisions 1 Feedback Analysis Mechanism 7 Relevant Feedback Release Plan 8 Simplified Development Process Research Context and Motivation
- 9. Versioning Control Repository (VCR) Continuous Integration (CI) Server Fail Fetch Changes Commit Commit Success Stakeholders Deployment Phase Release 5 6 2 3 4 Get Feedback Release Planning Decisions 1 Feedback Analysis Mechanism 7 Relevant Feedback Release Plan 9 Simplified Development Process Research Context and Motivation
- 10. Versioning Control Repository (VCR) Continuous Integration (CI) Server Fail Fetch Changes Commit Commit Success Stakeholders Deployment Phase Release 5 6 2 3 4 Get Feedback Release Planning Decisions 1 Feedback Analysis Mechanism 7 Relevant Feedback Release Plan 10 Simplified Development Process Research Context and Motivation We argue that SBST can play a critical role in the supporting the verification and validation (V&V) of complex (e.g., cloud-native) applications
- 11. Versioning Control Repository (VCR) Continuous Integration (CI) Server Fail Fetch Changes Commit Commit Success Stakeholders Deployment Phase Release 5 6 2 3 4 Get Feedback Release Planning Decisions 1 Feedback Analysis Mechanism 7 Relevant Feedback Release Plan 11 Simplified Development Process We argue that SBST can play a critical role in the supporting the verification and validation (V&V) of complex (e.g., cloud-native) applications Industrial Relevance of the Problem?
- 12. https://bit.ly/2LGg7nI “Why and How to Get a PhD?” Lionel Briand Research Context 5-10% Join !12 Industrial Relevance of the Problem?
- 13. https://bit.ly/2LGg7nI “Why and How to Get a PhD?” Lionel Briand Research Context 5-10% Join !13 Initial Validation Industrial Validation Problem Statement Industrial Relevance of the Problem?
- 14. https://bit.ly/2LGg7nI “Why and How to Get a PhD?” Lionel Briand Research Context 5-10% Join Activities !14 ? Problem Statement Industrial Relevance of the Problem?
- 15. Research Context Millions of embedded systems (controlled by complex embedded software) are connected over the Internet and collect information about the real world 15 Etc.
- 16. 2) Artificial Intelligence (AI) 3) DevOps, IoT, Intelligence (AI), Automated Testing (AT) 1) Robotics Millions of embedded systems (controlled by complex embedded software) are connected over the Internet and collect information about the real world 16 Etc. Research Challenges and Opportunities
- 17. 2) Artificial Intelligence (AI) 3) DevOps, IoT, Intelligence (AI), Automated Testing (AT) 1) Robotics Research Challenges and Opportunities 17 75% of these companies interested to solutions for debugging and testing their cloud-native applications Test Suite Generation
- 18. 1) Research Context and Motivation 3) Future Directions 2) Literature review on: - Cloud Testing Challenges - Automated Testing Outline PART I PART II !18
- 19. https://bit.ly/2LGg7nI “Why and How to Get a PhD?” Lionel Briand Research Context 5-10% Join Testing for the Cloud is a Problem Activities !19
- 20. https://bit.ly/2LGg7nI “Why and How to Get a PhD?” Lionel Briand Research Context 5-10% Join Activities !20 Problem Statement Literature Review on Testing for the Cloud
- 21. Literature Review on Testing for the Cloud Test Suite Generation The Cloudification Perspectives of Search-based Software Testing Diego Martin Zurich University of Applied Science (ZHAW) Winterthur, Switzerland [email protected] Sebastiano Panichella Zurich University of Applied Science (ZHAW) Winterthur, Switzerland [email protected] Abstract—To promote and sustain the future of our society, the most critical challenge of contemporary software engineer- ing and cloud computing experts are related to the efficient integration of emerging cloudification and DevOps practices in the development and testing processes of modern systems. In this context, we argue that SBST can play a critical role in improving testing practices and automating the verification and validation (V&V) of cloudification properties of Cloud Native Applications (CNA). Hence, in this paper, we focus on the untouched side of SBST in the cloud field, by discussing (1) the testing challenges in the cloud research field and (2) summarizing the recent contributions of SBST in supporting development practices of CNA. Finally, we discuss the emerging research topics characterizing the cloudification perspectives of SBST in the cloud field. Keywords-Cloud Native Applications, Search-based Software Testing, Test Suite Generation I. INTRODUCTION The first work on Search-Based Software Testing (SBST) appeared in 1976 [1] and in the last decade the SBST research field reached a high maturity, with several research work and tools [2]–[5] aimed at supporting test data gener- ation and test suite quality assessment [6]–[9]. The most critical challenge of software engineering and cloud computing experts is related to the efficient integration of emerging cloudification [10] and DevOps practices in the development and testing processes of modern systems. This put in place the need of solutions ensuring, quantifying, and verifying the elastic scalability (i.e., adjusting their capacity by adding or removing resources) and resiliency (i.e., antici- pating failures and fluctuation) [11] of the software/hardware (micro)services composing the systems. In the cloud field, SBST strategies have been recently experimented for task scheduling [12] and service composi- tion [13]. We argue that SBST can play a critical role in the supporting the verification and validation (V&V) of cloud native applications. Hence, in this paper, we summarize the main challenges and opportunities on the untouched side of the SBST in the cloud field. II. ANALYSIS AND DISCUSSION A. Cloud Testing Literature Insights Cloud-native applications (CNAs) are “distributed, elas- tic and horizontal-scalable systems composed of (mi- cro)services which isolates states in a minimum of stateful components” [14]. Each self-contained deployment unit of CNAs is designed according to cloud-focused design patterns and operated on a self-service elastic platform. This means that CNAs are supposed to be composable, decoupled, elastic and resilient [14]. Even thought most of these properties have been inves- tigated/analyzed in the literature [15], [16], there are still open problems/challenges where SBST could contribute: 1) V&V of CNA properties: Provide automated solu- tions for V&V of CNA properties like composition, decou- pling, elasticity and resilience. 2) CNAs Adaptations: Provide tools for V&V of cloud migrations or migration between cloud providers. 3) Microservices Evolution: Support developers with tools providing an architectural view of microservices evo- lution, with V&V on the side-effects of specific changes in microservices composition and orchestration. 4) Local v.s. Global V&V: Provide automated V&V of microservices evolutions that verify the behaviour of each microservice in isolation (local level) as well as the global microservices behaviour and interactions in the system. 5) Execution Time: Execution of test suites should be efficient and effective, and should encapsulate advanced coverage criteria for microservices based architecture. B. Literature review of SBST for the cloud SBST literature in the cloud field focused on the optimization of testing frameworks [17], [18] and the stress of basic CNA properties (elasticity [19] and resilience [20]), with some work analyzing the challenges of cloud migrations [21], and the possi- bilities of using combinatorial testing [22]. However, some of the the challenges previously identified are completely untouched. We conducted a literature review on SBST papers pub- lished in the last 5 years (period 2014-2019), focusing on the works that contributed to the cloud research field. The literature review has been performed by using DBLP as main source of information. Specifically, in the period analyzed, the selection of papers in DBLP was performed by using specific sets of search keywords, according to the following logic formula: CLOUD ^ (GENETIC _ COMBINATORIAL _ SIMULATED ANNEALING _ TABU SEARCH _ PROFILING _ SLICING _ COVERAGE _ COEVOLUTION _ MUTATION _ MORPH _ HEALING _ SELF REPAIR _ HILL CLIMBING _ SEARCH BASED). The aforementioned filter resulted in only 6 papers rele- vant to SBST applications in the cloud field from the 396 “…we discuss how automated testing is a potential solution to the challenges identified in the cloud testing field” SBST 2019 !21
- 22. The Cloudification Perspectives of Search-based Software Testing Diego Martin Zurich University of Applied Science (ZHAW) Winterthur, Switzerland [email protected] Sebastiano Panichella Zurich University of Applied Science (ZHAW) Winterthur, Switzerland [email protected] Abstract—To promote and sustain the future of our society, the most critical challenge of contemporary software engineer- ing and cloud computing experts are related to the efficient integration of emerging cloudification and DevOps practices in the development and testing processes of modern systems. In this context, we argue that SBST can play a critical role in improving testing practices and automating the verification and validation (V&V) of cloudification properties of Cloud Native Applications (CNA). Hence, in this paper, we focus on the untouched side of SBST in the cloud field, by discussing (1) the testing challenges in the cloud research field and (2) summarizing the recent contributions of SBST in supporting development practices of CNA. Finally, we discuss the emerging research topics characterizing the cloudification perspectives of SBST in the cloud field. Keywords-Cloud Native Applications, Search-based Software Testing, Test Suite Generation I. INTRODUCTION The first work on Search-Based Software Testing (SBST) appeared in 1976 [1] and in the last decade the SBST research field reached a high maturity, with several research work and tools [2]–[5] aimed at supporting test data gener- ation and test suite quality assessment [6]–[9]. The most critical challenge of software engineering and cloud computing experts is related to the efficient integration of emerging cloudification [10] and DevOps practices in the development and testing processes of modern systems. This put in place the need of solutions ensuring, quantifying, and verifying the elastic scalability (i.e., adjusting their capacity by adding or removing resources) and resiliency (i.e., antici- pating failures and fluctuation) [11] of the software/hardware (micro)services composing the systems. In the cloud field, SBST strategies have been recently experimented for task scheduling [12] and service composi- tion [13]. We argue that SBST can play a critical role in the supporting the verification and validation (V&V) of cloud native applications. Hence, in this paper, we summarize the main challenges and opportunities on the untouched side of the SBST in the cloud field. II. ANALYSIS AND DISCUSSION A. Cloud Testing Literature Insights Cloud-native applications (CNAs) are “distributed, elas- tic and horizontal-scalable systems composed of (mi- cro)services which isolates states in a minimum of stateful components” [14]. Each self-contained deployment unit of CNAs is designed according to cloud-focused design patterns and operated on a self-service elastic platform. This means that CNAs are supposed to be composable, decoupled, elastic and resilient [14]. Even thought most of these properties have been inves- tigated/analyzed in the literature [15], [16], there are still open problems/challenges where SBST could contribute: 1) V&V of CNA properties: Provide automated solu- tions for V&V of CNA properties like composition, decou- pling, elasticity and resilience. 2) CNAs Adaptations: Provide tools for V&V of cloud migrations or migration between cloud providers. 3) Microservices Evolution: Support developers with tools providing an architectural view of microservices evo- lution, with V&V on the side-effects of specific changes in microservices composition and orchestration. 4) Local v.s. Global V&V: Provide automated V&V of microservices evolutions that verify the behaviour of each microservice in isolation (local level) as well as the global microservices behaviour and interactions in the system. 5) Execution Time: Execution of test suites should be efficient and effective, and should encapsulate advanced coverage criteria for microservices based architecture. B. Literature review of SBST for the cloud SBST literature in the cloud field focused on the optimization of testing frameworks [17], [18] and the stress of basic CNA properties (elasticity [19] and resilience [20]), with some work analyzing the challenges of cloud migrations [21], and the possi- bilities of using combinatorial testing [22]. However, some of the the challenges previously identified are completely untouched. We conducted a literature review on SBST papers pub- lished in the last 5 years (period 2014-2019), focusing on the works that contributed to the cloud research field. The literature review has been performed by using DBLP as main source of information. Specifically, in the period analyzed, the selection of papers in DBLP was performed by using specific sets of search keywords, according to the following logic formula: CLOUD ^ (GENETIC _ COMBINATORIAL _ SIMULATED ANNEALING _ TABU SEARCH _ PROFILING _ SLICING _ COVERAGE _ COEVOLUTION _ MUTATION _ MORPH _ HEALING _ SELF REPAIR _ HILL CLIMBING _ SEARCH BASED). The aforementioned filter resulted in only 6 papers rele- vant to SBST applications in the cloud field from the 396 1) Cloud Testing Challenges 2) Literature review Automated Testing 3) Cloud future perspectives !22 Literature Review on Testing for the Cloud
- 23. !23 Cloud Testing Challenges 1) Cloud Testing Challenges 2) Literature review Automated Testing 3) Cloud future perspectives
- 26. !26 Literature Review on Testing for the Cloud Search keywords formula (DBLP & Google Scholar): CLOUD ∧ ( GENETIC ∨ COMBINATORIAL ∨ SIMULATED ANNEALING ∨ TABU SEARCH ∨ PROFILING ∨ SLICING ∨ COVERAGE ∨ COEVOLUTION ∨ MUTATION ∨ MORPH ∨ HEALING ∨ SELF REPAIR ∨ HILL CLIMBING ∨ SEARCH BASED ) https://github.com/GoDieNow/CloudSBSTPerspectives Verifica(on and Valida(on (V&V) of Cloud-na(ve (CNA) proper(es (e.g., composi(on, elas(c scalability) CNAs Adapta(ons (e.g., migra(on) Microservices Evolu(on (e.g., providing an architectural view of microservices evolu(on) Local v.s. Global Verifica(on and Valida(on (e.g., V&V local microservices v.s. all microservices) Execu(on Time (Efficient and Effec(ve) Cloud Challenges
- 27. !27 Literature Review on Testing for the Cloud Verifica(on and Valida(on (V&V) of Cloud-na(ve (CNA) proper(es (e.g., composi(on, elas(c scalability) CNAs Adapta(ons (e.g., migra(on) Microservices Evolu(on (e.g., providing an architectural view of microservices evolu(on) Local v.s. Global Verifica(on and Valida(on (e.g., V&V local microservices v.s. all microservices) Execu(on Time (Efficient and Effec(ve) Cloud Challenges Only 6 papers addressing Cloud Challenges
- 28. !28 Literature Review on Testing for the Cloud Verifica(on and Valida(on (V&V) of Cloud-na(ve (CNA) proper(es (e.g., composi(on, elas(c scalability) CNAs Adapta(ons (e.g., migra(on) Microservices Evolu(on (e.g., providing an architectural view of microservices evolu(on) Local v.s. Global Verifica(on and Valida(on (e.g., V&V local microservices v.s. all microservices) Execu(on Time (Efficient and Effec(ve) Cloud Challenges Only 6 papers addressing Cloud Challenges SBST literature in the cloud field focused on the optimization of testing frameworks and the stress of basic CNA properties (elasticity and resilience)
- 30. Future Directions Verifica(on and Valida(on (V&V) of Cloud-na(ve (CNA) proper(es (e.g., composi(on, elas(c scalability) CNAs Adapta(ons (e.g., migra(on) Microservices Evolu(on (e.g., providing an architectural view of microservices evolu(on) Local v.s. Global Verifica(on and Valida(on (e.g., V&V local microservices v.s. all microservices) Execu(on Time (Efficient and Effec(ve) Cloud ChallengesTest case generation for the V&V of CNA properties is still a widely unexplored field Metrics and tools should be designed in order to guide the generation and the quality assessment of corresponding tests. SBST strategies, encapsulating smart mechanisms able to assess the inputs that most impact, affect, interacts with the coordination and evolution of the microservices architecture.
- 31. Future Directions Verifica(on and Valida(on (V&V) of Cloud-na(ve (CNA) proper(es (e.g., composi(on, elas(c scalability) CNAs Adapta(ons (e.g., migra(on) Microservices Evolu(on (e.g., providing an architectural view of microservices evolu(on) Local v.s. Global Verifica(on and Valida(on (e.g., V&V local microservices v.s. all microservices) Execu(on Time (Efficient and Effec(ve) Cloud Challenges SBST techniques have a perfect match the assessment/verification of CNA adaptation and migration challenges Profiling and current monitoring techniques can be combined with SBST strategies to better verify/monitor flow bottlenecks in the execution time … helping developers achieve the typical execution-time requirements of CNA.
- 32. https://bit.ly/2LGg7nI “Why and How to Get a PhD?” Lionel Briand Research Context 5-10% Join CNA Activities !32 Future Work
- 33. Serverless Day (Zurich) 33 Future work
- 34. 34 Future work Serverless Day (Zurich)
- 35. 35 Future work Serverless Day (Zurich)
- 36. 36 Future work Serverless Day (Zurich)
- 37. “Beyond Unit Testing?” Beyond Unit-Testing in Search-Based Test Case Generation: Challenges and Opportunities !37 “Do not miss the next related talk”
- 38. State-of-the-art on Testing For the Cloud Test Suite Generation The Cloudification Perspectives of Search-based Software Testing Diego Martin Zurich University of Applied Science (ZHAW) Winterthur, Switzerland [email protected] Sebastiano Panichella Zurich University of Applied Science (ZHAW) Winterthur, Switzerland [email protected] Abstract—To promote and sustain the future of our society, the most critical challenge of contemporary software engineer- ing and cloud computing experts are related to the efficient integration of emerging cloudification and DevOps practices in the development and testing processes of modern systems. In this context, we argue that SBST can play a critical role in improving testing practices and automating the verification and validation (V&V) of cloudification properties of Cloud Native Applications (CNA). Hence, in this paper, we focus on the untouched side of SBST in the cloud field, by discussing (1) the testing challenges in the cloud research field and (2) summarizing the recent contributions of SBST in supporting development practices of CNA. Finally, we discuss the emerging research topics characterizing the cloudification perspectives of SBST in the cloud field. Keywords-Cloud Native Applications, Search-based Software Testing, Test Suite Generation I. INTRODUCTION The first work on Search-Based Software Testing (SBST) appeared in 1976 [1] and in the last decade the SBST research field reached a high maturity, with several research work and tools [2]–[5] aimed at supporting test data gener- ation and test suite quality assessment [6]–[9]. The most critical challenge of software engineering and cloud computing experts is related to the efficient integration of emerging cloudification [10] and DevOps practices in the development and testing processes of modern systems. This put in place the need of solutions ensuring, quantifying, and verifying the elastic scalability (i.e., adjusting their capacity by adding or removing resources) and resiliency (i.e., antici- pating failures and fluctuation) [11] of the software/hardware (micro)services composing the systems. In the cloud field, SBST strategies have been recently experimented for task scheduling [12] and service composi- tion [13]. We argue that SBST can play a critical role in the supporting the verification and validation (V&V) of cloud native applications. Hence, in this paper, we summarize the main challenges and opportunities on the untouched side of the SBST in the cloud field. II. ANALYSIS AND DISCUSSION A. Cloud Testing Literature Insights Cloud-native applications (CNAs) are “distributed, elas- tic and horizontal-scalable systems composed of (mi- cro)services which isolates states in a minimum of stateful components” [14]. Each self-contained deployment unit of CNAs is designed according to cloud-focused design patterns and operated on a self-service elastic platform. This means that CNAs are supposed to be composable, decoupled, elastic and resilient [14]. Even thought most of these properties have been inves- tigated/analyzed in the literature [15], [16], there are still open problems/challenges where SBST could contribute: 1) V&V of CNA properties: Provide automated solu- tions for V&V of CNA properties like composition, decou- pling, elasticity and resilience. 2) CNAs Adaptations: Provide tools for V&V of cloud migrations or migration between cloud providers. 3) Microservices Evolution: Support developers with tools providing an architectural view of microservices evo- lution, with V&V on the side-effects of specific changes in microservices composition and orchestration. 4) Local v.s. Global V&V: Provide automated V&V of microservices evolutions that verify the behaviour of each microservice in isolation (local level) as well as the global microservices behaviour and interactions in the system. 5) Execution Time: Execution of test suites should be efficient and effective, and should encapsulate advanced coverage criteria for microservices based architecture. B. Literature review of SBST for the cloud SBST literature in the cloud field focused on the optimization of testing frameworks [17], [18] and the stress of basic CNA properties (elasticity [19] and resilience [20]), with some work analyzing the challenges of cloud migrations [21], and the possi- bilities of using combinatorial testing [22]. However, some of the the challenges previously identified are completely untouched. We conducted a literature review on SBST papers pub- lished in the last 5 years (period 2014-2019), focusing on the works that contributed to the cloud research field. The literature review has been performed by using DBLP as main source of information. Specifically, in the period analyzed, the selection of papers in DBLP was performed by using specific sets of search keywords, according to the following logic formula: CLOUD ^ (GENETIC _ COMBINATORIAL _ SIMULATED ANNEALING _ TABU SEARCH _ PROFILING _ SLICING _ COVERAGE _ COEVOLUTION _ MUTATION _ MORPH _ HEALING _ SELF REPAIR _ HILL CLIMBING _ SEARCH BASED). The aforementioned filter resulted in only 6 papers rele- vant to SBST applications in the cloud field from the 396 SBST 2019 !38 Thanks for the Attention! Question?