Mining Assumptions for Software Components using Machine Learning
0 likes1,307 views
EPIcuRus is an approach to automatically generate assumptions for software components in cyber-physical systems modeled in Simulink. It uses machine learning techniques to mine assumptions from test case results. The approach includes generating tests cases using an important feature boundary test generation method, model checking candidate assumptions, and selecting the most informative safe assumptions. An evaluation on industrial case studies found it can learn non-vacuous assumptions for most requirements within a practical time limit, with the important feature boundary test generation performing best.
![Assumption Generation
!22
2
Node 1
Total data: 1000
Node 2
Total data: 494
98%
Label: fail
Node 3
Total data: 532
Node 4
Total data: 200
100%
Label: pass
Throttle1 < 60Throttle1 < 60Throttle1 < 60 Throttle1 ≥ 60Throttle1 ≥ 60Throttle1 ≥ 60
Throttle1 < 90Throttle1 < 90Throttle1 < 90 Throttle1 ≥ 90Throttle1 ≥ 90Throttle1 ≥ 90
Node 5
Total data: 332
Node 6
Total data: 145
32%
Label: pass
Node 7
Total data: 187
85%
Label: fail
Pwheel < 2Pwheel < 2Pwheel < 2 Pwheel ≥ 2Pwheel ≥ 2Pwheel ≥ 2
Node 1
Total data: 1000
Node 2
Total data: 494
98%
Label: fail
Node 3
Total data: 532
Node 4
Total data: 200
100%
Label: pass
Throttle1 < 60Throttle1 < 60Throttle1 < 60 Throttle1 ≥ 60Throttle1 ≥ 60Throttle1 ≥ 60
Throttle1 < 90Throttle1 < 90Throttle1 < 90 Throttle1 ≥ 90Throttle1 ≥ 90Throttle1 ≥ 90
Node 5
Total data: 332
Node 6
Total data: 145
32%
Label: pass
Node 7
Total data: 187
85%
Label: fail
Pwheel < 2Pwheel < 2Pwheel < 2 Pwheel ≥ 2Pwheel ≥ 2Pwheel ≥ 2
8t 2 [0, t1] : 60 Throttle(t) < 908t 2 [0, t1] : 60 Throttle(t) < 908t 2 [0, t1] : 60 Throttle(t) < 90
60 Throttle1 < 9060 Throttle1 < 9060 Throttle1 < 90](https://image.slidesharecdn.com/fse2020-epicurus-presentation-201112120503/85/Mining-Assumptions-for-Software-Components-using-Machine-Learning-22-320.jpg)
![Model Checking
!23
3
Exhaustively checks if the obtained assumption is accurate
• QVTrace from QRA Corp, Canada
• SMT-based model checker for Simulink
• Z3, Mathematica
2.4.Accessing QVtrace:
Once the QVtrace server is running, QVtrace will be accessed through a web browser with
the address: http://localhost:2999
If accessing the QVtrace server on a networked computer then use the address:
http://[server_name]:2999.
QVtrace has been fully tested to be accessed with the Google Chrome web browser.
Although other browsers may render QVtrace appropriately, these have not been fully
tested and their performance is not well known. We recommend you use the Google
Chrome browser for QVtrace.
3. Using QVtrace
3.1.Understanding the QVtrace user interface
QVtrace has been designed to optimize the workflow for model-based design analysis. The
interface has three main sections as shown in the image below and described in detail on
the next page.
QVtrace User Manual v0.11.7 qracorp.com of4 21
1
2
3](https://image.slidesharecdn.com/fse2020-epicurus-presentation-201112120503/85/Mining-Assumptions-for-Software-Components-using-Machine-Learning-23-320.jpg)
![IFBT-Important Feature Boundary
Test
[ ]][ ][
0% 100%
boundary areas
Conjecture-1Conjecture-2
Identifying control points with high
impact on the fitness value and
focusing the search on them
!24
Generating test cases in boundary
areas of the input domain
Two conjectures enable more effective learning of v-safe
assumptions.](https://image.slidesharecdn.com/fse2020-epicurus-presentation-201112120503/85/Mining-Assumptions-for-Software-Components-using-Machine-Learning-24-320.jpg)
![Node 1
Total data: 1000
Node 2
Total data: 494
-0.1
Node 3
Total data: 532
Node 4
Total data: 200
0.1
Throttle1 < 60Throttle1 < 60Throttle1 < 60 Throttle1 ≥ 60Throttle1 ≥ 60Throttle1 ≥ 60
Throttle1 < 90Throttle1 < 90Throttle1 < 90 Throttle1 ≥ 90Throttle1 ≥ 90Throttle1 ≥ 90
Node 5
Total data: 332
Node 6
Total data: 145
-0.3
Node 7
Total data: 187
0.8
Pwheel < 2Pwheel < 2Pwheel < 2 Pwheel ≥ 2Pwheel ≥ 2Pwheel ≥ 2
!28
IFBT-Important Feature Boundary
Test
4. Get the ranges associated with the most
important feature
[54 , 66]
[81 , 99]
How do we generate the test case?](https://image.slidesharecdn.com/fse2020-epicurus-presentation-201112120503/85/Mining-Assumptions-for-Software-Components-using-Machine-Learning-28-320.jpg)
![!29
IFBT-Important Feature Boundary
Test
5. For each test case, get the ranges associated
with the most important feature
How do we generate the test case?
Node 1
Total data: 1000
Node 2
Total data: 494
-0.1
Node 3
Total data: 532
Node 4
Total data: 200
0.1
Throttle1 < 60Throttle1 < 60Throttle1 < 60 Throttle1 ≥ 60Throttle1 ≥ 60Throttle1 ≥ 60
Throttle1 < 90Throttle1 < 90Throttle1 < 90 Throttle1 ≥ 90Throttle1 ≥ 90Throttle1 ≥ 90
Node 5
Total data: 332
Node 6
Total data: 145
-0.3
Node 7
Total data: 187
0.8
Pwheel < 2Pwheel < 2Pwheel < 2 Pwheel ≥ 2Pwheel ≥ 2Pwheel ≥ 2
[54 , 66]
[81 , 99]](https://image.slidesharecdn.com/fse2020-epicurus-presentation-201112120503/85/Mining-Assumptions-for-Software-Components-using-Machine-Learning-29-320.jpg)
More Related Content
Similar to Mining Assumptions for Software Components using Machine Learning (20)
More from Lionel Briand (20)
Recently uploaded (20)
Mining Assumptions for Software Components using Machine Learning
- 1. .lusoftware verification & validation VVS Mining Assumptions for Software Components using Machine Learning Khouloud Gaaloul Claudio Menghi Shiva Nejati Lionel C. Briand QRA Corp, Canada David Wolfe University of Luxembourg, Luxembourg University of Ottawa, Canada University of Luxembourg, Luxembourg University of Ottawa, Canada University of Luxembourg, Luxembourg University of Luxembourg, Luxembourg
- 3. Cyber-Physical Systems (CPS) Systems in which the physical and software components are deeply intertwined !3
- 4. Requirements Check Model Requirements Check !4 PHASE 1: Modeling (Simulink) PHASE 2: Verification PHASE 3: Coding PHASE 2: Verification Requirements Check
- 5. Problem • Usually, exhaustive verification can not analyse complex industrial models • However, model checking can verify models’ components • When a sub-component is analysed, assumptions are usually not explicitly documented. !5
- 6. • A software component to guide small aircrafts • Controls the aircraft orientation (Pitch, Roll, and Yaw) The Autopilot Case Study !6 Yaw Roll Pitch Yaw Roll Pitch Yaw Yaw Autopilot Indicators Actuators
- 7. The Autopilot Case Study !7 Altitude Control Component When the autopilot is enabled, the aircraft altitude should reach the desired altitude within 500 seconds in calm air = ? 0% 100% tt+500s desired altitude 20%40%60%80%
- 8. Goal !8 To provide the aircraft with enough boost so that it can reach the desired altitude, the pilot should manually adjust the power given to the engines of the aircraft to ensure that the aircraft does not enter a stall condition Advanced Avionics Handbook
- 9. Goal Our goal is to mine assumptions for software components !9 Altitude Control Component >
- 10. An assumption is v-safe for a model ! and its requirement ! if ! M hAiMh i > v !10 V-safe Assumption RequirementModelAssumption ! : Degree of satisfactionv an assumption is considered 0-safe, if the requirement is satisfied under the assumption v 0 v < 0
- 11. !11 The throttle is higher than 60% The throttle is higher than 80% is More informative than Informativeness 0% 100% 60% Altitude Control Component > 0% 100% 80% Altitude Control Component > A1 A2
- 12. Goal !12 The goal is to generate the most informative v-safe assumptions for software components
- 13. Preriquisites The model is specified in Simulink Preriquisite-1 Preriquisite-2 The requirement is in a logical languageThe model is supported by a model checker Preriquisite-3 The model satisfies neither the requirement nor its negation !13 Preriquisite-4
- 15. EPIcuRus (assumPtIon geneRation approach for CPS) We propose EPIcuRus, an automated approach to infer environment assumptions for system components !15
- 16. EPIcuRus (assumPtIon geneRation approach for CPS) !16
- 17. EPIcuRus (assumPtIon geneRation approach for CPS) !17
- 18. EPIcuRus (assumPtIon geneRation approach for CPS) !18
- 19. EPIcuRus (assumPtIon geneRation approach for CPS) !19 IFBT
- 20. Generates input signals that are meaningful The signals are encoded using these parameters: Test Generation !20 1 • The input domain • The number of control points • The interpolation function
- 21. Model Requirements Check Input (U) Output (Y) Test Generation1 Test Generation Policy UR, ART, … !21 Pass/Fail Altitude Control Component
- 22. Assumption Generation !22 2 Node 1 Total data: 1000 Node 2 Total data: 494 98% Label: fail Node 3 Total data: 532 Node 4 Total data: 200 100% Label: pass Throttle1 < 60Throttle1 < 60Throttle1 < 60 Throttle1 ≥ 60Throttle1 ≥ 60Throttle1 ≥ 60 Throttle1 < 90Throttle1 < 90Throttle1 < 90 Throttle1 ≥ 90Throttle1 ≥ 90Throttle1 ≥ 90 Node 5 Total data: 332 Node 6 Total data: 145 32% Label: pass Node 7 Total data: 187 85% Label: fail Pwheel < 2Pwheel < 2Pwheel < 2 Pwheel ≥ 2Pwheel ≥ 2Pwheel ≥ 2 Node 1 Total data: 1000 Node 2 Total data: 494 98% Label: fail Node 3 Total data: 532 Node 4 Total data: 200 100% Label: pass Throttle1 < 60Throttle1 < 60Throttle1 < 60 Throttle1 ≥ 60Throttle1 ≥ 60Throttle1 ≥ 60 Throttle1 < 90Throttle1 < 90Throttle1 < 90 Throttle1 ≥ 90Throttle1 ≥ 90Throttle1 ≥ 90 Node 5 Total data: 332 Node 6 Total data: 145 32% Label: pass Node 7 Total data: 187 85% Label: fail Pwheel < 2Pwheel < 2Pwheel < 2 Pwheel ≥ 2Pwheel ≥ 2Pwheel ≥ 2 8t 2 [0, t1] : 60 Throttle(t) < 908t 2 [0, t1] : 60 Throttle(t) < 908t 2 [0, t1] : 60 Throttle(t) < 90 60 Throttle1 < 9060 Throttle1 < 9060 Throttle1 < 90
- 23. Model Checking !23 3 Exhaustively checks if the obtained assumption is accurate • QVTrace from QRA Corp, Canada • SMT-based model checker for Simulink • Z3, Mathematica 2.4.Accessing QVtrace: Once the QVtrace server is running, QVtrace will be accessed through a web browser with the address: http://localhost:2999 If accessing the QVtrace server on a networked computer then use the address: http://[server_name]:2999. QVtrace has been fully tested to be accessed with the Google Chrome web browser. Although other browsers may render QVtrace appropriately, these have not been fully tested and their performance is not well known. We recommend you use the Google Chrome browser for QVtrace. 3. Using QVtrace 3.1.Understanding the QVtrace user interface QVtrace has been designed to optimize the workflow for model-based design analysis. The interface has three main sections as shown in the image below and described in detail on the next page. QVtrace User Manual v0.11.7 qracorp.com of4 21 1 2 3
- 24. IFBT-Important Feature Boundary Test [ ]][ ][ 0% 100% boundary areas Conjecture-1Conjecture-2 Identifying control points with high impact on the fitness value and focusing the search on them !24 Generating test cases in boundary areas of the input domain Two conjectures enable more effective learning of v-safe assumptions.
- 25. !25 IFBT-Important Feature Boundary Test 1. Build a regression tree How do we generate the test case? Node 1 Total data: 1000 Node 2 Total data: 494 -0.1 Node 3 Total data: 532 Node 4 Total data: 200 0.1 Throttle1 < 60Throttle1 < 60Throttle1 < 60 Throttle1 ≥ 60Throttle1 ≥ 60Throttle1 ≥ 60 Throttle1 < 90Throttle1 < 90Throttle1 < 90 Throttle1 ≥ 90Throttle1 ≥ 90Throttle1 ≥ 90 Node 5 Total data: 332 Node 6 Total data: 145 -0.3 Node 7 Total data: 187 0.8 Pwheel < 2Pwheel < 2Pwheel < 2 Pwheel ≥ 2Pwheel ≥ 2Pwheel ≥ 2
- 26. !26 IFBT-Important Feature Boundary Test 2. Get the most important feature among the control points (Conjecture-1) How do we generate the test case? Node 1 Total data: 1000 Node 2 Total data: 494 -0.1 Node 3 Total data: 532 Node 4 Total data: 200 0.1 Throttle1 < 60Throttle1 < 60Throttle1 < 60 Throttle1 ≥ 60Throttle1 ≥ 60Throttle1 ≥ 60 Throttle1 < 90Throttle1 < 90Throttle1 < 90 Throttle1 ≥ 90Throttle1 ≥ 90Throttle1 ≥ 90 Node 5 Total data: 332 Node 6 Total data: 145 -0.3 Node 7 Total data: 187 0.8 Pwheel < 2Pwheel < 2Pwheel < 2 Pwheel ≥ 2Pwheel ≥ 2Pwheel ≥ 2
- 27. Node 1 Total data: 1000 Node 2 Total data: 494 -0.1 Node 3 Total data: 532 Node 4 Total data: 200 0.1 Throttle1 < 60Throttle1 < 60Throttle1 < 60 Throttle1 ≥ 60Throttle1 ≥ 60Throttle1 ≥ 60 Throttle1 < 90Throttle1 < 90Throttle1 < 90 Throttle1 ≥ 90Throttle1 ≥ 90Throttle1 ≥ 90 Node 5 Total data: 332 Node 6 Total data: 145 -0.3 Node 7 Total data: 187 0.8 Pwheel < 2Pwheel < 2Pwheel < 2 Pwheel ≥ 2Pwheel ≥ 2Pwheel ≥ 2 3. Extract the test cases that are the closest to the boundary (Conjecture-2) !27 IFBT-Important Feature Boundary Test How do we generate the test case?
- 28. Node 1 Total data: 1000 Node 2 Total data: 494 -0.1 Node 3 Total data: 532 Node 4 Total data: 200 0.1 Throttle1 < 60Throttle1 < 60Throttle1 < 60 Throttle1 ≥ 60Throttle1 ≥ 60Throttle1 ≥ 60 Throttle1 < 90Throttle1 < 90Throttle1 < 90 Throttle1 ≥ 90Throttle1 ≥ 90Throttle1 ≥ 90 Node 5 Total data: 332 Node 6 Total data: 145 -0.3 Node 7 Total data: 187 0.8 Pwheel < 2Pwheel < 2Pwheel < 2 Pwheel ≥ 2Pwheel ≥ 2Pwheel ≥ 2 !28 IFBT-Important Feature Boundary Test 4. Get the ranges associated with the most important feature [54 , 66] [81 , 99] How do we generate the test case?
- 29. !29 IFBT-Important Feature Boundary Test 5. For each test case, get the ranges associated with the most important feature How do we generate the test case? Node 1 Total data: 1000 Node 2 Total data: 494 -0.1 Node 3 Total data: 532 Node 4 Total data: 200 0.1 Throttle1 < 60Throttle1 < 60Throttle1 < 60 Throttle1 ≥ 60Throttle1 ≥ 60Throttle1 ≥ 60 Throttle1 < 90Throttle1 < 90Throttle1 < 90 Throttle1 ≥ 90Throttle1 ≥ 90Throttle1 ≥ 90 Node 5 Total data: 332 Node 6 Total data: 145 -0.3 Node 7 Total data: 187 0.8 Pwheel < 2Pwheel < 2Pwheel < 2 Pwheel ≥ 2Pwheel ≥ 2Pwheel ≥ 2 [54 , 66] [81 , 99]
- 32. Evaluation 32
- 33. Research Questions • RQ1: Which test case generation policy learns assumptions most effectively and efficiently? • RQ2: Can EPIcuRus generate assumptions for real world Simulink models within a practical time limit? !33
- 34. RQ1: Effectiveness and Efficiency • RQ1: Which test case generation policy learns assumptions most effectively and efficiently? !34
- 35. RQ1: Effectiveness and Efficiency 11 case studies • Developed by a company in the defence and aerospace sector • Represent different types of CPS Simulink models • Each model has a list of (textual) functional requirements !35
- 36. RQ1: Effectiveness and Efficiency • 92 requirements • 18 satisfy the prerequisites • 74 violate the prerequisites • Evaluated the UR, ART, IFBT-UR, IFBT-ART test case generation policies !36
- 37. RQ1: Effectiveness and Efficiency • For each policy, we run EPIcuRus • We consider input signals with one (IP), two (IP’) and three (IP”) control points • we measured among 50 experiment runs: • V-SAFE: the percentage of runs, in which a v-safe assumption was computed • AVG_TIME: the average execution time • INF_IDX: the number of times an assumption was more informative than another !37
- 38. RQ1: Effectiveness and Efficiency !38 200 300 400 500 600 700 800 900 AVG_TIME (s) 54 56 58 60 62 64 V_SAFE(%) 978 991 1054 1030 UR ART IFBT-UR IFBT-ART
- 39. RQ1: Effectiveness and Efficiency !39 Conclusion1: Among the four test case generation policies we compared, IFBT-UR learns the most v-safe assumptions in less time 200 300 400 500 600 700 800 900 AVG_TIME (s) 54 56 58 60 62 64 V_SAFE(%) 978 991 1054 1030 UR ART IFBT-UR IFBT-ART
- 40. RQ1: Effectiveness and Efficiency !40 Conclusion2: The assumptions learned by IFBT- UR are more informative than those learned by other test generation policies 200 300 400 500 600 700 800 900 AVG_TIME (s) 54 56 58 60 62 64 V_SAFE(%) 978 991 1054 1030 UR ART IFBT-UR IFBT-ART
- 41. RQ2: Usefulness RQ2: Can EPIcuRus generate assumptions for real world Simulink models within a practical time limit? !41
- 42. • We select the best performing test case generation policy: IFBT-UR • We considered four models and 18 requirements. • Among 50 experiment runs: • We compute the percentage of requirements for which a v-safe assumption is computed • We examine the usefulness, the structure and the length of all the computed assumptions !42 RQ2: Usefulness
- 43. RQ2: Usefulness !43 • EPIcuRus computed a v-safe assumption, within one hour, for ≈78% of the requirements • Across all 50 runs, which take around four hours per requirement, EPIcuRus computed a v-safe assumption for all the 18 requirements • EPIcuRus learnt non-vacuous and short assumptions
- 44. Conclusions 44
- 45. Conclusions • EPIcuRus: infer assumptions for software components • It is applicable to complex signal-based modelling notations • Combines search-based software testing, Machine Learning and model checking • IFBT: A test case generation technique to guide the search through the most informative features and areas in the search space !45
- 46. Conclusion • We were able to compute v-safe assumptions • The computed assumptions are short and non-vacuous • Assumptions are computed based on a large set of requirements !46
- 47. .lusoftware verification & validation VVS Mining Assumptions for Software Components using Machine Learning Khouloud Gaaloul Claudio Menghi Shiva Nejati Lionel C. Briand QRA Corp, Canada David Wolfe University of Luxembourg, Luxembourg University of Ottawa, Canada University of Luxembourg, Luxembourg University of Ottawa, Canada University of Luxembourg, Luxembourg University of Luxembourg, Luxembourg