Metamorphic Testing for Web System Security
0 likes101 views
This document summarizes a presentation on metamorphic testing for web system security given by Nazanin Bayati on September 13, 2023. Metamorphic testing uses relations between the outputs of multiple test executions to test systems when specifying expected outputs is difficult. It was applied to web systems by generating follow-up inputs based on transformations of valid interactions and checking that output relations held. The approach detected over 60% of vulnerabilities in tested systems and addressed more vulnerability types than static and dynamic analysis tools. It provides an effective and automated way to test for security issues in web systems.
Ad
More Related Content
What's hot (20)
Similar to Metamorphic Testing for Web System Security (20)
Ad
More from Lionel Briand (20)
Ad
Recently uploaded (20)
Metamorphic Testing for Web System Security
- 1. University of Ottawa | University of Luxembourg University of Ottawa | University of Luxembourg 1 Nazanin Bayati - Metamorphic Testing for Web System Security 13 September 2023 Journal First – IEEE Transaction on Software Engineering Presented by: Nazanin Bayati 13 September 2023 University of Ottawa University of Luxembourg Nazanin Bayati University of Ottawa Fabrizio Pastore University of Luxembourg Lionel Briand University of Ottawa University of Luxembourg Arda Goknil SINTEF Digital, Norway Metamorphic Testing for Web System Security
- 2. 2 Security vulnerabilities are subtle Discovered when testing with many inputs Specifying expected results is infeasible
- 3. University of Ottawa | University of Luxembourg University of Ottawa | University of Luxembourg 3 Nazanin Bayati - Metamorphic Testing for Web System Security 13 September 2023 Metamorphic Testing alleviates the Oracle Problem • Metamorphic Testing (MT) is based on the idea that • it may be simpler to reason about relations between outputs of multiple test executions, called Metamorphic Relations (MRs), than to specify the output of the system for a given input • In MT, system properties are captured as MRs that • specify how to automatically transform an initial set of test inputs (source inputs) into follow-up test inputs • specify the relation between the outputs obtained from source and follow-up inputs • A failure is observed when such relations are violated.
- 4. University of Ottawa | University of Luxembourg University of Ottawa | University of Luxembourg 4 Nazanin Bayati - Metamorphic Testing for Web System Security 13 September 2023 Metamorphic Security Testing • Source input: a sequence of valid interactions with the system {login(Admin), RequestURL(settings_page)} • Follow-up input: generated by altering valid interactions as an attacker would do {login(User1), RequestURL(settings_page)} • Relations: capture properties that hold when the system is not vulnerable if the user in the follow-up input cannot access the URL from her GUI then the output of the source and follow-up inputs should be different
- 5. University of Ottawa | University of Luxembourg University of Ottawa | University of Luxembourg 5 Nazanin Bayati - Metamorphic Testing for Web System Security 13 September 2023 MST-wi: Metamorphic Security Testing for Web Interfaces Web System Execute the Data Collection Framework Catalog of 76 Metamorphic Relations Select or Specify the Metamorphic Relations Execute the Metamorphic Testing Framework Test results Translate Metamorphic Relations to Java List of Metamorphic Relations Executable Metamorphic Relations in Java Source Inputs 1 2 3 4 Submit form logout Log in logout Log in
- 6. University of Ottawa | University of Luxembourg University of Ottawa | University of Luxembourg 6 Nazanin Bayati - Metamorphic Testing for Web System Security 13 September 2023 MST-wi – MR Example • Security issue: Bypass Authorization Schema
- 7. University of Ottawa | University of Luxembourg University of Ottawa | University of Luxembourg 7 Nazanin Bayati - Metamorphic Testing for Web System Security 13 September 2023 MST-wi – MR Example • Security issue: Bypass Authorization Schema
- 8. University of Ottawa | University of Luxembourg University of Ottawa | University of Luxembourg 8 Nazanin Bayati - Metamorphic Testing for Web System Security 13 September 2023 MST-wi – MR Example • Security issue: Bypass Authorization Schema
- 9. University of Ottawa | University of Luxembourg University of Ottawa | University of Luxembourg 9 Nazanin Bayati - Metamorphic Testing for Web System Security 13 September 2023 MST-wi – MR Example • Security issue: Bypass Authorization Schema
- 10. University of Ottawa | University of Luxembourg University of Ottawa | University of Luxembourg 10 Nazanin Bayati - Metamorphic Testing for Web System Security 13 September 2023 MST-wi – MR Example • Security issue: Bypass Authorization Schema
- 11. University of Ottawa | University of Luxembourg University of Ottawa | University of Luxembourg 11 Nazanin Bayati - Metamorphic Testing for Web System Security 13 September 2023 MST-wi – MR Example • Security issue: Bypass Authorization Schema
- 12. University of Ottawa | University of Luxembourg University of Ottawa | University of Luxembourg 12 Nazanin Bayati - Metamorphic Testing for Web System Security 13 September 2023 MST-wi – MR Example • Security issue: Bypass Authorization Schema Our metamorphic testing algorithm executes each MR multiple times, to ensure that every possible combination of source and follow-up inputs is exercised
- 13. University of Ottawa | University of Luxembourg University of Ottawa | University of Luxembourg 13 Nazanin Bayati - Metamorphic Testing for Web System Security 13 September 2023 MST-wi – Research Questions • RQ1. What testing activities can be automated thanks to oracle automation provided by MST-wi? • RQ2. What vulnerability types can MST-wi detect? • RQ3. What testability guidelines can we define to enable effective test automation with MST-wi? • RQ4. How does MST-wi compare to state-of-the-art SAST and DAST tools? • RQ5. Can we identify patterns for writing MST-wi relations? • RQ6. Is MST-wi effective? • RQ7. Is MST-wi efficient?
- 14. University of Ottawa | University of Luxembourg University of Ottawa | University of Luxembourg 14 Nazanin Bayati - Metamorphic Testing for Web System Security 13 September 2023 MST-wi – What vulnerability types can MST-wi detect? • We investigated the feasibility of implementing MRs that discover the vulnerability types described in the MITRE Common Weakness Enumeration (CWE) database • Considered three subsets: • CWE view for common security architectural tactics • CWE Top 25 most dangerous software errors • OWASP Top 10 Web security risks • To implement an MR, for each weakness, we first inspect its description, its demonstrative examples, the description of concrete vulnerabilities (CVE) and common attack patterns (CAPEC) associated with the weakness. • This process led to a catalog of 76 MRs.
- 15. University of Ottawa | University of Luxembourg University of Ottawa | University of Luxembourg 15 Nazanin Bayati - Metamorphic Testing for Web System Security 13 September 2023 MST-wi – What vulnerability types can MST-wi detect? Security Design Principle Vulnerability types Addressed by MST-wi Rank Audit 6 1(16%) 10th Authenticate Actors 28 12 (43%) 4th Authorize Actors 60 34 (57%) 3rd Cross Cutting 9 3 (33%) 6th Encrypt Data 38 8 (21%) 8th Identify Actors 12 3 (25%) 7th Limit Access 8 3 (38%) 5th Limit Exposure 6 0 (0%) 11th Lock Computer 1 0 (0%) 11th Manage User Session 6 4 (67%) 2nd Validate Inputs 39 31 (79%) 1st Verify Message Integrity 19 2 (20%) 9th Total 223 101 (45%) Summary of the CWE architectural security design principles and weaknesses addressed by MST-wi.
- 16. University of Ottawa | University of Luxembourg University of Ottawa | University of Luxembourg 16 Nazanin Bayati - Metamorphic Testing for Web System Security 13 September 2023 MST-wi – How does MST-wi compare to state-of-the-art SAST and DAST tools? • We compared the vulnerability types detected by MST-wi, with the vulnerability types detected by state- of-the-art SAST and DAST tool reported in a recent empirical study
- 17. University of Ottawa | University of Luxembourg University of Ottawa | University of Luxembourg 17 Nazanin Bayati - Metamorphic Testing for Web System Security 13 September 2023 MST-wi – How does MST-wi compare to state-of-the-art SAST and DAST tools? Security Design Principle Weaknesses Addresses by Weaknesses Addressed by MST but not by Weaknesses bot addresses by MST but addresses by MST Zap DA2 Sonar SA2 Zap DA2 Sonar SA2 Zap DA2 Sonar SA2 Audit 1 0 0 0 3 1 1 1 0 0 0 0 2 Authenticate Actors 12 0 2 1 9 12 11 11 7 0 1 0 4 Authorize Actors 34 2 0 1 13 32 34 34 25 0 0 1 4 Cross Cutting 3 0 0 2 0 3 3 2 3 0 0 1 0 Encrypt Data 8 2 5 8 10 8 8 7 4 2 5 7 6 Identify Actors 3 1 1 1 7 3 3 3 1 1 1 1 5 Limit Access 3 0 1 1 5 3 3 2 0 0 1 0 2 Limit Exposure 0 1 0 0 1 0 0 0 0 1 0 0 1 Lock Computer 0 0 0 0 0 0 0 0 0 0 0 0 0 Manage User Session 4 0 0 0 2 4 4 4 2 0 0 0 0 Validate Inputs 31 10 7 2 14 24 25 30 19 3 1 1 2 Verify Message Integrity 2 1 0 0 3 2 2 2 1 1 0 0 2 Total 101 17 16 16 67 92 94 96 62 8 9 11 28 84 The set of weaknesses targeted by MST-wi is larger than what can be targeted by applying all four competing approaches together.
- 18. University of Ottawa | University of Luxembourg University of Ottawa | University of Luxembourg 18 Nazanin Bayati - Metamorphic Testing for Web System Security 13 September 2023 MST-wi – Is MST-wi effective? Applied MST-wi to test well-known Web systems: • Jenkins v 2.121 • Joomla v. 3.8.7. Assessed MST-wi capability to detect known vulnerabilities: • 11 for Jenkins, 3 for Joomla. • One of them discovered by MST-wi (CVE-2018-17857) Considered two setups: • Derive source inputs with crawler only • Consider additional manually implemented functional test cases Metrics: • Sensitivity: proportion of vulnerabilities identified • Specificity: proportion of inputs not leading to false alarms
- 19. University of Ottawa | University of Luxembourg University of Ottawa | University of Luxembourg 19 Nazanin Bayati - Metamorphic Testing for Web System Security 13 September 2023 MST-wi – Is MST-wi effective? • The high specificity indicates that only a negligible fraction of follow-up inputs leads to false alarms • Since sensitivity reflects the fault detection rate (i.e., the proportion of vulnerabilities discovered), we conclude that our approach is highly effective • We can discover more than 60% of vulnerabilities in a completely automated manner, using only the crawler • And up to 85% using both crawler and manual inputs
- 20. University of Ottawa | University of Luxembourg University of Ottawa | University of Luxembourg 20 Nazanin Bayati - Metamorphic Testing for Web System Security 13 September 2023 https://github.com/MetamorphicSecurityTesting/MST
- 21. University of Ottawa | University of Luxembourg University of Ottawa | University of Luxembourg 21 Nazanin Bayati - Metamorphic Testing for Web System Security 13 September 2023 Metamorphic Testing for Web System Security Presented by: Nazanin Bayati 13 September 2023 N. Bayati Chaleshtari, F. Pastore, A. Goknil, and L. Briand, "Metamorphic Testing for Web System Security", IEEE Transactions on Software Engineering, 2023, https://ieeexplore.ieee.org/document/10089522 [email protected] University of Ottawa University of Luxembourg
- 22. University of Ottawa | University of Luxembourg University of Ottawa | University of Luxembourg 23 Nazanin Bayati - Metamorphic Testing for Web System Security 13 September 2023 MST-wi – How does MST-wi compare to state-of-the-art SAST and DAST tools? Security Design Principle Weaknesses Addresses by Weaknesses Addressed by MST but not by Weaknesses bot addresses by MST but addresses by MST Zap DA2 Sonar SA2 Zap DA2 Sonar SA2 Zap DA2 Sonar SA2 Audit 1 0 0 0 3 1 1 1 0 0 0 0 2 Authenticate Actors 12 0 2 1 9 12 11 11 7 0 1 0 4 Authorize Actors 34 2 0 1 13 32 34 34 25 0 0 1 4 Cross Cutting 3 0 0 2 0 3 3 2 3 0 0 1 0 Encrypt Data 8 2 5 8 10 8 8 7 4 2 5 7 6 Identify Actors 3 1 1 1 7 3 3 3 1 1 1 1 5 Limit Access 3 0 1 1 5 3 3 2 0 0 1 0 2 Limit Exposure 0 1 0 0 1 0 0 0 0 1 0 0 1 Lock Computer 0 0 0 0 0 0 0 0 0 0 0 0 0 Manage User Session 4 0 0 0 2 4 4 4 2 0 0 0 0 Validate Inputs 31 10 7 2 14 24 25 30 19 3 1 1 2 Verify Message Integrity 2 1 0 0 3 2 2 2 1 1 0 0 2 Total 101 17 16 16 67 92 94 96 62 8 9 11 28 56 MST can detect 56 weaknesses that any other approach cannot address.
- 23. University of Ottawa | University of Luxembourg University of Ottawa | University of Luxembourg 24 Nazanin Bayati - Metamorphic Testing for Web System Security 13 September 2023 MST-wi – How does MST-wi compare to state-of-the-art SAST and DAST tools? Security Design Principle Weaknesses Addresses by Weaknesses Addressed by MST but not by Weaknesses not addresses by MST but addresses by MST Zap DA2 Sonar SA2 Zap DA2 Sonar SA2 Zap DA2 Sonar SA2 Audit 1 0 0 0 3 1 1 1 0 0 0 0 2 Authenticate Actors 12 0 2 1 9 12 11 11 7 0 1 0 4 Authorize Actors 34 2 0 1 13 32 34 34 25 0 0 1 4 Cross Cutting 3 0 0 2 0 3 3 2 3 0 0 1 0 Encrypt Data 8 2 5 8 10 8 8 7 4 2 5 7 6 Identify Actors 3 1 1 1 7 3 3 3 1 1 1 1 5 Limit Access 3 0 1 1 5 3 3 2 0 0 1 0 2 Limit Exposure 0 1 0 0 1 0 0 0 0 1 0 0 1 Lock Computer 0 0 0 0 0 0 0 0 0 0 0 0 0 Manage User Session 4 0 0 0 2 4 4 4 2 0 0 0 0 Validate Inputs 31 10 7 2 14 24 25 30 19 3 1 1 2 Verify Message Integrity 2 1 0 0 3 2 2 2 1 1 0 0 2 Total 101 17 16 16 67 92 94 96 62 8 9 11 28 39 Combined together, DAST/SAST approaches can address only 39 weaknesses that MST-wi does not address.
- 24. University of Ottawa | University of Luxembourg University of Ottawa | University of Luxembourg 25 Nazanin Bayati - Metamorphic Testing for Web System Security 13 September 2023 MST-wi – How does MST-wi compare to state-of-the-art SAST and DAST tools? Security Design Principle Weaknesses Addresses by Weaknesses Addressed by MST but not by Weaknesses not addresses by MST but addresses by MST Zap DA2 Sonar SA2 Zap DA2 Sonar SA2 Zap DA2 Sonar SA2 Audit 1 0 0 0 3 1 1 1 0 0 0 0 2 Authenticate Actors 12 0 2 1 9 12 11 11 7 0 1 0 4 Authorize Actors 34 2 0 1 13 32 34 34 25 0 0 1 4 Cross Cutting 3 0 0 2 0 3 3 2 3 0 0 1 0 Encrypt Data 8 2 5 8 10 8 8 7 4 2 5 7 6 Identify Actors 3 1 1 1 7 3 3 3 1 1 1 1 5 Limit Access 3 0 1 1 5 3 3 2 0 0 1 0 2 Limit Exposure 0 1 0 0 1 0 0 0 0 1 0 0 1 Lock Computer 0 0 0 0 0 0 0 0 0 0 0 0 0 Manage User Session 4 0 0 0 2 4 4 4 2 0 0 0 0 Validate Inputs 31 10 7 2 14 24 25 30 19 3 1 1 2 Verify Message Integrity 2 1 0 0 3 2 2 2 1 1 0 0 2 Total 101 17 16 16 67 92 94 96 62 8 9 11 28 39 Combined together, DAST/SAST approaches can address only 39 weaknesses that MST-wi does not address. • The weaknesses that MST-wi cannot address are mostly those (i) that can only be discovered using program analysis, (ii) that are not related to user-system interactions, or (iii) that concern non-Web-based systems.
- 25. University of Ottawa | University of Luxembourg University of Ottawa | University of Luxembourg 26 Nazanin Bayati - Metamorphic Testing for Web System Security 13 September 2023 MST-wi – How does MST-wi compare to state-of-the-art SAST and DAST tools? Security Design Principle Weaknesses Addresses by Weaknesses Addressed by MST but not by Weaknesses bot addresses by MST but addresses by MST Zap DA2 Sonar SA2 Zap DA2 Sonar SA2 Zap DA2 Sonar SA2 Audit 1 0 0 0 3 1 1 1 0 0 0 0 2 Authenticate Actors 12 0 2 1 9 12 11 11 7 0 1 0 4 Authorize Actors 34 2 0 1 13 32 34 34 25 0 0 1 4 Cross Cutting 3 0 0 2 0 3 3 2 3 0 0 1 0 Encrypt Data 8 2 5 8 10 8 8 7 4 2 5 7 6 Identify Actors 3 1 1 1 7 3 3 3 1 1 1 1 5 Limit Access 3 0 1 1 5 3 3 2 0 0 1 0 2 Limit Exposure 0 1 0 0 1 0 0 0 0 1 0 0 1 Lock Computer 0 0 0 0 0 0 0 0 0 0 0 0 0 Manage User Session 4 0 0 0 2 4 4 4 2 0 0 0 0 Validate Inputs 31 10 7 2 14 24 25 30 19 3 1 1 2 Verify Message Integrity 2 1 0 0 3 2 2 2 1 1 0 0 2 Total 101 17 16 16 67 92 94 96 62 8 9 11 28 Combining MST-wi with SA2 seems to be a particularly effective combination as it enables detecting 129 weaknesses (i.e., 101 + 28), which is 92% of the 140 weaknesses that can be detected by any approach.
- 26. University of Ottawa | University of Luxembourg University of Ottawa | University of Luxembourg 29 Nazanin Bayati - Metamorphic Testing for Web System Security 13 September 2023 MST-wi: Metamorphic Security Testing for Web Interfaces Web System Execute the Data Collection Framework List of Predefined Metamorphic Relations Select and Specify the MRs Execute the Metamorphic Testing Framework Test results Transform MRs to Java List of MRs Executable MRs S(x,y) Source Inputs
- 27. University of Ottawa | University of Luxembourg University of Ottawa | University of Luxembourg 30 Nazanin Bayati - Metamorphic Testing for Web System Security 13 September 2023 MST-wi – Is MST-wi effective? • The high specificity indicates that only a negligible fraction of follow-up inputs leads to false alarms • Since sensitivity reflects the fault detection rate (i.e., the proportion of vulnerabilities discovered), we conclude that our approach is highly effective • We can discover more than 60% of vulnerabilities in a completely automated manner, using only the crawler • And up to 85% using both crawler and manual inputs