AppSec Survey 2.0 Fine-Tuning an AppSec Training Program Based on Data
2 likes1,829 views
Measuring the effectiveness of any security activity is widely discussed – security leaders debate the topic with a religious fervor rivaling that of any other hot button issue. Virtually every organization has some sort of application security training effort, but data on training effectiveness remains scarce. Last year our research team delivered the first-ever survey that captured developer awareness of secure coding concepts and the impact of formal application security training on a developer’s ability to write secure code. We learned that most software developer were aware of certain application security concepts, yet when asked how to write more secure code, they faired poorly. This year’s 600-developer survey provides more quantitative data on what software developers understand about application security, both concepts and practices. It dives most deeply into awareness of defensive coding practices, which most developers largely did not grasp in the 2013 survey. It also is separates respondents by roles, so we can better understand how architects, developers, and QA staff grasp key application security concepts and put them to work. It better captures how software developers learn in general, so one can tailor any security training effort to how software developers, in practice, actually learn. This information will provide data to application security managers responsible for corporate security training that should allow them them to make more fact-based decisions about security training.
Ad
More Related Content
What's hot (20)
Viewers also liked (20)
Ad
Similar to AppSec Survey 2.0 Fine-Tuning an AppSec Training Program Based on Data (20)
Ad
More from Denim Group (20)
Recently uploaded (20)
AppSec Survey 2.0 Fine-Tuning an AppSec Training Program Based on Data
- 1. AppSec USA 2014 Denver, Colorado AppSec Survey 2.0: Fine-Tuning an AppSec Training Program Based on Data John B. Dickson, CISSP @johnbdickson September 18, 2014
- 2. Introduction John B. Dickson, CISSP • Application Security Enthusiast • Ex-AF Guy & ISSA Distinguished Fellow • Serial Entrepreneur & MBA Type • Dad
- 3. When Not Thinking about AppSec… I am Snake Hunting on a Ranch in South Texas
- 4. Snake Hunting Essentials Cooler Hat Cool Hat Snake Guards Common Gardening Tools Guy who has a machete and who is actually good at “catching” snakes Machete OWASP AppSec 2011 t-‐shirt © Copyright 2014 Denim Group - All Rights Reserved
- 5. • Background • Premise • AppSec Study 1.0 Results – What We Learned • Approach and Survey ParKcipants • Key Results • What We Can Put To Work • Conclusions and QuesKons & Answers Overview
- 6. AppSec Study 1.0 Results • Things we Knew Last Year • Key Findings of Last Year’s Study • AddiKonal Stuff We Learned Along the Way • Development training is hard • Results are rarely measured for ROI • Training is typically part of any AppSec program
- 7. AppSec Study 1.0 Results • Things we Knew Last Year • Key Findings of Last Year’s Study • AddiKonal Stuff We Learned Long the Way • 25% retenKon aXer training • QA did worse than architects and soXware developers • Respondents answered basic awareness quesKons but not coding pracKces
- 8. • Things we Knew Last Year • Key Findings of Last Year’s Study • AddiConal Stuff We Learned Long the Way • SoXware developers learn differently than companies teach • IncenKves ma[er • Surveys are hard! AppSec Study 1.0 Results
- 9. Overview of 2014 “2.0” Study • 600 respondents • Represents mulKple industries • Asked the same applicaKon security quesKons as 2013 survey • Expanded to include training method quesKons • No “before” and “aXer” analysis • No classroom training opportuniKes • Used more social media • Data collecKon ongoing
- 10. Approach and Survey Participants Sample QuesCons QuesKons that tested basic knowledge of applicaKon security: • ApplicaKon security is best defined as… • Threat Modeling is… • Input ValidaKon is…
- 11. Approach and Survey Participants Sample QuesCons QuesKons that tested understanding of defensive coding: • Marking a cookie as “secure” will… • Which of the following will help protect against XSS… • Which of the following is NOT an example of good session policy…
- 12. Approach and Survey Participants Delivery Means • Direct Delivery of Customized Links via E-‐mail • Survey Monkey paid • Social Media – Facebook – Linkedin Targets • SoXware Developers • Architects • Quality Assurance
- 13. Demographic Questions Asked • What is your primary job funcKon? • What is your company's size? • How many years of soXware development experience do you have? • How much previous applicaKon security training have you received?
- 14. 2014 Study Demographics How many years of soMware development experience do you Less than a Year 18% 1-‐2 Years 9% 2-‐4 Years 10% 4-‐7 Years 13% More than 12 7-‐12 Years 16% Years 34% have?
- 15. 2014 Study Demographics What is your primary job Other 35% Quality Assurance 6% SoXware Developer 53% funcCon? Architect 6%
- 16. 2014 Study Demographics What is your company size? 8% 8% 29% 8% 10% 37% 1-‐24 Employees 25-‐99 Employees 100-‐499 Employees 500-‐2499 Employees 2500-‐9999 Employees 10,000 or more Employees
- 17. 2014 Study Demographics How much previous applicaCon security training experience have None 31% Less than a Day 19% More than 3 At least 1 day, but less than 2 days 17% At least 2 days, but less than 3 days 8% days 25% you received?
- 18. Key Survey Results • Data shows soXware developers posiKvely answer quesKons about applicaKon security 56% of the Kme • 2013 Denim Group study results: 58% • 2014 Aspect Study: 60%
- 19. Change Implementation Did your organizaCon implement any SDLC or process improvement steps to formalize concepts learned in training? Yes 33% No 25% I don't know 42%
- 20. Types of Training Received Types of Training Received 0 50 100 150 200 250 Other Wri[en Materials 1-‐on-‐1 Coaching Webinars or Videos Websites Crowdsourcing Sites Developer E-‐mail Lists or RSS feeds Social Learning Plaqorms Social Media e-‐Learning, CBT Instructor-‐Led PresentaKons
- 21. E-Learning & Instructor-Led Training Types of Training Received 0 100 200 300 Other Wri[en Materials 1-‐on-‐1 Coaching Webinars or Videos Websites Crowdsourcing Sites E-‐Learning & Instructor-‐led Training are SKll the Primary ApplicaKon Security Training Approach Developer E-‐mail Lists or RSS feeds Social Learning Plaqorms Social Media e-‐Learning, CBT Instructor-‐Led PresentaKons
- 22. Perceived Effectiveness of Training 0 50 100 150 200 250 300 350 400 450 500 Wri[en Materials 1-‐on-‐1 Coaching Webinars or Videos Websites Crowdsourcing Sites Developer E-‐mail Lists or RSS feeds Social Learning Plaqorms Social Media e-‐Learning, CBT Instructor-‐Led PresentaKons 1: Not EffecKve 2: Somewhat EffecKve 3: Very EffecKve
- 23. Question Types Respondents Fared Far Worse on QuesKons Involving Secure Coding PracKces versus ApplicaKon Security Awareness QuesKons 41% 59% 0% 10% 20% 30% 40% 50% 60% 70% Awareness QuesKons PrescripKve QuesKons % of QuesKons Answered Correctly
- 24. Pass Rate by Job Function Quality Assurance respondents Fared 50% worse than soXware developers and architects Average Pass Rate 25% 20% 15% 10% 5% 0% Other SoXware Developer Quality Assurance Architect 70% or more quesKons answered correctly
- 25. Pass Rate by Previous Training The Pass Rate More Than Doubled for Respondents Who Had More Than Three Days ApplicaKon Security Training Average Pass Rate 30% 25% 20% 15% 10% 5% 0% Less than a Day or None At least 1 day, but less than 3 days More than 3 days 70% or more correct
- 26. Pass Rate by Job Function: Security Respondents that worked for security organizaKons or vendors DID fare well compared to other respondents Average Pass Rate 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Security-‐Related Everyone Else 70% or more quesKons answered correctly
- 27. What we Can Put to Work • Refresher training is criCcal • Even with 3+ days of appsec training, most respondents did not have a “passing” grade of 70% • Like any other training topic, leX unreinforced, what learned will be forgo[en over Kme • ParKcularly given the lack of SDLC changes • Likely an area for addiKonal study for 2015 appsec training study
- 28. What we Can Put to Work • Training without SDLC changes likely will produce the same results • 33% of the respondents said their organizaKon implemented some security SDLC improvements • 67% either answered “no” or “don’t know” • OrganizaKons cannot rely exclusively on developers retenKon and iniKaKve to produce long-‐term decline in applicaKon vulnerabiliKes
- 29. What we Can Put to Work • Augment QA with Focused AppSec Training • QA has consistently responded poorly relaKve to developers and architects • Many organizaKon put their most junior developers in QA to start • QA is where appsec “lives” in many organizaKons • OrganizaKons might considering “doubling down” on appsec training for QA staff to compensate for this fact
- 30. What we Can Put to Work • IncenCves Ma`er When Working with Developers • We used incenKves throughout the study to collect responses -‐ #Success! • SoXware developers have infinite reasons to ignore engagement by the AppSec team • Rewards help nudge soXware developers
- 31. What we Can Put to Work • Training programs must be tailored to be effecCve • Formal programs like classroom training and e-‐ Learning are sKll the bread and bu[er of appsec training programs • ConsumpKon rates of e-‐Learning sKll abysmal without incenKves or internal markeKng • Add newer ways of learning to reinforce certain key points and to serve AppSec corner cases • Leverage current events to reinforce other key points
- 32. Conclusions • Data shows soXware developers posiKvely answer quesKons about applicaKon security 56% of the Kme • Data-‐driven applicaKon security programs will likely be more successful and chart improvement • SophisKcated security managers use incenKves and tailor programs to improve appsec IQ
- 33. Questions and Answers White Paper? MenCon it on Twi`er John B. Dickson, CISSP @johnbdickson #appsecstudy