The Four(ish) Appsec Metrics You Can’t Ignore

Editor's Notes

• #3: Sometimes it seems like application security programs are a never ending chasm. Why do we need to measure? For several important reasons— Sometimes you need to communicate to your sponsors what you’re doing with the money they provided for the program. Sometimes you need a way to communicate with your development teams that is anchored in something more than just encouragement. Sometimes you need a tool to show yourself how much progress you’re making.
• #4: Sometimes it seems like application security programs are a never ending chasm. Why do we need to measure? For several important reasons— Sometimes you need to communicate to your sponsors what you’re doing with the money they provided for the program. Sometimes you need a way to communicate with your development teams that is anchored in something more than just encouragement. Sometimes you need a tool to show yourself how much progress you’re making.
• #5: Don’t forget – it’s a hostile environment out there. Applications have been a top vector for data breaches over the last five years (see the Verizon Data Breach Reports) because they’re not coded with security in mind. The software industry’s shift to composing applications via pre-built—some would say “pre-0wned”—components has made it more challenging for security teams by introducing risk via the software supply chain. So application security is important, but how do you show progress?
• #6: Don’t forget – it’s a hostile environment out there. Applications have been a top vector for data breaches over the last five years (see the Verizon Data Breach Reports) because they’re not coded with security in mind. The software industry’s shift to composing applications via pre-built—some would say “pre-0wned”—components has made it more challenging for security teams by introducing risk via the software supply chain. So application security is important, but how do you show progress?
• #7: We’re going to walk through four ways to look at your portfolio of applications and benchmark it against other organizations. Each metric has its strength but they’re definitely better together. Let’s get started…
• #9: I think of policy compliance as the bedrock measurement (pardon the pun) of AppSec. You have to have some way of measuring the quality of applications; it should be aligned with the needs of the business. A lot of your program measurements are going to be anchored in how well your portfolio does against a policy. But what sort of pass rate should you expect? There’s the problem…
• #10: Let’s look at the industry perspective for a second—using a kind-of-generic policy, the OWASP Top 10, to look at how your peers are doing. Not well. (Highlight relatively high pass rate in FinSvc which is still bad news – more than 50% fail)
• #11: OK, so now that we understand how bad the problem is, how do we fix it? One way is to understand the types of issues we’re facing.
• #12: When vulnerabilities are all around you might feel like your world is on fire. Let’s try to get our arms around how common some of these fatal flaws really are.
• #13: It’s important to note that not every application is subject to the same risks. For instance, SQL Injection, a leading cause of data loss, is only present in about 30% of applications (40% if you’re in the government). Note though that crypto is found in 45 to 80% of applications depending on industry—a problem if your customers or regulators require you to protect sensitive data.
• #14: Okay, so we know how good (or bad) our applications are, and we know what the nature of the vulnerabilities are. Now what?
• #15: You know that guy, the one who always insists that the hole you’re in isn’t as deep as you think it is… It turns out that’s true of AppSec. There are a lot of people out there making their applications safer, never accepting “no” for an answer. And it turns out that tracking the flaws fixed can be powerfully motivational.
• #16: You know that guy, the one who always insists that the hole you’re in isn’t as deep as you think it is… It turns out that’s true of AppSec. There are a lot of people out there making their applications safer, never accepting “no” for an answer. And it turns out that tracking the flaws fixed can be powerfully motivational.
• #17: Some industries are making huge progress in fixing the vulnerabilities they’ve found. In fact across all Veracode’s customers, they fixed 3 out of every four flaws found by automated scans last year.
• #18: So a strategy for reducing risk is “real-time” developer education, enabling developers to go faster.
• #19: Policy, top flaw prevalence, fix rate. What’s our fourth metric?