Weaponizing Your DevOps Pipeline

Weaponizing Your DevOps Pipeline
OWASP MSP
Thursday, July 19th 2018
Eric Johnson (@emjohn20)

Puma Security
• Principal Security
Engineer
• Modern static code
analysis
• DevSecOps automation
• Secure Development
Lifecycle
SANS Institute
• Certified Instructor
DEV541: Secure Coding in Java
DEV534: Secure DevOps
• Course Author
DEV531: Mobile App Security
Essentials
DEV540: Secure DevOps &
Cloud Application Security
DEV544: Secure Coding in .NET
Eric Johnson, CISSP, AWS CD, GSSP, GWAPT
©2018 – Puma Security, LLC

Roadmap
• The DevOps Problem
• DevSecOps Program
Pre-Commit
Commit
Acceptance
Production
Operations
• Conclusion

State of DevOps
State of DevOps report (2017) indicates high
performing organizations:
• Deploy 46x more frequently
• Have 440x shorter lead times
• Recover from failures 96x faster
• Spend 50% less time remediating security
issues

The DevOps toolchain enabling the rapid
delivery cycles (not inclusive):
The DevOps Toolchain
Git Jenkins Docker Artifactory
Puppet / Chef Terraform Kubernetes Packer

• External vendor performing
annual assessments
• Internal security team receives
1,000 page PDF reports
• Internal security team
manually running scanners,
fuzzers, etc.
State of Traditional Security
img:https://paperlesschase.com/wp-content/uploads/2013/08/Tired-clerk-with-paper-on-desk.jpg

• Published
October 2016
• Release
frequency up 30x
• 42% indicate silos
still exist between
Sec and DevOps
HPE | AppSec & DevOps Survey
20%
38%
25%
17%
Security in DevOps
SecDevOps Gated Reviews Network Defenses Nothing

• Security is not invited to the DevOps party
• Internal security team does not have
development background
• Frequent deployments invalidate assessment
results
• Missing a huge opportunity for security in the
pipeline
The Problem

Why The Cold Shoulder?
"DevOps is an excuse for developers to have
global access to production. No way."
- The dictator CISO
"Perfect, I get to wire up crappy security
scanners and break the build."
- The security jerk
"We cannot use continuous delivery and
remain PCI compliant. "
- The uninformed compliance manager

What is DevSecOps
DevSecOps / SecDevOps / DevOpsSec is about
breaking down walls between security and:
• Development
• Operations
• Business
"In DevSecOps, security is a first-class
problem and the security team is a
first-class citizen."
- Jim Bird, CTO, SANS Analyst & DEV540 co-author

Applying security to Wills, Edwards, & Humble's CALMS:
• Culture - No security jerks (Etsy), turning "no" into "yes"
• Automation - Rely on security tools for efficiency + repeatability
• Lean - Apply lean engineering practices to risk assessments /
code reviews
• Measurement - Use security data to drive decisions, improve,
and respond in real time (or near real time)
• Sharing - Sharing threat intel, secure frameworks, and
postmortems across the organization
Keeping CALM & DevSecOps On

DevSecOps Phases
• DevSecOps cycles through 5
key phases
• SANS DevSecOps Toolchain
lists several OSS tools for
each phase
– Written by Ben Allen, Jim Bird,
Eric Johnson, & Frank Kim
• https://sans.org/u/zAi
PRE-COMMIT COMMIT (CI) ACCEPTANCE PRODUCTION OPERATIONS
sans.org/u/zAi

Breaking down the security controls in each
DevSecOps phase:
DevSecOps Security Controls
PRE-COMMIT COMMIT (CI) ACCEPTANCE PRODUCTION
IDE SECURITY
PLUGINS
PRE-COMMIT
HOOKS
PEER CODE
REVIEWS
STATIC CODE
ANALYSIS
SECURITY UNIT
TESTS
CONTAINER
SECURITY
INFRASTRUCTURE
SCANNING
DYNAMIC
SECURITY TESTS
SECURITY SMOKE
TESTS
SECURITY
CONFIGURATION
SECRETS
MANAGEMENT
THREAT
MODELING
DEPENDENCY
MANAGEMENT
SECURITY
ACCEPTANCE TESTS
SERVER
HARDENING
OPERATIONS
BLAMELESS
POSTMORTEMS
CONTINUOUS
MONITORING
PENETRATION
TESTING
THREAT
INTELLIGENCE
CLOUD
INFRASTRUCTURE

Applying security controls before code is written
and committed:
DevSecOps Phases | Pre-Commit
IDE SECURITY
PLUGINS
PRE-COMMIT
HOOKS
PEER CODE
REVIEWS
STATIC CODE
ANALYSIS
SECURITY UNIT
TESTS
CONTAINER
SECURITY
INFRASTRUCTURE
SCANNING
DYNAMIC
SECURITY TESTS
SECURITY SMOKE
TESTS
SECURITY
CONFIGURATION
SECRETS
MANAGEMENT
THREAT
MODELING
DEPENDENCY
MANAGEMENT
SECURITY
ACCEPTANCE TESTS
SERVER
HARDENING
OPERATIONS
BLAMELESS
POSTMORTEMS
CONTINUOUS
MONITORING
PENETRATION
TESTING
THREAT
INTELLIGENCE
CLOUD
INFRASTRUCTURE

Threat modeling must apply lean
engineering principles:
• Lightweight and incremental review
• The source code is the design
• Focus on data classification, entry points, high risk code,
and writing security stories / abuse cases
• Categorize the risk level (high risk, paved road, control
gates)
Pre-Commit| Threat Modeling
PRE-COMMIT
THREAT MODELING

Weaponizing the toolchain:
• Raindance
– https://github.com/devsecops/raindance
• Mozilla's Rapid Risk Assessment (RRA)
– https://infosec.mozilla.org/guidelines/risk/rapid_risk_assessme
nt.html
• OWASP Threat Dragon
– https://www.owasp.org/index.php/OWASP_Threat_Dragon
Pre-Commit| Threat Modeling Tools
PRE-COMMIT
THREAT MODELING

High risk code may perform any of following
functionality (not inclusive):
• Infrastructure Code
• Pipeline definitions
• Authentication
• Access control
• Output encoding
• Input validation
• Automated security /
compliance tests
• High risk business logic
• Data entitlement checks
• Handling confidential data
• Cryptography
Pre-Commit | High Risk Code Examples

Mozilla's rapid risk assessment guidance and Google
Doc provide a blueprint for 30 minute RRAs:
Pre-Commit| Threat Modeling Example

Opportunity to identify vulnerabilities in
infrastructure and application code as
code is written or saved to disk:
Pre-Commit | IDE Security Plugins
IDE SECURITY
PLUGINS
PRE-
COMMIT
• Security becomes part of the engineering
workflow
• Shiftings as far left as possible in the kill chain
• Must have low false positive rates (important)

Pre-Commit | IDE Security Tools
IDE SECURITY
PLUGINS
PRE-
COMMIT
• FindSecurityBugs (Java)
• Puma Scan (C#)
• Sonar Lint (Java, C#, JavaScript)
• DevSkim (C#, JavaScript)

Puma Scan identifying a JSON deserialization vulnerability:
Pre-Commit | IDE Security Example

Run security checkers before committing
code to git:
Pre-Commit | Git Hooks
PRE-
COMMIT
• Invoke additional CLI scans / security checks before
code reaches continuous integration
• Use for secrets management, keys, access keys, etc.
• Important to note these client-side protections can be
disabled by engineers
PRE-COMMIT
HOOKS

Pre-Commit | Git Hook Tools
PRE-
COMMIT
• AWS Labs git-secrets
– https://github.com/awslabs/git-secrets
• Talisman
– https://github.com/thoughtworks/talisman
• Auth0 repo-supervisor
– https://github.com/auth0/repo-supervisor
• Yelp Pre-Commit Framework
– https://pre-commit.com/
PRE-COMMIT
HOOKS

AWS git-secrets blocking a commit that contains an
access key and secret key id:
Pre-Commit | Git Hook Example
$ git commit -m "testing git-secrets"
Web/PumaScan.Licensing.Web/appsettings.json:5:
"AccessKey": "AKIAJNQ7C2FCRR6B4VWA",
Web/PumaScan.Licensing.Web/appsettings.json:6:
"SecretKey": "ry8F6PlPTBP4bFGqZ0IzvZ71Oh2gkgZvFK/CZecw"
[ERROR] Matched one or more prohibited patterns
1
2
3
4
5
6
7

Peer code reviews are mandatory in
disciplined DevSecOps organizations:
Pre-Commit | Peer Reviews
PRE-
COMMIT
• Allows engineers to discover hard-coded secrets, logic
flaws in high risk code, backdoors
• Compensating control for separation of duties in
continuous deployment
• Relies on the reviewer's application security skillset
PEER CODE
REVIEWS

Pre-Commit | Peer Review Toolchain
PRE-
COMMIT
• GitHub Pull Request
• GitLab Merge Request
• Bitbucket Pull Request
• Gerrit (Google)
• Review Board
– https://github.com/reviewboard/reviewboard
PEER CODE
REVIEWS

GitHub pull request requiring peer review approval:
Pre-Commit | Peer Review Example

Applying automated, fast, accurate security
controls in the CI pipeline:
DevSecOps Phases | Commit
IDE SECURITY
PLUGINS
PRE-COMMIT
HOOKS
PEER CODE
REVIEWS
STATIC CODE
ANALYSIS
SECURITY UNIT
TESTS
CONTAINER
SECURITY
INFRASTRUCTURE
SCANNING
DYNAMIC
SECURITY TESTS
SECURITY SMOKE
TESTS
SECURITY
CONFIGURATION
SECRETS
MANAGEMENT
THREAT
MODELING
DEPENDENCY
MANAGEMENT
SECURITY
ACCEPTANCE TESTS
SERVER
HARDENING
OPERATIONS
BLAMELESS
POSTMORTEMS
CONTINUOUS
MONITORING
PENETRATION
TESTING
THREAT
INTELLIGENCE
CLOUD
INFRASTRUCTURE

Limited opportunity for static analysis in CI & CD
pipelines:
Commit | Static Code Analysis
STATIC CODE
ANALYSIS
COMMIT
• Speed matters (< 5 minutes)
• High accuracy rules
• Low false positive rates
• Disable rules that do not provide value to
engineers

Commit | Static Code Analysis Tools
STATIC CODE
ANALYSIS
COMMIT
• Brakeman (Ruby)
• ESLint (NodeJS)
• Puma Scan (C#)
• FindSecurityBugs (Java)
• Puppet Lint Security
• And many, many commercial offerings….

Puma Scan failing a build in a Jenkins CI pipeline:
Commit | Static Code Analysis Example

Capturing and reporting vulnerability data in a
Jenkins CI pipeline:
Commit | Static Code Analysis Example

Built on top of standard unit and integration tests to
enforce security requirements:
Commit | Security Unit Tests
SECURITY UNIT
TESTS
COMMIT
• Leverage abuse cases and evil user stories
from rapid risk assessment
• Focus on high risk code and business logic
flaws
• Fast execution in the IDE / CI pipeline
• Can be used to enforce security requirements

Commit | Security Unit Test Tools
SECURITY UNIT
TESTS
COMMIT
• JUnit
• XUnit
• Mocha (NodeJS)
• RSpec

• Engineers often stay on the "happy path"
• Prove the code works under normal usage
Commit | Happy Path Unit Test Example
[Theory]
[InlineData("bob@app.com", "L1ttleB0bbyTable$", "1", HttpStatusCode.Found)]
public async Task License_DownloadTest(string username, string
password, string id, HttpStatusCode responseCode)
{
…
var request = new HttpRequestMessage(HttpMethod.Get, $"/download/{id}");
request.Headers.Add("Cookie", $"app-portal=${authCookie};");
var response = await _client.SendAsync(request);
Assert.Equal(responseCode, response.StatusCode);
}
1
2
3
4
5
6
7
8
9
10
11

Testing common SQL injection characters:
Commit | Validation Unit Text Example
[Theory]
[InlineData("bob@app.com", "L1ttleB0bbyTable$", "'", HttpStatusCode.NotFound)]
[InlineData("bob@app.com", "L1ttleB0bbyTable$", "*", HttpStatusCode.NotFound)]
[InlineData("bob@app.com", "L1ttleB0bbyTable$", ")", HttpStatusCode.NotFound)]
[InlineData("bob@app.com", "L1ttleB0bbyTable$", ",", HttpStatusCode.NotFound)]
[InlineData("bob@app.com", "L1ttleB0bbyTable$", ";", HttpStatusCode.NotFound)]
[InlineData("bob@app.com", "L1ttleB0bbyTable$", "#", HttpStatusCode.NotFound)]
[InlineData("bob@app.com", "L1ttleB0bbyTable$", "&", HttpStatusCode.NotFound)]
public async Task License_DownloadTest(string username, string
password, string id, HttpStatusCode responseCode)
{
…
var request = new HttpRequestMessage(HttpMethod.Get, $"/download/{id}");
request.Headers.Add("Cookie", $"app-portal=${authCookie};");
var response = await _client.SendAsync(request);
Assert.Equal(responseCode, response.StatusCode);
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

Verifying checksums on high risk code:
Commit | High Risk Code Unit Test Example
[Theory]
[InlineData("/Web/Controllers/AccountController.cs",
"2ffbf33b66ddb07616f882ceed0718826af298a7")]
[InlineData("/Shared/Services/Cryptography/Hash.cs",
"d51bfd137d37a7ed908737552568bcc5241f5021")]
[InlineData("/Shared/Services/Cryptography/Asymmetric.cs",
"fe83bf6f453698c5f78cab167bca14c72daf32c0")]
[InlineData("/Shared/Services/Cryptography/Symmetric.cs",
"ae951207f4fbdbe2d9661297f285dc99857f32d4")]
public void HighRiskCode_CheckSumTest(string file, string checksum)
{
bool match = checksum.Equals(Hash.GetChecksum(file));
if(!match) NotificaionService.RequestCodeReview(file);
Assert.True(match);
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

Containers present an entirely new attack surface
for engineering teams to deal with:
Commit | Container Security
COMMIT
• Managing container secrets
• Poisoned / untrusted images
• Hardening image operating system
• Docker daemon / API attack surface
• This topic alone is an hour + conversation
CONTAINER SECURITY

Commit | Container Security Tools
COMMIT
• Anchore
– https://anchore.com/opensource/
• Actuary
– https://github.com/diogomonica/actuary
• Clair
– https://github.com/coreos/clair
• Falco
– https://github.com/draios/falco
CONTAINER SECURITY

Anchore scan results via Jenkins CI:
Commit | Container Security Scan Example

Builds a bill of material from operation
and application dependencies
Commit | Dependency Management
COMMIT
• Scans manifests, templates, and libraries
• Identifies packages and libraries with known
vulnerabilities
• Suggests package version updates to remediate
vulnerabilities
DEPENDENCY
MANAGEMENT

Commit | Dependency Management Tools
COMMIT
• OWASP Dependency Check
• PHP Security Checker
• Retire.JS
• Node Security Project
DEPENDENCY
MANAGEMENT

Dependency check scan results via Jenkins CI:
Commit | Dependency Management Example

Applying security controls during delivery of
infrastructure or applications to acceptance:
DevSecOps Phases | Acceptance
IDE SECURITY
PLUGINS
PRE-COMMIT
HOOKS
PEER CODE
REVIEWS
STATIC CODE
ANALYSIS
SECURITY UNIT
TESTS
CONTAINER
SECURITY
INFRASTRUCTURE
SCANNING
DYNAMIC
SECURITY TESTS
CLOUD
INFRASTRUCTURE
SECURITY SMOKE
TESTS
SECURITY
CONFIGURATION
SECRETS
MANAGEMENT
THREAT
MODELING
DEPENDENCY
MANAGEMENT
SECURITY
ACCEPTANCE TESTS
SERVER
HARDENING
OPERATIONS
BLAMELESS
POSTMORTEMS
CONTINUOUS
MONITORING
PENETRATION
TESTING
THREAT
INTELLIGENCE

Applying security controls during deployment of
infrastructure or application to production:
DevSecOps Phases | Production
IDE SECURITY
PLUGINS
PRE-COMMIT
HOOKS
PEER CODE
REVIEWS
STATIC CODE
ANALYSIS
SECURITY UNIT
TESTS
CONTAINER
SECURITY
INFRASTRUCTURE
SCANNING
DYNAMIC
SECURITY TESTS
CLOUD
INFRASTRUCTURE
SECURITY SMOKE
TESTS
SECURITY
CONFIGURATION
SECRETS
MANAGEMENT
THREAT
MODELING
DEPENDENCY
MANAGEMENT
SECURITY
ACCEPTANCE TESTS
SERVER
HARDENING
OPERATIONS
BLAMELESS
POSTMORTEMS
CONTINUOUS
MONITORING
PENETRATION
TESTING
THREAT
INTELLIGENCE

Continuous security monitoring, testing, and
compliance checks in production:
DevSecOps Phases | Operations
IDE SECURITY
PLUGINS
PRE-COMMIT
HOOKS
PEER CODE
REVIEWS
STATIC CODE
ANALYSIS
SECURITY UNIT
TESTS
CONTAINER
SECURITY
INFRASTRUCTURE
SCANNING
DYNAMIC
SECURITY TESTS
CLOUD
INFRASTRUCTURE
SECURITY SMOKE
TESTS
SECURITY
CONFIGURATION
SECRETS
MANAGEMENT
THREAT
MODELING
DEPENDENCY
MANAGEMENT
SECURITY
ACCEPTANCE TESTS
SERVER
HARDENING
OPERATIONS
BLAMELESS
POSTMORTEMS
CONTINUOUS
MONITORING
PENETRATION
TESTING
THREAT
INTELLIGENCE

• Open source security source code analyzers
• 50+ application security-specific rules
• Install guide, rule docs, source code:
https://www.pumascan.com/community
https://github.com/pumasecurity
@puma_scan
• Presenting Wednesday August 8th at Black Hat Arsenal
https://www.blackhat.com/us-18/arsenal/schedule/#puma-
scan-12003
Puma Scan | Black Hat Arsenal 2018

Questions?
Contact Info:
eric.johnson@pumascan.com
@emjohn20

Weaponizing Your DevOps Pipeline

Recommended

More Related Content

What's hot (20)

Similar to Weaponizing Your DevOps Pipeline (20)

More from Puma Security, LLC (10)

Recently uploaded (20)

Weaponizing Your DevOps Pipeline