AWS reInforce 2021: TDR202 - Lessons learned from the front lines of Incident Response

© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
A U G . 2 4 – 2 5 , 2 0 2 1 | H O U S T O N , T X

Lessons learned from the
front lines of incident response
Brian “BasementCat” Andrzejewski
T D R 2 0 2
Lead Operator, Customer Incident Response Team (CIRT)
AWS Professional Services

Who we are
Why we are here
Common causes for customer security events
Critical security patterns to reduce customer risks
Where to go next
Agenda

Who we are
A specialized AWS Customer Incident Response team that assists and
advises customers during their active security events on the customer’s
side of the AWS Shared Responsibility Model
Experienced team of
AWS Professional Services
and Solution Architects
in incident response
Assist in root cause
analysis of a customer’s
AWS service logs for their
active security event
Assist and advise
customers with active
triage & recovery of their
security event on AWS
Provide advise to
customers for long-term
recovery from their

Available escalations during an AWS customer’s
1. Validate AWS account ownership
to customer(s) impacted
2. Assist with triage & recovery with
customer and AWS teams
3. Investigate root cause(s) with
customer for their event
4. Provide recommendations for
next steps
AWS support case (all tiers)
AWS Account team
AWS Security
AWS Customer Incident Response Team
(CIRT)
External
Internal
AWS Customer Incident
Response team
Customer triage path

AWS shared responsibility model
Customers and AWS Partners
https://aws.amazon.com/compliance/shared-responsibility-model/
Security IN
the cloud
Managed by
customers
Security OF
the cloud
Managed by
AWS

The line varies. . .
Hardware/AWS global
infrastructure
Compute/storage/
database/network
Client-side data
encryption/integrity
Server-side encryption
Network traffic protection
OS, network, firewall
Configuration
Platform and application
management
Customer data
AWS
IAM
Customer
IAM
Infrastructure
services
Client-side data
encryption
Firewall configuration
configuration
management
Customer data
Hardware/AWS global
infrastructure
Compute/storage/
database/network
AWS
IAM
Customer
IAM
Container
services
Abstracted
services
Client-side data
encryption
configuration
management
Customer data
Hardware/AWS global
infrastructure
Compute/storage/
database/network
AWS
IAM
Customer
IAM
Server-side encryption
More
customizable
+
More customer
responsibility
Less customizable
+
Less customer
responsibility
+
More best
practices built in
https://aws.amazon.com/whitepapers/aws-security-best-practices/

Additional AWS customer responsibilities
All activities that occur under your account –
including unauthorized access (4.1)
Properly configuring and using an AWS service (4.3)
Keeping AWS root account email current for
notifications (13.10.a)
Taking appropriate action to secure, protect, and
back up your account and your content (4.3)
Not disclosing login credentials and access keys to
unauthorized third parties (4.4)
https://aws.amazon.com/agreement/

Why are we here?
challenges 24/7
best practices
seeking right
AWS skillsets and knowledge
share
excel

AWS incident response methodology
• Incident Response Phases
 Prepare and prevent
 Detect and assess impact
 Triage and recovery
 Investigate to root cause(s)
• Improve and iterate
 Develop people and technology
 Update playbooks and runbooks
 Simulate security events in environment
 Apply lessons learned and iterate to improve
https://docs.aws.amazon.com/whitepapers/latest/aws-security-incident-response-guide/

Common causes for
customer security events

Understanding the
risk equation
Threat actor:
A cat that wants to scratch you
Threat:
The cat’s paw reaching out to scratch you
Vulnerability:
Your inability to defend against the scratch
Risk:
The likelihood of being scratched
Acceptable risk:
Your willingness to be scratched by the cat

Gartner estimates that
most cloud security
failures will be IN the
cloud on the user side
Gartner “Is the Cloud Secure?”
https://www.gartner.com/smarterwithgartner/is-the-cloud-secure/

Common causes for customer security events
Insecure
AWS resource
configuration
Inaccurate
AWS account contact
information
Lack of continuous
vulnerability
management executed
Unintended disclosure
of security credentials
and secrets
Inadvertent response to
Amazon GuardDuty and
other detective controls
Unmanaged
application
software security

Critical security patterns
to reduce customer risks

Implementation goals and objectives
Scenario-driven guidance to common root causes of security events
• Grounded to real-world security events experienced by AWS customers
• Prescriptive guidance on how prevent and detect by root cause
Reduce customer’s security risks to their AWS accounts and its resources
• Applicable to all AWS customers and their existing architectures
• Core AWS services to start your security journey and iterate beyond
• Prioritized to critical security practices observed to prevent and detect

Bill of materials
Core AWS services AWS service tools
AWS Organizations Amazon GuardDuty
AWS Security Hub
AWS CloudTrail
AWS IAM Access
Analyzer and advisor
AWS IAM temporary
security credential
AWS Config
AWS Personal
Health Dashboard
AWS Well-Architected Tool Amazon CloudWatch
AWS Identity and Access
Management (IAM)
AWS Backup
Amazon VPC
Reachability Analyzer
AWS Secrets Manager
AWS IAM policy
simulator

Inaccurate AWS account information by customer
Prevent
• Ensure AWS account email uses both
 Corporate email domain
(i.e., example.com)
 Distribution email address
• AWS account root user
 Use IAM principals for day-to-day access
 Eliminate use of root access keys
 Use an MFA device for root console use
Detect
• Leverage CloudTrail and CloudWatch
events to detect AWS account changes
• Monitor your AWS notifications
 AWS account email
 AWS Health event alerts
• Use AWS Cost Anomaly Detection for
monitoring unusual AWS account costs
Affects AWS account holder’s ability to
• Act upon AWS-provided notifications that require timely resolution
• Provide account owner verification during AWS account recovery process
• Perform break-glass access for AWS account and during root user password reset

Insecure AWS resource configuration by customer
Prevent
• Enable public block for supported AWS services to restrict public access
• Configure backups and validate restores with AWS Backup for critical resources and data
• Use different IAM principals and roles to manage vs. operate AWS resources to reduce impacts
• Deploy AWS resources into private VPCs to reduce unintended access
• Implement AWS Organizations security control polices to restrict
 Modify and delete AWS resource changes to system admin roles
 AWS access key usage to restricted policy conditionals
Critical configuration to focus on
• AWS Foundational Security Best Practices with severity critical or high
• Public-facing assets for defense-in-depth and restricted network access
• Deny-then-allow authentication for AWS resources that contain sensitive data

Insecure AWS resource configuration by customer
Detect
• Enable Security Hub for all regions with AWS Foundational Security Best Practices to
detect common AWS resource misconfigurations
• Prioritize GuardDuty anomalous behavior findings for unexpected resources changes
• Leverage AWS Config for recording and building inventory
 AWS resources by name and service
 Individual AWS resource configurations
Critical configuration to focus on
• AWS Foundational Security Best Practices with severity critical or high
• Public-facing assets for defense-in-depth and restricted network access
• Deny-then-allow authentication for AWS resources that contain sensitive data

Unintended disclosure of security
credentials and secrets by customer
Prevent
• Disable and delete all AWS account root access keys
• Use temporary role-based access over static credentials and keys
• Require MFA for your most sensitive operations and privileged access
• Use Secrets Manager to vault and audit use of non-IAM credentials
• Build identity-based and resource-based policies for least-privilege access to reduce impact of
unintended access and disclosure
 Use explicit deny-then-allow policy conditions in identity and resource policies
 Tailor identity-based policies to use named actions to resource names or tagged resources
 Specify resource-based policies for explicit identity-based roles and/or identity
principal tags, and/or to restrict to specific VPC endpoints or source IP addresses

Example: Policy evaluation logic
https://amzn.to/3CLktQQ

Example: Resource-based policy restrictions
{
...
"Statement": [
{
"Sid": "VPCe",
"Action": "s3:*",
"Effect": "Deny",
"Resource": [
"arn:aws:s3:::DOC-EXAMPLE-BUCKET",
"arn:aws:s3:::DOC-EXAMPLE-BUCKET/*”
],
"Condition": {
"StringNotEquals": {
"aws:SourceVpce": [ "vpce-1111111” ]
}
},
”Principal": "*"
}
]
}
{
...
"Statement": [
{
"Sid": ”SourceIP",
"Action": "s3:*",
"Effect": "Deny",
"Resource": [
"arn:aws:s3:::DOC-EXAMPLE-BUCKET",
"arn:aws:s3:::DOC-EXAMPLE-BUCKET/*”
],
"Condition": {
"NotIpAddress": {
"aws:SourceIP": [ "11.11.11.11/32” ]
}
},
”Principal": "*"
}
]
}
VPC source restriction Source IP restriction
https://amzn.to/2VObNIA
S3 Bucket
Policy

Unintended disclosure of security
credentials and secrets by customer
Detect
• Monitor for identity behavioral changes recorded in AWS CloudTrail events:
 GuardDuty IAM findings
 CloudTrail Insights for unusual mutating events
• Continuously evaluate IAM principal usage to reduce over-privileged access impact
 IAM Credential Report to disable and remove unused IAM user and access keys
 IAM Access Advisor to refine IAM principal permissions using last-accessed information to AWS services
 IAM Access Analyzer to adjust IAM principal permissions through its past actions from CloudTrail trail and identify
resources shared with an external entity
 IAM Policy Simulator to test and simulate actions for their effective permissions of identity-based policies, permission
boundaries, SCPs, and resource-based policies
• Implement application security scanning for static credentials and secrets to reduce disclosure
• Monitor your AWS account email address for AWS notifications of compromised credentials
https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html

Example: Tools for least-access assessment
https://amzn.to/3CJZGwL
Use IAM access advisor
• Enabled per IAM principal by
default in IAM service
• Review which AWS services have
been used up to last 400 days
per IAM principal
Use IAM Access Analyzer to
generate least-access policy
• Requires CloudTrail trail is
enabled to Amazon S3
• Evaluates last 90 days of specific
IAM principal access from selected
CloudTrail trail S3 bucket
• Generates a suggested IAM policy
from evaluation to use

Inadvertent response to GuardDuty and other
detective controls by customer
Prevent and detect
• Tailor findings and detections to your threat models for criticality of data protection
https://cheatsheetseries.owasp.org/cheatsheets/Threat_Modeling_Cheat_Sheet.html
• Use AWS Security Hub to aggregate, organize, review, and prioritize findings from AWS and
AWS Partners for further response, such as
Enabling GuardDuty detections of malicious activity and unauthorized behaviors from
CloudTrail management & data events, Amazon VPC Flow Logs, and Route 53 DNS logs
Enabling and reviewing AWS Security Hub findings from security checks of
standards and controls (i.e. CIS, PCI DSS, AWS Foundational)
Impacts ability for actionable incident response
• Mean times to detect, respond, and recover from a security event
• Increases scope of resources to triage and recover from security event
• Raises risk of data exfiltration and/or destruction as mean times increase to detect & respond

Lack of continuous vulnerability management
executed by customer
Not maintaining software updates continuously – common sources of compromises
• Operating systems and their services
• Installed application software and their dependencies
• Continuous deployment and integration systems (CI/CD)
Exposing unmanaged systems and applications to public internet
• Open ports and applications with no inbound or outbound network restrictions
• Using default configuration of common applications and services
• Zero-day exploits against common applications
Prevent and detect
• Perform continuous vulnerability scanners against resources, source code, and network ports
Examples: Amazon Inspector, VPC Reachability Analyzer
• Implementing defense-in-depth approach to restrict discovery and exploitation

Unmanaged application software security by
customer
Source of compromise when unmanaged leads to additional security issues
• For in-house, open-source, and acquired software
• For application software dependencies to build, install, and operate
Prevent and detect
• Implementing OWASP Top 10 controls for secure coding practices
• Apply static and dynamic analysis tools for application software security practice
• Mitigate risk through defense-in-depth approach to application design, network, and
identity controls
• Use third-party endpoint security clients to protect process-to-network executions
• Red team “trust, then verify” application software implementations to validate defense-in-depth
security controls to protect, detect, and respond

Where to
go next
AWS account
Detect
Front-end application
Backend application
Public internet

Core references
1. Top 10 security items to improve in your AWS account
https://amzn.to/3AA1RkT
2. Security Pillar – AWS Well-Architected Framework
https://docs.aws.amazon.com/wellarchitected/latest/security-pillar/
3. AWS Security Reference Architecture (AWS SRA)
https://amzn.to/3jWjxk6
4. AWS Security Incident Response Guide
https://amzn.to/3xFXxP5
5. AWS Security Guides
https://docs.aws.amazon.com/security/

Call to action: Top 10
1. Ensure you have a defined cloud security strategy and incident response plan,
including people, processes, and technology for cloud
2. Use business email distribution lists for AWS account contact information to
respond to AWS notifications
3. Configure backups plans with AWS Backup for critical resources and data, and
periodically verify and their order and priority for system restores
4. Ensure enablement of GuardDuty, AWS Config, Security Hub, CloudTrail, and
service access and audit logs for detection of security event observables
5. Use AWS Foundational Security Best Practices to continuously assess risks for
critical and high severities for common AWS resource misconfigurations

Call to action: Top 10
6. Continuously assess for least-privileged access with IAM tools
7. Replace long-lived credentials with short-lived credentials to reduce risks of
security event impact and scope
8. Implement OWASP Top 10 – especially input validation and rate limits – for
applications within your code and with AWS services (e.g., AWS WAF)
9. Continuously patch to latest security patches for your OS, applications, and
dependencies
10. Routinely train and simulate for cloud security events to iterate and improve
Security is an iterative process, not a one-time project

Thank you!
Brian Andrzejewski
bcandrze@amazon.com

Please complete
the session survey

AWS reInforce 2021: TDR202 - Lessons learned from the front lines of Incident Response

More Related Content

What's hot (20)

Similar to AWS reInforce 2021: TDR202 - Lessons learned from the front lines of Incident Response (10)

Recently uploaded (20)

AWS reInforce 2021: TDR202 - Lessons learned from the front lines of Incident Response

Editor's Notes