![Mozilla SOPS - Diffs in Cleartext in Git
~ > cat ~/.gitattributes
*.enc diff=sopsdiffer
*.enc.yaml diff=sopsdiffer
*.enc.json diff=sopsdiffer
.gitattributes
~ > grep -A1 sops ~/.gitconfig
[diff "sopsdiffer"]
textconv = "sops -d"
.gitconfig
In my testing, this worked if
.sops.yaml is at the root directory
of the git project and regex expression
is based on the filename but not
directory name
dev/secret-values.enc.yaml →
dev-secret-values.enc.yaml](https://image.slidesharecdn.com/k8s-secretsmanagement-181121182055/85/Secrets-Management-and-Delivery-to-Kubernetes-Pods-20-320.jpg)
Secrets Management and Delivery to Kubernetes Pods
- 1. Secrets Management and Delivery to Kubernetes Pods Satish Devarapalli
- 2. $whoami Satish Devarapalli Cloud Platform Architect @devasat https://www.linkedin.com/in/satish-devarapalli-48a82a5/
- 3. Agenda ● Project Intro ● Secrets Delivery Pipeline View ● Mozilla SOPS (Secret OPerationS) ● Questions
- 4. Project Intro Product auth 𝞵 svc𝞵 svc ... Application Secrets ● Database Credentials ● AWS Credentials ● API Keys ● License Keys ● Encryption Keys CI/CD Secrets ● AWS Credentials ● Kube Config Files ● Encryption Keys
- 5. ● Manage secrets as code → check-in, build and deploy ○ Secrets should be encrypted before check-in ● Encryption keys management ○ Different encryption keys for environments ○ Centralized access → do not store keys locally ○ Audit usage ○ Limit key access to individual members based on environment ○ Grant and revoke access easily ● Jenkins → minimal configuration on Jenkins nodes ○ Do not store encryption keys on Jenkins master or nodes ○ Secrets should not be left in plain text format on build success or failure ○ Store encrypted kube config files and AWS credentials in Git ● Limit secrets access to microservices that use it Secrets Management Requirements
- 6. Agenda ● Project Intro ● Secrets Delivery Pipeline View ● Mozilla SOPS (Secret OPerationS) ● Questions
- 7. Secrets Delivery Pipeline AWS KMSdec enc Mozilla SOPS + Dev QA Prod AWS KMS decenc Mozilla SOPS + Jenkins job runs inside a container Helm install Helm install Helm install Helm chart versions: env-config-1.0.0-101-af837dh env-config-1.0.0-102-x7jwy62
- 8. Agenda ● Project Intro ● Secrets Delivery Pipeline View ● Mozilla SOPS (Secret OPerationS) ● Questions
- 9. AWS KMS - Envelope Encryption Picture source: https://docs.aws.amazon.com/encryption-sdk/latest/developer-guide/how-it-works.html
- 10. Mozilla Secret OPerationS (SOPS) ~ > export SOPS_KMS_ARN="arn:aws:kms:us-east-1:212121:key/3434-c5f1-4040 ~ > sops -e secret-values.dec.yaml > secret-values.enc.yaml Picture source: https://docs.aws.amazon.com/encryption-sdk/latest/developer-guide/how-it-works.html SOPS Generates 256-bit data key secret-values .dec.yaml SOPS_KMS_ARN secret-values .enc.yaml AWS Boundary
- 11. Mozilla Secret OPerationS (SOPS) secret-values.dec.yaml secret-values.enc.yaml
- 12. Mozilla Secret OPerationS (SOPS) secret-values.enc.yaml = data + encryption information Data Encryption Key ID Base64 encoded value of Encrypted Data key Message Authentication Code Prevents File Tampering JSON and YAML keys are not encrypted
- 13. Mozilla SOPS - Key Providers Microsoft Azure Key Vault SOPS_AZURE_KEYVAULT_URL Google Cloud KMS SOPS_GCP_KMS_IDS AWS KMS SOPS_KMS_ARN PGP SOPS_PGP_FP
- 14. ● Supported types: JSON, YAML and Binary ● Special support for JSON and YAML files ○ Encrypts only the values ○ File extension is used to determine the type ■ secret-values.yaml.enc → binary file ■ Secret-values.enc.yaml → yaml file ○ Some YAML types (anchors, streams, top-level arrays) are not supported Mozilla SOPS - File Types Input file is treated as blob Key fingerprint Encrypted data key
- 15. Mozilla SOPS - Multiple Master Keys Shared Services Global Master Key DEV QA PROD DEV QA PROD Business Unit 1 Business Unit 2Multiple Master Key Uses: - Encryption key in master AWS account - Encryption keys in different regions - Encryption keys in different key providers (hybrid cloud) - PGP key (stored offline)
- 16. Mozilla SOPS - .sops.yaml Filename path passed to sops is matched against this regex dev/secret-values.enc.yaml ✔ dev/secret-values.dec.yaml ✔ dev/aws-credentials.dec ✔ dev/aws-credentials ❌
- 17. Mozilla SOPS - .sops.yaml At root directory ~> sops -i -e dev/secret-values.enc.yaml ✔ ~> cd dev && sops -i -e secret-values.enc.yaml ❌
- 18. Option 1: in-place edit ● Data key doesn’t change ● Only the key value that was modified appears in the git diff Mozilla SOPS - Edit Files ~ > export EDITOR=vi ~ > sops -i secret-values.enc.yaml
- 19. Option 2: decrypt and encrypt ● Data key changes ● All key values appears to have changed in git diff Mozilla SOPS - Edit Files ~ > #Decrypt ~ > sops -d secret-values.enc.yaml > secret-values.dec.yaml ~ > #Encrypt ~ > sops -e secret-values.dec.yaml > secret-values.enc.yaml
- 20. Mozilla SOPS - Diffs in Cleartext in Git ~ > cat ~/.gitattributes *.enc diff=sopsdiffer *.enc.yaml diff=sopsdiffer *.enc.json diff=sopsdiffer .gitattributes ~ > grep -A1 sops ~/.gitconfig [diff "sopsdiffer"] textconv = "sops -d" .gitconfig In my testing, this worked if .sops.yaml is at the root directory of the git project and regex expression is based on the filename but not directory name dev/secret-values.enc.yaml → dev-secret-values.enc.yaml
- 21. Mozilla SOPS - Demo ● Install SOPS ● Create and export GPG keys ● Set up .sops.yaml, folder per environment and encrypt input file ● In-place edit
- 22. ● Manage secrets as code → check-in, build and deploy ○ Secrets should be encrypted before check-in ✔ ● Encryption keys management ○ Different encryption keys for environments ✔ ○ Centralized access → do not store key locally ✔ ○ Audit usage ✔ ○ Limit key access to individual members based on environment ✔ ○ Grant and revoke access easily ✔ ● Jenkins → minimal configuration on Jenkins nodes ○ Do not store encryption keys on Jenkins master or nodes ○ Secrets should not be left in plain text format on build success or failure ○ Store encrypted kube config files and AWS credentials in Git ● Limit secrets access to microservices that use it Secrets Management Requirements
- 23. Jenkins /jenkins/job1/ws Memory Jenkins job in a container /build /ws tmpfs bind ● Jenkins → minimal configuration on Jenkins nodes ○ Do not store encryption keys on Jenkins master or nodes → AWS KMS ○ Secrets should not be left in plain text format on build success or failure → Docker tmpfs ○ Store encrypted kube config files and AWS credentials in Git → AWS KMS Decrypted files are stored only in /build directory Jenkins Access: - ECR - KMS - S3
- 24. ● Manage secrets as code → check-in, build and deploy ○ Secrets should be encrypted before check-in ✔ ● Encryption keys management ○ Different encryption keys for environments ✔ ○ Centralized access → do not store key locally ✔ ○ Audit usage ✔ ○ Limit key access to individual members based on environment ✔ ○ Grant and revoke access easily ✔ ● Jenkins → minimal configuration on Jenkins nodes ○ Do not store encryption keys on Jenkins master or nodes ✔ ○ Secrets should not be left in plain text format on build success or failure ✔ ○ Store encrypted kube config files and AWS credentials in Git ✔ ● Limit secrets access to microservices that use it Secrets Management Requirements
- 25. Limit Secrets Access to Microservices Separate Secret object for each credential + {{ if .Values.secret1Enabled }} - name: {{ .Values.secret.secret1 }} secret: secretName: {{ .Values.secret.secret1 }} {{ end }} +secret1Enabled: true secret2Enabled: false Helm’s values.yaml Helm’s Deployment.yaml
- 26. ● Manage secrets as code → check-in, build and deploy ○ Secrets should be encrypted before check-in ✔ ● Encryption keys management ○ Different encryption keys for environments ✔ ○ Centralized access → do not store key locally ✔ ○ Audit usage ✔ ○ Limit key access to individual members based on environment ✔ ○ Grant and revoke access easily ✔ ● Jenkins → minimal configuration on Jenkins nodes ○ Do not store encryption keys on Jenkins master or nodes ✔ ○ Secrets should not be left in plain text format on build success or failure ✔ ○ Store encrypted kube config files and AWS credentials in Git ✔ ● Limit secrets access to microservices that use it ✔ Secrets Management Requirements
- 27. Questions ?