Secrets management vault cncf meetup
- 1. Do’s and Don’ts | Peter Gasper | 16.6.2021 Secrets Management
- 2. Whoami Peter Gasper - security engineer - practicing DevOps at Deutsche Telekom Pan-Net - currently „Access & Encryption“ squad lead Slack: cncfsk - #Peter Gasper e-mail: [email protected] Blog: https://malgregator.com GitHub: https://github.com/viralpoetry
- 3. Agenda - Problems with secrets - HashiCorp Vault - Vault‘s journey in Pan-Net - Vault Open Source limitations - Conclusion
- 4. If you are maintaining applications, at some point you have to: - rebuild infrastructure - change password - share credentials - revoke access Sensitive data often used during deployment: - API keys - SSH credentials - passwords Problems with secrets
- 5. Secret is anything used for authentication, authorization or encryption: - Webserver (TLS cert, DB credentials, API keys) - FreeRADIUS (shared secret with the VPN HW) - Database (credentials – user/password) Common problems: - Sensitive credentials and keys are stored in code repository (GitLab, GitHub, …) - Sensitive credentials and keys are stored in plain text - Sensitive credentials and keys are shared in numerous places 5 Problems with secrets
- 6. Traditional approach – small teams: - PGP - git-crypt - ansible-vault Problems: - secrets are still committed to a version control repository - people leaving organization – access to keys/passwords can‘t be revoked – rotate all the secrets - basically, no lifecycle Problems with secrets
- 7. Problems with secrets Solution - infrastructure “password” manager Basic Requirements: - single source of truth - provides API interface - encryption - detailed auditing - ability to revoke access - multiple authentication methods - highly available 7 Existing solutions - Keywhiz by Square - Confidant by Lyft - Conjur by CyberArk - Vault by HashiCorp
- 8. We chose Hashicorp Vault - Golang binary - highly available key/value store - encryption – Shamir secret sharing scheme - easy prototyping (vault server -dev) Vault by HashiCorp
- 9. Vault by HashiCorp Storage backends – Raft, Consul, Etcd, FoundationDB Secrets Engines - Static - k/v store for any blob of data – passwords, API tokens etc. - Dynamic - Database credentials, SSH access, AWS, Google Cloud, etc. - Encryption - PKI certificate authority, Transit backend Auth Methods - machine oriented (TLS, JWT, Tokens) - user oriented (user/pass, LDAP, GitHub, OKTA, Kubernetes, Radius, ...) 9 VAULT IS FOR SECRETS
- 10. Vault by HashiCorp 10 Every succesful authentication backend results in a token. Every token has access rights based on defined policy. Secrets are accessed using tokens.
- 11. Vault by HashiCorp 11 Token: - has expiration (TTL) - can be renewed
- 12. Vault by HashiCorp - CLI usage 12
- 13. Vault by HashiCorp - Ansible roles You define Vault address, mount point, path & name of a secret, e.g. radius_vpn_secret. Roles can generate the passwords if it does not exist 13 - hosts: radius-vpn gather_facts: no become: yes vars: - vault_mount: ”secret” - vault_path: ”my_project” - vars_stored: - { var: ’ldap_bind_password’ , key: ‘password’ , password: yes } - { var: ’radius_vpn_secret’ , key: ‘password’ , password: yes , length: 12 } roles: - ansible-load-secrets - ansible-save-secrets - ansible-freeradius
- 14. 2017 – MVP, basic workflow - Vault v0.6.5, single instance with filesystem backend, deployed with Ansible - Operation: exchange GPG keys, create policy, issue & deliver tokens - ansible-load-secrets, ansible-save-secrets roles Vault‘s journey in Pan-Net
- 15. 2018 – resiliency, more automation - whole provisioning & deployment as a code - access policies provisioned from Gitlab repo - rolling updates & HA setup with Consul Vault‘s journey in Pan-Net
- 16. 2019 – Self-Service Due to operation hell, we started designing self-service for the most common use case. You ask for a “safe“, then you can issue tokens for path and subpaths by yourself. Modus operandi: - logged-in person is mapped to an identity - “Safe“ is Vault identity group - group has policies - groups are added/removed to identities Vault‘s journey in Pan-Net
- 17. 2021 – way more clusters - migrating to Raft integrated storage - Gitlab integration using JWT in the pipeline - multiple new clusters deployed using Helm charts Vault‘s journey in Pan-Net
- 18. Limitations of the open-source version when doing advanced topics: - georedundancy - shamir secret unsealing for the first Vault - no PKCS #11 support for unsealing nor PKI - audit backend settings are not propagated in HA Vault Open Source limitations
- 19. Conclusion If I would start again: - secrets management reflects organizational structure, start with authn/authz lifecycle - use dynamic secrets engines for new infra - use policy templates, identity groups, automate role provisioning - write less Ansible for a setup phase, used libraries like Python HVAC instead