HashiCorp Vault configuration as code via HashiCorp Terraform- stories from trenches

Vault conﬁguration as code via Terraform:
stories from trenches
Andrey Devyatkin

I’m Andrey
● Enjoying life as technology
specialist, father and endurance
athlete
● 10+ years in the industry
● Writing tools
● Fixing automation, projects and
organisations
● Meetups/conferences organizer
● Trainer
● Certiﬁed this and that

Why this presentation? What to expect?
● Not pretending to be an expert just sharing what worked and what didn’t
● Share what worked for us and hopefully save some time for some of you
● A lot of technical details and references
● Slides will be available online - you don’t have to remember/photo everything

Agenda
● Context
● Deployment
● Conﬁguration
● Integration
● Access control and development workflow
● Unexpected ﬁndings

Why Vault?
● Centrally Manage Secrets to Reduce
Secrets Sprawl
● Shift from static secrets to short-time
dynamically generated ones
● Protect Sensitive Data Across Clouds
and Private Datacenters

Where do we start?
Collect requirements and clarify context

Questions to ask - deployment and operations
● Where to deploy? VM? Container? Baremetal?
● Patch or scratch?
● How to access? VPN? Public? Service mesh?
● How to auto-unseal?
● How to get in initial secrets? (Ex. TLS)
● What storage is available?
● Where to stream logs?
● Where to stream telemetry?
● How to extract audit ﬁles?
● HA? DR?
● One per env or one for all?

Look for best practices and templates
● Why not to make use of https://github.com/hashicorp/terraform-aws-vault, right?
● With some small tweaks
● And little more tweaks
● And some more tweaks
(TLS certs handling, consul-less, dynamic configuration depending on environment,
rolling upgrades via multiple auto scaling configurations, audit files sync, etc)
Another option - https://github.com/hashicorp/vault-helm
Some inspiration
https://learn.hashicorp.com/vault/operations/ops-reference-architecture

Vault production (min) readiness checklist
● TLS termination
● Vault HA storage - ACL and encryption
● Local storage encryption
● Auto-unseal using KMS
● Stripped down image, infra as code,
encryption, minimal exec rights
● No ssh or other kind of remote access, NACL
for outgoing traffic
● IDS
● Backups and DR
● Logs and telemetry export from the node
● Audit on, sync audit files to remote storage,
integrity check for audit files
● Sync audit files to archive
● Audit files parsing and anomaly detection
● Availability/performance monitoring and
alerting
More here https://learn.hashicorp.com/vault/operations/production-hardening

Context - before Vault
● Applications running in containers
● Orchestrated by Kubernetes
● Running in AWS
● Secrets in conﬁgmaps/secrets
● Apps require database connection and connection to other cloud services (i.e.
database creds and cloud access creds), other static secrets
● Developers pulling secrets from k8s secrets

Deployment
● EC2, classic ELB, auto-scaling group per AZ (for rolling updates)
● Immutable infra
● Initial secrets baked in as encrypted archive that could be un-encrypted only with key
accessible to Vault instances
● Behind VPN
● Auto-unseal with KMS
● HA storage in DynamoDB with point in time restore
● Logs in CloudWatch
● Telemetry in Prometheus

It would be a good idea
to
split deployment terraform spec and
conﬁguration terraform spec

Vault is up and running. What is next?

To start conﬁguring Vault via Terraform we need...
● Vault URL conﬁgured as VAULT_ADDR env variable
● Vault token (root token will do for the start but revoke it afterwards together with the
rest of the root tokens)
● A good idea what are you after…
More here https://www.youtube.com/watch?v=fOybhcbuxJ0 and here
https://www.terraform.io/docs/providers/vault/index.html

One slide Vault intro
LDAP
k8s
App
Role
AWS
...
Auth methods
Vault
token
AWS
Data
base
Secret Engines
Rabbit
MQ
PKI
Database login credentials
AWS access keys
RabbitMQ logic credentials
Certiﬁcates
Lease
Audit device
More here https://www.youtube.com/watch?v=VYfl-DpZ5wM
KV
Transit Encrypted data
Secret value
Vault
policies

Token
Be aware of TTL and Max TTL

Auth methods and policies
Boring but very important

You probably need more than one...
● Humans - operators and developers
● Machines - CI/CD, bots, etc
● Things - Apps, Infra etc
A good idea is to use MFA for humans, limit from where auth methods could be invoked

Auth -> Role -> Token with policy
Ex.
LDAP -> LDAP Group -> Token with policy

LDAP
● Leverages existing IAM setup
● Delegates credentials validation
● Used my humans
● Would be a good idea to simplify login procedure for your users
More here https://www.vaultproject.io/docs/auth/ldap.html

LDAP
resource "vault_ldap_auth_backend" "ldap" {
path = "ldap"
url = "ldaps://dc-01.example.org"
userdn = "OU=Users,OU=Accounts,DC=example,DC=org"
userattr = "sAMAccountName"
upndomain = "EXAMPLE.ORG"
binddn = "${var.binddn}"
bindpass = "${var.bindpass}"
discoverdn = false
groupdn = "OU=Groups,DC=example,DC=org"
groupfilter = "(&(objectClass=group)(member:1.:={{.UserDN}}))"
}
https://www.terraform.io/docs/providers/vault/r/ldap_auth_backend.html

Things to consider
● token_no_default_policy
● token_bound_cidrs
● token_ttl
● token_max_ttl

LDAP role (backend group actually)
resource "vault_ldap_auth_backend_group" "group" {
groupname = "dba"
policies = ["dba"]
backend = "${vault_ldap_auth_backend.ldap.path}"
}
https://www.terraform.io/docs/providers/vault/r/ldap_auth_backend_group.html

Policy
data "vault_policy_document" "example" {
rule {
path = "secret/*"
capabilities = ["create", "read", "update", "delete", "list"]
description = "allow all on secrets"
}
}
resource "vault_policy" "example" {
name = "example_policy"
policy = "${data.vault_policy_document.example.hcl}"
}
https://www.terraform.io/docs/providers/vault/d/policy_document.html

Policy
● You will need a policy to manage policy...
● Deny by default
● Do not have to match LDAP group name but easier for users if it
does
● Member of multiple groups gets multiple policies
More here https://learn.hashicorp.com/vault/getting-started/policies

AppRole if you really have to...
● If you don’t have a better way
● Mostly used for CI
● Initial secret issue
● No good way to audit access
More here https://www.vaultproject.io/docs/auth/approle.html

AppRole
resource "vault_auth_backend" "approle" {
type = "approle"
}
resource "vault_approle_auth_backend_role" "example" {
backend = vault_auth_backend.approle.path
role_name = "test-role"
token_policies = ["default", "dev", "prod"]
}
https://www.terraform.io/docs/providers/vault/r/approle_auth_backend_role.html

AppRole
resource "vault_approle_auth_backend_role_secret_id" "secret" {
backend = "approle"
role_name = "${vault_approle_auth_backend_role.role.role_name}"
}
locals {
kv = {
role_id = "${vault_approle_auth_backend_role.role.role_id}"
secret_id = "${vault_approle_auth_backend_role_secret_id.secret.secret_id}"
}}
resource "vault_generic_secret" "kv" {
path = "${vault_mount.kv.path}/approle"
data_json = "${jsonencode(local.kv)}"
}

Cloud IAM, K8S, etc
● Better way for non-interactive auth
● Leverages existing entities
● Delegates entity validation

AWS IAM
resource "vault_auth_backend" "aws" {
description = "Auth method for CI engines"
type = "aws"
}
resource "vault_aws_auth_backend_role" "ci_builder" {
backend = "${vault_auth_backend.aws.path}"
role = "ci-builder"
auth_type = "iam"
bound_iam_principal_arns = ["${data.aws_iam_role.ci_builder.arn}"]
token_ttl = 3600
token_max_ttl = 3600
token_policies = ["${vault_policy.ci_builder.name}"]
}
https://www.terraform.io/docs/providers/vault/r/aws_auth_backend_role.html

AWS IAM
resource "aws_iam_user" "ci_builder" {
name = "${module.iam_auth_for_ci_user_name.qualified_name}"
tags = "${module.tags.default}"
}
resource "aws_iam_access_key" "ci_builder" {
user = "${aws_iam_user.ci_builder.name}"
}
resource "aws_iam_user_policy" "ci_builder" {
name = "Allow-Vault-to-look-up-users-for-iam-auth"
user = "${aws_iam_user.ci_builder.name}"
policy = "${data.aws_iam_policy_document.ci_builder.json}"
}

AWS IAM
# https://www.vaultproject.io/docs/secrets/aws/index.html#example-iam-policy-for-vault
data "aws_iam_policy_document" "iam_auth_for_ci" {
statement {
effect = "Allow"
actions = ["iam:GetUser","iam:GetRole",]
resources = ["arn:aws:iam::${data.aws_caller_identity.i.account_id}:*"]
}
statement {
effect = "Allow"
actions = ["sts:AssumeRole"]
resources = ["arn:aws:iam::${data.aws_caller_identity.i.account_id}:role/vault-cluster*"]
}
}

AWS IAM
resource "vault_aws_auth_backend_client" "ci_builder" {
backend = "${vault_auth_backend.ci_builder.path}"
access_key = "${aws_iam_access_key.ci_builder.id}"
secret_key = "${aws_iam_access_key.ci_builder.secret}"
}

K8S
resource "kubernetes_service_account" "vault_auth" {
metadata {
name = "vault-auth"
namespace = "${var.k8s_namespace}"
}
automount_service_account_token = "true"
}
data "kubernetes_secret" "sa" {
depends_on = ${kubernetes_service_account.vault_auth}
metadata {
name = "${kubernetes_service_account.vault_auth.default_secret_name}"
}
}

K8S
resource "kubernetes_cluster_role_binding" "vault_auth" {
metadata { name = "role-tokenreview-binding" }
role_ref {
api_group = "rbac.authorization.k8s.io"
kind = "ClusterRole"
name = "system:auth-delegator"
}
subject {
kind = "ServiceAccount"
name = "${kubernetes_service_account.vault_auth.metadata.name}"
namespace = "${var.k8s_namespace}"
api_group = ""
}
}

K8S
resource "vault_kubernetes_auth_backend_config" "config" {
backend = "${vault_auth_backend.kubernetes.path}"
kubernetes_host = "${var.k8s_api}"
kubernetes_ca_cert = "${data.kubernetes_secret.sa.data.ca.crt}"
token_reviewer_jwt = "${data.kubernetes_secret.sa.data.token}"
}
resource "vault_kubernetes_auth_backend_role" "role" {
backend = "kubernetes"
role_name = "${var.app_name}"
bound_service_account_names = ["${kubernetes_service_account.app.metadata.0.name}"]
bound_service_account_namespaces = ["${var.k8s_namespace}"]
token_ttl = "${var.ttl}"
token_policies = ["${vault_policy.read-psql.name}, "${vault_policy.read-aws.name}"]
}

K8S
Here are more details if you are interested
https://www.youtube.com/watch?v=t6ZKhY0-_cA

KV
AWS SSM Parameters Store -> KV for humans
Conﬁgmaps/Secrets -> KV for Apps

KV
Human initiated secret storage for static secrets
and
Machine initiated-readable storage for static secrets

AWS
resource "aws_iam_user" "user" {
name = "vault-aws-admin"
}
resource "aws_iam_access_key" "key" {
user = "${aws_iam_user.user.name}"
}
resource "aws_iam_user_policy" "policy" {
name = "Allow-Vault-to-create-temp-users"
policy = "${data.aws_iam_policy_document.document.json}"
}

AWS
https://www.vaultproject.io/docs/secrets/aws/index.html#example-iam-policy-for-vault
"iam:AttachUserPolicy" "iam:ListGroupsForUser"
"iam:CreateAccessKey" "iam:ListUserPolicies"
"iam:CreateUser" "iam:PutUserPolicy"
"iam:DeleteAccessKey" "iam:RemoveUserFromGroup"
"iam:DeleteUser"
"iam:DeleteUserPolicy"
"iam:DetachUserPolicy"
"iam:ListAccessKeys"
"iam:ListAttachedUserPolicies"

AWS
resource "vault_aws_secret_backend" "aws" {
description = "AWS secret engine so operators can get temporary keys"
path = "aws"
region = "${data.aws_region.r.name}"
access_key = "${aws_iam_access_key.key.id}"
secret_key = "${aws_iam_access_key.key.secret}"
default_lease_ttl_seconds = "28800"
max_lease_ttl_seconds = "86400"
}

AWS
resource "aws_iam_role" "admin" {
name = "admin"
max_session_duration = "28800"
assume_role_policy = "${data.aws_iam_policy_document.trust.json}"
}
resource "vault_aws_secret_backend_role" "access-aws-admin-role" {
backend = "${vault_aws_secret_backend.aws.path}"
name = "access-aws-admin-role"
role_arns = ["${aws_iam_role.admin.arn}"]
credential_type = "assumed_role"
}

AWS
# https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_job-functions.html
data "aws_iam_policy" "admin" {
arn = "arn:aws:iam::aws:policy/SystemAdministrator"
}
resource "aws_iam_role_policy_attachment" "admin" {
role = "${aws_iam_role.admin.name}"
policy_arn = "${data.aws_iam_policy.admin.arn}"
}

AWS
data "aws_iam_policy_document" "trust" {
statement {
effect = "Allow"
actions = ["sts:AssumeRole"]
principals {
type = "AWS"
identifiers = ["${aws_iam_user.user.arn}"]
}
}
}

AWS
Works in the same way for apps and humans!

AWS
Use temporary AWS creds to generate sign-in AWS console URL! No SSO needed!
https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_enable-console
-custom-url.html

For more inspiration https://youtu.be/Y0er4UCmqiA

Database creds
● Creation and revocation statements are hard
● Use RDS IAM auth if possible

Secrets rotation
DB_SECRET_ENGINE_MOUNTS=$(vault secrets list -format=json | jq -r '. | to_entries[] | select(.value.type |
startswith("database")) | .key')
for DB_SECRET_ENGINE_MOUNT in ${DB_SECRET_ENGINE_MOUNTS}; do
DB_CONNECTION_NAMES=$(vault list -format=json ${DB_SECRET_ENGINE_MOUNT}config | jq --raw-output .[])
for DB_CONNECTION_NAME in ${DB_CONNECTION_NAMES}; do
vault write -force ${DB_SECRET_ENGINE_MOUNT}rotate-root/${DB_CONNECTION_NAME}
done
done

Secrets rotation
AWS_USERS=$(aws iam list-users --query "Users[?starts_with(UserName, 'vault-aws-')].UserName" --output text)
for AWS_USER in ${AWS_USERS}; do
KEYS_ID=$(aws iam list-access-keys --user-name ${AWS_USER} --query "AccessKeyMetadata[*].AccessKeyId" --output
text)
for KEY_ID in ${KEYS_ID}; do
aws iam delete-access-key --access-key-id ${KEY_ID} --user-name ${AWS_USER}
done
done
terraform apply
Note! Keys are still in Terraform state - encrypt state storage and state itself!

KV state issue
● Terraform provider for Vault in some cases(?) does not re-read KV and newly added
values are not readable/found
● terraform state rm data-source
data "vault_generic_secret" "rundeck_auth" {
path = "secret/rundeck_auth"
}
provider "rundeck" {
url = "http://rundeck.example.com/"
auth_token = "${data.vault_generic_secret.rundeck_auth.data["auth_token"]}"
}

Vault and Terraform not always play together
resource "vault_database_secret_backend_connection" "postgres" {
count = "${var.enable_postgresql}"
backend = "${vault_mount.db.path}"
name = "${var.postgresql_db_name}"
allowed_roles = ["${var.postgresql_role_name}", "${local.read_only_role_name}", "${local.admin_role_name}"]
data = {
username = "${var.postgresql_db_username}"
password = "${var.postgresql_db_username_password}"
}
postgresql {
connection_url =
"postgres://{{username}}:{{password}}@${var.postgresql_db_endpoint}:${var.postgresql_db_port}/${var.postgresql_db_name}"
max_open_connections = "${var.postgresql_max_open_connections}"
}
lifecycle {
ignore_changes = ["data.password"]
}
}

Vault and Terraform not always play together
resource "aws_iam_access_key" "key" {
}
resource "aws_iam_user_policy" "policy" {
name = "Allow-Vault-to-create-temp-users"
policy = "${data.aws_iam_policy_document.document.json}"
}
resource "vault_aws_secret_backend" "aws" {
description = "AWS secret engine for operators to get temporary keys"
path = "$humans-aws"
region = "${data.aws_region.r.name}"
access_key = "${aws_iam_access_key.key.id}"
secret_key = "${aws_iam_access_key.key.secret}"
default_lease_ttl_seconds = "28800"
max_lease_ttl_seconds = "86400"
}

Some misconceptions
data "vault_generic_secret" "db_creds" {
path = "${module.vault_config.database_path}/creds/${var.db_role_name}"
depends_on = ["module.vault_config"]
}
locals {
db_username = "${data.vault_generic_secret.db_creds.data["username"]}"
db_password = "${data.vault_generic_secret.db_creds.data["password"]}"
}
locals {
psql_url = "postgres://${local.db_username}:${local.db_password}@${module.db.connection_string}?sslmode=require”
}
resource "vault_generic_secret" "config" {
path = "${module.vault_config.kv_path}/config"
data_json = <<EOF
{
"Psql_url": "${local.psql_ur}"
}
EOF
}

Thanks!
Questions?
@andrey9kin
info@andreydevyatkin.com
https://www.linkedin.com/in/andreydevyatkin/

HashiCorp Vault configuration as code via HashiCorp Terraform- stories from trenches

Recommended

More Related Content

What's hot (20)

Similar to HashiCorp Vault configuration as code via HashiCorp Terraform- stories from trenches (20)

More from Andrey Devyatkin (13)

Recently uploaded (20)

HashiCorp Vault configuration as code via HashiCorp Terraform- stories from trenches