A tale of application development

A tale of application
development
Nicolas Corrarello - HashiConf 2018

Disclaimer
• Argentinian by birth, Italian by blood

• English is not my ﬁrst language

• I’m not a developer

• Don’t criticise my programming language choices, I
basically had Christmas week last year to write this :D.
Client libraries available in lots of languages (https://
www.vaultproject.io/api/libraries.html / https://
www.consul.io/api/libraries-and-sdks.html)

(What’s the story) Morning
glory?
• Spending lots of time authoring documents

• Write an app to automate the process

• Don’t have time to maintain it so make it as readable as
possible, zero touch maintenance as possible

1. IMPORTANT - READ BEFORE SIGNING. Are there Consul / Vault Tokens readily
available on the system (see Nomad or the myriad of Vault Authentication Methods
available, then use Vault to get a Consul token :D)

Are you provisioning this yourself or someone else doing it for you?

J. Developer

AbstractionsNot HashiCorp speciﬁc (duh)
“The best thing to come out of Java” - Variety

Abstract your code from lock-in

Minimal eﬀort, greater portability

Using existing libraries (datamapper, dal.py,
hibernate) may help. Also may make things way
more complicated (in certain cases, see Rails)

require 'json'
require './models/vault'
##
# Persistence layer. Implemented in S3, but the interface is generic enough so
it can be migrated to other platforms.
class Persistence
##
# Create a persistence layer. Initialises a HashiCorp Vault session and obtains
credentials from it using the existing abstraction.
def initialize
@vault = LocalVault.new
@awscreds = self.awsClient
@data = {}
end
##
# Store something in the persistence layer. Uses namespace (which translates to
an s3 bucket), and a simple key/value. Object is stored as JSON.
def Store(key,value,namespace)
s3 = @awscreds
begin
obj = s3.bucket(namespace).object(key)
obj.put(body: value.to_json)
return true, nil
rescue
return false, "Error persisting object"
end
end
##
# Retrieve something from the persistence layer. Uses namespace (which
translates to an s3 bucket), and a key. Returns the full object in it's native
format.
def Retrieve(key,namespace)
[…]
end
##
# List objects by "glob". Returns a list of objects available. In this case as
S3 is somewhat hierarchical, glob would be 'customers' to to retrieve customer
objets, or 'pov' to return POV documents.
def List(glob,namespace)
[…]
end
end
Initialize the object
Store something

Let’s talk secrets… lots…
• AWS API Credentials for Deployment

• AWS Credentials for the application to read/write the
bucket

• JWT Issuer / Secret

• Google API Keys for logins

• Crypto

Crypto?
Not the kind that your
coworker keeps talking
about, you know who,
the self named Bitcoin
expert….

##
# Creates a customer in the object, requires an sfdcAccountId (unique)
and a customer name. Information is encrypted and persisted, but kept in
plaintext in memory.
def Create(sfdcAccountId,name,domain)
unless sfdcAccountId.nil? || name.nil?
begin
cyphertext = @vault.logical.write("transit/encrypt/#{key}",
plaintext: Base64.encode64(value).gsub('n',''), context:
Base64.encode64(context).gsub('n',''))
cyphername = cyphertext.data[:ciphertext]
rescue
return false, "Error encrypting data"
end
customer = {
:id => sfdcAccountId,
:name => cyphername,
:contacts => nil,
}
customerm = {
:name => name,
:contacts => nil,
}
key = 'customers/' + sfdcAccountId + '/customer.json'
begin
@persistence.Store(key,customer,"jamo-#{domain}")
if @data[domain] == nil
@data[domain] = {}
end
@data[domain][sfdcAccountId] = customerm
return true, nil
rescue
return false, "error persisting customer"
end
end
end
Encrypt a string
Persist it

##
# Creates a customer in the object, requires an sfdcAccountId (unique)
and a customer name. Information is encrypted and persisted, but kept
in plaintext in memory.
def Create(sfdcAccountId,name,domain)
unless sfdcAccountId.nil? || name.nil?
begin
cyphername = @vault.encrypt(name,'foo', @jwt_secret)
rescue
return false, "Error encrypting data"
end
customer = {
:name => cyphername,
:contacts => nil,
}
customerm = {
:name => name,
:contacts => nil,
}
key = 'customers/' + sfdcAccountId + '/customer.json'
begin
@persistence.Store(key,customer,"jamo-#{domain}")
if @data[domain] == nil
@data[domain] = {}
end
@data[domain][sfdcAccountId] = customerm
return true, nil
rescue
return false, "error persisting customer"
end
end
end
##
# Encrypt a value with a defined key and
context for symmetric encryption.
def encrypt(value,key,context)
cyphertext =
@vault.logical.write("transit/encrypt/#{key}",
plaintext:
Base64.encode64(value).gsub('n',''), context:
return cyphertext.data[:ciphertext]
end

require 'vault'
require 'base64'
##
# This object abstract the crypto / secrets management functions of HashiCorp Vault
class LocalVault
def initialize
@vault = Vault::Client.new
end
##
# Obtain a set of credentials to access the S3 Bucket used by the persistence layer
def getAwsCreds
credentials = @vault.logical.read("aws/creds/s3-bucket")
return credentials
end
def getGoogleCredentials
credentials = @vault.logical.read("secret/gsuite")
return credentials
end
def getJWTsecret
jwtsecret = @vault.logical.read("secret/JWT")
return jwtsecret
end
##
# Encrypt a value with a defined key and context for asymmetric encryption.
def encrypt(value,key,context)
cyphertext = @vault.logical.write("transit/encrypt/#{key}", plaintext:
Base64.encode64(value).gsub('n',''), context:
return cyphertext.data[:ciphertext]
end
##
# Decrypt a value with a defined key and context.
def decrypt(value,key,context)
plaintext = @vault.logical.write("transit/decrypt/#{key}", ciphertext: value,
context: Base64.encode64(context).gsub('n',''))
return Base64.decode64(plaintext.data[:plaintext]).gsub('n','')
end
end
Initialize the object
Get dynamically generated AWS Tokens
Get static GSuite API Keys
Get info to sign JWTs
Encrypt a string
Decrypt a string

2. How is this application getting its Vault / Nomad / Consul token?

How is this application supposed to get it’s runtime conﬁguration?

J. Developer

group "webs" {
count = 1
task "jamo" {
vault {
policies = ["jamo"]
change_mode = "signal"
change_signal = "SIGHUP"
}
driver = "docker"
env {
VAULT_ADDR = “https://vault.service.consul:8200"
MEMCACHE_ADDR = "memcache.service.consul:11211"
}
[...]
}
Let Nomad get me a Token
What to do if that token is rotated
Environment variables.
DNS resolved by Consul

What if I’m using K8s?
• The good option, is using the K8s authentication method
and doing a login in the Dockerﬁle so the pod gets it
VAULT_TOKEN and then consume secrets

• The not bad option, is using consul-template to deploy a
helm chart with the secrets, doing authentication in the
pipeline that deploys it (see https://www.hashicorp.com/
resources/how-to-share-secrets-pipeline or http://
nicolas.corrarello.com/general/vault/security/ci/
2017/04/23/Reading-Vault-Secrets-in-your-Jenkins-
pipeline.html)

Perfect > Good Enough
• Getting credentials out of Vault is super easy, so
it’s easy to fall on the trap of having very short TTL
on secrets.

• Don’t underestimate complexity in recovering from
failures in running state

• Focus on business logic! (Always with reasonable
abstractions)

get '/' do
begin
client = Mysql2::Client.new(:host =>
mysqladdr, :username =>
mysqlvars.data[:username], :password =>
mysqlvars.data[:password])
mysqlstatus = client.query("SHOW STATUS")
rescue
puts "Asking for new credentials"
mysqlvars = getmysqlcreds(localvault)
client = Mysql2::Client.new(:host =>
mysqladdr, :username =>
mysqlvars.data[:username], :password =>
mysqlvars.data[:password])
mysqlstatus = client.query("SHOW STATUS")
end
##
# Returns an S3 client
protected
def awsClient
awscreds = self.awsCreds
begin
s3 =
Aws::S3::Resource.new(access_key_id:
awscreds[:access_key], secret_access_key:
awscreds[:secret_key], region: 'us-east-1')
return s3
rescue
awscreds = self.awsCreds
s3 =
Aws::S3::Resource.new(access_key_id:
awscreds[:access_key], secret_access_key:
awscreds[:secret_key], region: 'us-east-1')
return s3
end
end

3. Are you supposed to provide credentials / conﬁguration to the application owner? Do
application teams chuck releases over the fence to you?

J. Developer

Friction-less alternatives
• Consul-template

• Great for PKI & Existing servers (see https://www.yet.org/
2018/10/vault-pki/ by Sebastién Braun)

• Envconsul

• Similar pattern, but with environment variables

##
# Wait for lock to be acquired before returning form
data
sessionid = Diplomat::Session.create({:hostname =>
"server1", :ipaddress => "4.4.4.4"})
lock_acquired = Diplomat::Lock.wait_to_acquire("/
key/to/lock", sessionid)
##
# Set a lock for the record that is about to
start being modified
sessionid = Diplomat::Session.create({:Node
=> "consul.service.consul", :Name =>
"locking session"})
lock_acquired = Diplomat::Lock.acquire("/
jamo/locks/#{docid}", sessionid)
# Do stuff
[…]
# Release lock
Diplomat::Lock.release("/jamo/locks/
#{docid}", sessionid )

Load balancing
• Fabio is awesome! I don’t even consider it’s there, it just
works!

• Not the only great pattern, see https://www.hashicorp.com/
resources/favorite-consul-customer-success-story

Not used in this project but
really cool….
• K/V Store

• Blocking queries

• Prepared queries

• Leader election

Don’t do this at home…
##
# Get Docker Hub Webhook
post '/dockerhubwebhook' do
callback = request.body.read
docker = JSON.parse(callback)
puts docker['push_data']['tag']
if docker['push_data']['tag'] != 'latest'
puts 'Deploying new Jamo Version' + docker['push_data']['tag']
nomadjob = erb :jsonnomad, :locals => { :tag =>
docker['push_data']['tag'] }
Nomad.job.create(nomadjob)
end
end

So what was
accomplished?
• 100% of the code is business code (alright, minus the
HTTP interface). Decoupled logical objects, from HTTP API,
from presentation layer that can be maintained individually

• Data encrypted at rest and in transit

• All credentials are ephemeral / short lived

• Microservices loosely coupled, discovered via Consul

• Cloud agnostic

What’s next?
• Already have a scheduler, stop running long task on
runtime and just schedule things! (I can get Nomad tokens
from the Vault API)

• Stop TLS Nightmare / port forwarding. Have Consul
Connect handle that

Stop TLS nightmare
▪ Service Access Graph
▪ Certificate Authority (built-in, Vault, Others)
▪ Application Integration

A tale of application development

More Related Content

What's hot (20)

Similar to A tale of application development (20)

Recently uploaded (20)

A tale of application development