Reusable, composable, battle-tested

TERRAFORM
MODULES
Your project is code complete
and it’s time to deploy it!
I know, I’ll use AWS!
You login to the AWS console
(Spend hours reading docs)
OK, I have a server running!
What else do I need?
Well, you probably want more than
one server for high availability
And a load balancer to distribute
traffic across those servers
And EBS Volumes and RDS databases
to store all of your data
You’ll need an S3 bucket for files
CloudWatch for monitoring
Don’t forget the VPC, subnets, route
tables, NAT gateways
Route 53 for DNS
ACM for SSL/TLS certs and KMS to
encrypt / decrypt secrets
 stage prod

And you need all of that in separate

environments for stage and prod
 stage prod

Plus DevOps tooling to manage it all

 stage prod

And a CI server to test it all

 stage prod

Plus alerts and on-call rotation to

notify you when it all breaks
stage prod

And also…
And you have to maintain it all.
Forever.
AWS: 1,000 new releases in 2021
Terraform: release every ~2 weeks
Security vulnerabilities: daily
There’s a better way to deploy
and manage infrastructure:
TERRAFORM
MODULES
TERRAFORM
MODULES
Reusable, composable, battle-tested
infrastructure code
I’ll show you
How Terraform Modules work?
 stage prod

And how they will allow you to do all

of this…
> te r r afo rm i n i t <…>

> te r r afo rm appl y

In just a few simple commands

Outline
1. What’s a Module
2. How to use a Module
3. How Modules work
4. The future of Modules
Outline
1. What’s a Module
2. How to use a Module
3. How Modules work
4. The future of Modules
Terraform Modules

Code reuse
Remote or local source
Terraform evaluation
Mini-Terraform configuration
Multiple instances (no count)
 Module Components

Variables Resources Outputs

 Rails MySQL GitHub

CPU Memory Disk Drive Network Server DB

Platform as a Service (PaaS): e.g.,

Heroku, Docker Cloud, Engine Yard
 stage prod

But that requires dealing with this…

Terraform is a tool for defining and
managing infrastructure as code
provider "aws" {
region = "us- east - 1"
}

resource " aws_instance" "e xample " {

ami = "ami-408c7f28"
instance_type = " t 2 . m i c r o "
}

Example: Terraform code to

deploy a server in AWS
> te r r afo rm apply
aws_instance.example: Creating...
ami: "" => "ami-408c7f28"
instance_type: "" => " t 2 . m i c r o "
key_name: "" => "<computed>"
private_ip: "" => "<computed>"
public_ip: "" => "<computed>”
aws_instance.example: Creation complete

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Run terraform apply to deploy

the server
Terraform supports modules
They are like reusable blueprints
for your infrastructure
resource " aws_autoscal ing_group" "example" {
name = "${var.name}-service"
min_size = "${var.num_instances}"
max_size = "${var.num_instances}"
}

resource " aws_launch_co n f i g u ra t i on" " example" {

image_id = "${var.image_id}"
instance_type = " $ { v a r. i n s ta n c e _ ty p e } "
root_block_device {
volume_type = "gp2"
vol ume_si ze = 200
Exam ple: create a module to
}
}
deploy a microservice
module " ser vice_fo o" {
source = "/modules/microservice"
image_id = "ami-123asd1"
num_instances = 3
}

Now you can use the module to

deploy one microservice
module " ser vice_fo o" {
source = "/modules/microservice"
image_id = "ami-123asd1"
num_instances = 3
}
module " ser vice_bar " {
source = "/modules/microservice"
image_id = "ami-f2bb05ln"
num_instances = 6
}
module "service_baz" {
source = "/modules/microservice"
image_id = "ami-ny6v24xa"
Or multiple microservices
num_instances = 3
}
 CPU Memory Disk Drive Network Server DB

Modules allow you to use your

favorite IaaS provider…
 Rails MySQL GitHub

CPU Memory Disk Drive Network Server DB

With easy-to-use, high-level

abstractions, like a PaaS
 Rails MySQL GitHub

CPU Memory Disk Drive Network Server DB

But since you have all the code, you

still have full control!
Best of all: code can be shared
and re-used!
The Terraform Module Registry
A collection of reusable, verified,
supported Modules
Example: Vault Module for AWS
Example: Consul for Azure
Example: Nomad for GCP
Outline
1. What’s a Module
2. How to use a Module
3. How Modules work
4. The future of Modules
Imagine you wanted to deploy Vault
The old way:
1. Open up the Vault docs
2. Deploy a few servers
3. Install Vault
4. Install supervisord
5. Configure mlock
6. Generate self-signed TLS cert
7. Create Vault config file
8. Create an S3 bucket as storage backend
9. Figure out IAM policies for the S3 bucket
10. Tinker with security group rules
11. Figure out IP addresses to bind to and advertise
12. Fight with Vault for hours because it won’t accept your TLS cert
13. Regenerate cert with RSA encryption
14. Update OS certificate store to accept self-signed certs
15. Realize you need to deploy a Consul cluster for high availability
16. Open up the Consul docs…
The new way:
> te r r afo rm i n i t hashicorp/vault/aws

> te r r afo rm apply

And now you have this deployed
As simple as a PaaS!
> tree
.
├── README.md
├── m a i n . t f
├── o u t p u t s . t f
├── packer
├── user-data
└── v a r i a b l e s . t f

But you also have all the code.

Feel free to edit it!
module " vaul t _ c l u s t e r " {
source = "hashicorp/vault/aws"
cluster_name = " e x a m p l e - v a u l t - cl u s t e r "
cluster_size = 3
vpc_id = " $ { d a t a .a w s _ v p c .d e f a u l t. i d } "
subnet_ids = "${data.aws_subnets.default.ids}"
}
module " cons u l_ cl u ste r " {
source = "hashicorp/consul/aws"
cluster_name = "example-consul-cluster"
cluster_size = 3
vpc_id = " $ { d a t a .a w s _ v p c .d e f a u l t. i d } "
subnet_ids = "${data.aws_subnets.default.ids}"
Exam ple: modify main.tf
}
> te r r afo rm appl y

Run apply when you’re done!

Outline
1. What’s a Module
2. How to use a Module
3. How Modules work
4. The future of Modules
def add( x, y) :
return x + y

Most programming languages

support functions
def add( x , y ) :
return x + y

The function has a name

def add( x, y) :
return x + y

It can take in inputs

def add( x , y ) :
return x + y

And it can return outputs

def add( x , y ) :
return x + y

add(3, 5 )
add(10, 35)
add(- 45, 6)

Key idea: code reuse

def add( x , y ) :
return x + y

assert add(3, 5 ) == 8
assert add(10, 35) == 45
assert add(-45, 6 ) == -39

Key idea: testing

def add( x, y) :
return x + y

def sub( x, y) :
return x - y

sub(add( 5, 3) , add( 4, 7) )

Key idea: composition

Modules are Terraform’s
equivalent of functions
A simple module:
> t r e e mini mal- modu l e
.
├── m a i n . t f
├── o u t p u t s . t f
├── v a r i a b l e s . t f
└── README.md

It’s just Terraform code in a folder!

v a r i a b l e "name" {
desc r i pt i o n = " The name of t he EC2 i n s tance"
}

v a r i a b l e "image_id" {
d e s c r i p t i o n = "The ID o f the AMI t o run"
}

variable "port" {
description = "The p o r t t o l i s t e n on f o r HTTP requests"
}

The inputs are in v a r i a b l e s . t f

output " u rl " {
value = " h t t p : / / $ { a w s _ i n s t a n c e . e x a m p l e . i p } : $ { v a r. p o r t } "
}

The outputs are in o u t p u t s . t f

resource " aws_autoscal ing_group" "example" {
name = "${var.name}-service"
min_size = "${var.num_instances}"
max_size = "${var.num_instances}"
}

resource " aws_launch_co n f i g u ra t i on" " example" {

image_id = "${var.image_id}"
instance_type = " $ { v a r. i n s ta n c e _ ty p e } "
root_block_device {
volume_type = "gp2"
volume_size = 200
}
The resources are in m a i n . t f
}
# Foo Module f o r AWS

This i s a Terraform Module t o deploy

[Foo](http://www.example.com) on AWS, i n c l u d i n g :

* foo
* bar
* baz

Documentation is in README.md
A more complicated module:
> t r e e compl e t e - mod u le
.
├── m a i n . t f
├── o u t p u t s . t f
├── v a r i a b l e s . t f
├── README.MD
├── modules
├── examples
└── t e s t

We add three new folders:

modules, examples, t e s t
>t r e e complete-module/modules
modules/
├── submodule-bar
│ ├── m a i n . t f
│ ├── o u t p u t s . t f
│ └── v a r i a b l e s . t f
└── submodule-foo
├── m a i n . t f
├── o u t p u t s . t f
└── v a r i a b l e s . t f

The modules folder contains

standalone “submodules”
>t r e e complete-module/modules
modules/
├── submodule-bar
│ ├── i n s t a l l - v a u l t . s h
│ └── r u n - v a u l t . s h
└── submodule-foo
└── main.go

Some of the submodules may not

even be Terraform code
For example, one submodule can be
used to install Vault in an AMI
Another to create self-signed TLS
certificates
Another to deploy the AMI across an
Auto Scaling Group (ASG)
Another to create an S3 bucket and
IAM policies as a storage backend
Another to configure the Security
Group settings
And one more to deploy a load
balancer (ELB)
You can use all the submodules
Or pick the ones you want and swap
in your own for the rest
>t r e e complete-module/examples
examples/
├── example-foo
│ ├── m a i n . t f
│ ├── o u t p u t s . t f
│ └── v a r i a b l e s . t f
└── example-bar
├── m a i n . t f
├── o u t p u t s . t f
└── v a r i a b l e s . t f

The examples folder shows how to

use the submodules
> t r e e compl e t e - mod u le
.
├── m a i n . t f
├── o u t p u t s . t f
├── v a r i a b l e s . t f
├── README.MD
├── modules
├── examples
└── t e s t

Note: the code in the root is

usually a “canonical” example
It’s typically an opinionated way to
use all the submodules together
>t r e e complete-module/examples
examples/
├── example-foo
│ ├── m a i n . t f
│ ├── o u t p u t s . t f
│ └── v a r i a b l e s . t f
└── example-bar
├── m a i n . t f
├── o u t p u t s . t f
└── v a r i a b l e s . t f

The code in examples shows other

possible permutations
E.g., How to use just one or two of the
submodules together
Or how to combine with other
modules (e.g., Vault + Consul)
This is like function composition!
>t r e e compl e t e - mod u le/ t e s t
test/
├── example_foo_test.go
└── example_bar_test.go

The t e s t folder contains

automated tests
>t r e e compl e t e - mod u le/ t e s t
test/
├── example_foo_test.go
└── example_bar_test.go

The tests are typically “integration

tests”
func vau l t Te s t ( t * tes t ing.T, o p ti ons * te r r a t est.Op tion s) {
t l s C e r t : = generateSelfSignedTlsCert(t)
defer cleanupSelfSignedTlsCert(t, t l s C e r t )

amiId : = b u i l d Va u l t A m i ( t )
defer c l eanupAmi( t , amiId)

terratest.Apply(options)
defer t e r r a t e s t . D e s t r o y ( o p t i o n s )

assertCanInitializeAndUnsealVault(t, o p tio n s )
}

Example test case for Vault

func vau l t Te s t ( t * tes t ing.T, o p ti ons * te r r a t est.Op tion s) {
t l s C e r t : = generateSelfSignedTlsCert(t)
defer cleanupSelfSignedTlsCert(t, t l s C e r t )

amiId : = b u i l d Va u l t A m i ( t )
defer c l eanupAmi( t , amiId)

terratest.Apply(options)
defer t e r r a t e s t . D e s t r o y ( o p t i o n s )

assertCanInitializeAndUnsealVault(t, o p tio n s )
}

Create test-time resources

func vau l t Te s t ( t * tes t ing.T, o p ti ons * te r r a t est.Op tion s) {
t l s C e r t : = generateSelfSignedTlsCert(t)
defer cleanupSelfSignedTlsCert(t, t l s C e r t )

amiId : = b u i l d Va u l t A m i ( t )
defer c l eanupAmi( t , amiId)

terratest.Apply(options)
defer t e r r a t e s t . D e s t r o y ( o p t i o n s )

assertCanInitializeAndUnsealVault(t, o p tio n s )
}

Run terraform apply

func vau l t Te s t ( t * tes t ing.T, o p ti ons * te r r a t est.Op tion s) {
t l s C e r t : = generateSelfSignedTlsCert(t)
defer cleanupSelfSignedTlsCert(t, t l s C e r t )

amiId : = b u i l d Va u l t A m i ( t )
defer c l eanupAmi( t , amiId)

terratest.Apply(options)
defer t e r r a t e s t . D e s t r o y ( o p t i o n s )

assertCanInitializeAndUnsealVault(t, o p tio n s )
}

Run terraform destroy at the end

func vau l t Te s t ( t * tes t ing.T, o p ti ons * te r r a t est.Op tion s) {
t l s C e r t : = generateSelfSignedTlsCert(t)
defer cleanupSelfSignedTlsCert(t, t l s C e r t )

amiId : = b u i l d Va u l t A m i ( t )
defer c l eanupAmi( t , amiId)

terratest.Apply(options)
defer t e r r a t e s t . D e s t r o y ( o p t i o n s )

assertCanInitializeAndUnsealVault(t, o p tio n s )
}

Check the Vault cluster works!

Using a module:
module " ser vice_fo o" {
source = "./minimal-module"

name = "Foo"
image_id = "ami-123asd1"
port = 8080
}

For simple Modules and learning,

deploy the root
module " submodule_foo " {
source = "./complete-module/modules/submodule-foo"
param_foo = " f o o "
param_bar = 8080
}
module " submodule_bar" {
source = "./complete-module/modules/submodule-bar"
param_foo = "abcdef"
param_bar = 9091
}

For more complicated use-cases,

use the submodules
module " ser vice_fo o" {
source = "./minimal-module"
name = "Foo"
image_id = "ami-123asd1"
port = 8080
}

Abstraction: simple Module API

for complicated infrastructure
module " ser vice_fo o" {
source = "./minimal-module"
name = "Foo"
image_id = "ami-123asd1"
port = 8080
}
module " ser vice_bar " {
source = "./minimal-module"
name = "Bar"
image_id = "ami-abcd1234"
port = 9091
}
Re-use: create a Module once,
deploy it many times
Versioning:
module " ser vice_fo o" {
source = "./foo"

name = "Foo"
image_id = "ami-123asd1"
port = 8080
}

You can set source to point to

modules at local file paths
module " ser vice_fo o" {
source = "hashicorp/vault/aws"

name = "Foo"
image_id = "ami-123asd1"
port = 8080
}

Alternatively, you can use

Terraform registry URLs
module " ser vice_fo o" {
source = "git::[email protected]:foo/bar.git"

name = "Foo"
image_id = "ami-123asd1"
port = 8080
}

Or arbitrary Git URLs

module " ser vice_fo o" {
source = "git::[email protected]:foo/bar.git ?ref=v1.0.0"

name = "Foo"
image_id = "ami-123asd1"
port = 8080
}

You can even link to a specific Git

tag (recommended!)
module " ser vice_fo o" {
source = "git::[email protected]:foo/bar.git ?ref=v1.0.0"

name = "Foo"
image_id = "ami-123asd1"
port = 8080
}

Modules use semantic versioning

module " ser vice_fo o" {
source = "git::[email protected]:foo/bar.git ?ref=v2.0.0"

name = "Foo"
image_id = "ami-123asd1"
port = 8080
}

So upgrading infrastructure is just

a version number bump
 qa

stage

prod

Promote immutable, versioned

infrastructure across environments
Outline
1. What’s a Module
2. How to use a Module
3. How Modules work
4. The future of Modules
“You are not special. Your
infrastructure is not a beautiful
and unique snowflake. You
have the same tech debt as
everyone else.”

— your sysadmin, probably

stage prod

You need this

 stage prod

And so does everyone else

Stop reinventing the wheel
Start building on top of
battle-tested code
Start building on top of
commercially-supported code
Start building on top of
code
Advantages of code
1. Reuse
2. Compose
3. Configure
4. Customize
5. Debug
6. Test
7. Version
8. Document
 At Gruntwork, we’ve
been building
Modules for years

gruntwork.io
 Many companies, all
running on the same
infrastructure code

gruntwork.io
 stage prod

Modules allow us to turn this…

> te r r afo rm i n i t <…>

> te r r afo rm appl y

… into this
With some help from this
Questions?
Simple workflow
 Workflow:
Modules
● Code reuse
● Apply versioning
● Use version constraints
● Store code remotely
● Easier testing
● Encapsulation
● Use and contribute to Module Registry
 Workflow: Modules
What is the good terraform module?

● Clean and flexible code

● Well presented default values
● Covered with tests
● Examples
● Documentation
● Changelog
● Secure

Do not overload modules with features:

Terraform does cloning everything
https://www.endava.com/en/blog/Engineering/2019/11-Things-I-wish-
I-knew-before-working-with-Terraform-I
 Terraform Module

variable “name” {}

resource “aws_s3_bucket” “bucket”{

name = “${var.name}”
[…]
}

output “bucket_id” {
value = “${aws_s3_bucket.bucket.id}”
}
 Terraform Module
#Create module bucket
module “bucket” {
name = “MahBucket”
source = “.\\Modules\\s3”
}
#Use MahBucket
resource “aws_s3_bucket_object” {
bucket = “${module.bucket.bucket_id}”
key = “/walrus/bucket.txt”
source = “./mahbucket.txt”
}
Use a Module
Module Parameter