Terraform AWS modules and some best-practices - May 2019
11 likes1,547 views
Slides from my meetup talks during meetups in Germany. Follow me: https://twitter.com/antonbabenko https://github.com/antonbabenko
![- [ ] How to write modules?
- [ ] How to call modules?
- [ ] How to work with the code?
@antonbabenko](https://image.slidesharecdn.com/terraform-meetup-may-2019-190514150956/85/Terraform-AWS-modules-and-some-best-practices-May-2019-48-320.jpg)
![- [x] How to write modules?
- [x] Do not write, if you can
- [x] Avoid providers in modules, provisioners
- [ ] How to call modules?
- [ ] How to work with the code?
@antonbabenko](https://image.slidesharecdn.com/terraform-meetup-may-2019-190514150956/85/Terraform-AWS-modules-and-some-best-practices-May-2019-66-320.jpg)
![Orchestration = Terragrunt
@antonbabenko
subnet_id = ["subnet-1a2b3c4d"] # tfvars:terragrunt_output.vpc.public_subnets](https://image.slidesharecdn.com/terraform-meetup-may-2019-190514150956/85/Terraform-AWS-modules-and-some-best-practices-May-2019-90-320.jpg)
![- [x] How to write modules?
- [x] How to call modules?
- [x] 1-in-1 works beter over time
- [x] Orchestration = Terragrunt
- [x] Dynamic values in tfvars
- [ ] How to work with the code?
@antonbabenko](https://image.slidesharecdn.com/terraform-meetup-may-2019-190514150956/85/Terraform-AWS-modules-and-some-best-practices-May-2019-91-320.jpg)
![- [x] How to write modules?
- [x] How to call modules?
- [x] How to work with the code?
- [x] Lists in Terraform 0.11 can be painful
- [x] Perceive Terraform easier
@antonbabenko](https://image.slidesharecdn.com/terraform-meetup-may-2019-190514150956/85/Terraform-AWS-modules-and-some-best-practices-May-2019-105-320.jpg)
Ad
More Related Content
What's hot (20)
Similar to Terraform AWS modules and some best-practices - May 2019 (20)
Ad
More from Anton Babenko (12)
Ad
Recently uploaded (20)
Terraform AWS modules and some best-practices - May 2019
- 1. Terraform AWS modules and some best- practices Anton Babenko @antonbabenko May 2019
- 2. Anton Babenko Terraform AWS fanatic since 2015 Organiser of HashiCorp UG, AWS UG, DevOps Norway, DevOpsDays Oslo I 💚 open-source: terraform-community-modules + terraform-aws-modules antonbabenko/pre-commit-terraform — clean code and documentation antonbabenko/tfvars-annotations — update terraform.tfvars using annotations antonbabenko/modules.tf-lambda — generate Terraform code from visual diagrams antonbabenko/terragrunt-reference-architecture — Terragrunt reference architecture www.terraform-best-practices.com medium.com/@anton.babenko @antonbabenko — Twitter, GitHub, Linkedin
- 3. What do I do? All-things Terraform + AWS + DevOps Consulting Workshops Trainings Mentorship My email: [email protected] LinkedIn: https://www.linkedin.com/in/antonbabenko
- 4. Collection of open-source Terraform AWS modules supported by the community. More than 2 mil. downloads since September 2017. (VPC, Autoscaling, RDS, Security Groups, ELB, ALB, Redshift, SNS, SQS, IAM, EKS, ECS…) github.com/terraform-aws-modules registry.terraform.io/modules/terraform-aws-modules @antonbabenko
- 5. Cloudcraft.co — the best way to draw AWS diagrams @antonbabenko
- 6. cloudcraft.co features • Manage components in browser (EC2 instances, autoscaling groups, RDS, etc) • Connect components • Import live AWS infrastructure • Calculate the budget • Share link to a blueprint • Export as image • Embed drawing to wiki, Confluence, etc @antonbabenko
- 7. Infrastructure as code makes DevOps possible Key benefits: • Treat infrastructure like application code • Always know what changed • Validate infrastructure before deployment https://dzone.com/articles/infrastructure-as-code-the-benefits @antonbabenko
- 8. Tool for building, changing and versioning infrastructure safely and efficiently. www.terraform.io @antonbabenko
- 10. @antonbabenko
- 11. @antonbabenko
- 12. @antonbabenko
- 13. @antonbabenko
- 15. Google Cloud Deployment Manager Azure Resource Manager @antonbabenko
- 16. @antonbabenko
- 18. Why Terraform and not AWS CloudFormation, Azure ARM, Google Cloud Deployment Manager? Terraform manages 100+ providers, has easier syntax (HCL), has native support for modules and remote states, has teamwork related features, is an open-source project. Provides a high-level abstraction of infrastructure (IaC) Allows for composition and combination Orchestration, not merely configuration Supports parallel management of resources (graph, fast) Separates planning from execution (dry-run) @antonbabenko
- 19. Terraform — universal tool for everything with an API Google G Suite Dropbox files and access New Relic metrics Datadog users and metrics Jira issues Minecraft, or even order Domino’s pizza All Terraform providers — https://www.terraform.io/docs/providers/index.html @antonbabenko
- 20. Let’s begin — everything fits in main.tf @antonbabenko
- 21. @antonbabenko
- 22. Project grows — main.tf: 20+ resources and data sources @antonbabenko
- 24. Resources, regions, providers, … @antonbabenko
- 25. @antonbabenko
- 26. @antonbabenko
- 27. @antonbabenko
- 28. @antonbabenko
- 29. @antonbabenko
- 31. Emerging issues Code size is increasing Dependencies between resources become complicated @antonbabenko
- 32. Solution — Terraform modules @antonbabenko
- 33. Modules in Terraform are self-contained packages of Terraform configurations that are managed as a group. @antonbabenko
- 34. Resource modules Create resources in a very flexible configuration Open-source @antonbabenko
- 39. Would you use Terraform module to manage AWS EC2 security group? @antonbabenko
- 40. @antonbabenko
- 41. Would you use Terraform module to manage AWS EC2 security group? Yes :) @antonbabenko
- 42. Infrastructure modules Consist of resource modules Enforce tags and company standards Use preprocessors, jsonnet, cookiecutter @antonbabenko
- 46. @antonbabenko
- 47. Types of Terraform modules Resource modules (github.com/terraform-aws-modules , for eg) Infrastructure modules @antonbabenko
- 49. Tip №0 Check Terraform Registry before writing resource modules @antonbabenko
- 51. @antonbabenko
- 52. @antonbabenko
- 55. Things to avoid in modules @antonbabenko
- 56. Exception: logical providers (template, random, local, http, external) Providers in modules — evil @antonbabenko
- 57. @antonbabenko
- 58. Provisioner — evil Avoid provisioner in all resources @antonbabenko
- 59. Provisioner — evil Avoid provisioner in all resources @antonbabenko
- 60. Provisioner — evil Avoid provisioner even in EC2 resources @antonbabenko
- 61. Provisioner — evil Avoid provisioner even in EC2 resources @antonbabenko
- 62. @antonbabenko
- 63. @antonbabenko
- 64. null_resource provisioner — good @antonbabenko
- 65. Traits of good Terraform modules Documentation and examples Feature rich Sane defaults Clean code Tests Read more: http://bit.ly/common-traits-in-terraform-modules @antonbabenko
- 67. How to structure Terraform configurations? How to call them? @antonbabenko
- 68. Call Terraform modules Use Terraform modules, because amount of resources and code is increasing How to organize Terraform configurations and invoke them? How to orchestrate modules? @antonbabenko
- 69. All-in-one Good: Declare variables and outputs in fewer places Bad: Large blast radius Everything is blocked at once Impossible to specify dependencies between modules (depends_on) @antonbabenko
- 70. 1-in-1 Good: Smaller blast radius Possible to join invocation Easier and faster to work with Bad: Declare variables and outputs in more places @antonbabenko
- 71. Which way do you group your code? All-in-one or 1-in-1? @antonbabenko
- 73. Correct MFA (Most Frequent Answer): Somewhere in between @antonbabenko
- 74. All-in-one Undefined project scope Fast prototyping and initial development phase Small number of resources & developers Tightly connected resources 1-in-1 @antonbabenko Defined project scope Different types of developers involved * Code reuse is encouraged (across organization and environments) Use Terragrunt
- 75. What about Terraform workspaces? @antonbabenko
- 76. Problems with Terraform workspaces Terraform Workspaces aren’t infrastructure-as-code friendly. You can’t answer straight from the code: "How many workspaces do you have?" "What infrastructure has been deployed in workspaceX?" "What is the difference between workspaceX and workspaceY?" Introducing complexity almost in all cases. @antonbabenko
- 77. Solution — use re-usable modules instead of workspaces @antonbabenko
- 78. What kind of orchestration method do you use? -target Makefile … @antonbabenko
- 80. No really, do not try this at home! @antonbabenko
- 86. tfvars can’t contain dynamic values :( Orchestration = Terragrunt @antonbabenko subnet_id = "???"
- 87. Orchestration = Terragrunt tfvars can’t contain dynamic values, so I have fixed it :) @antonbabenko subnet_id = "???"
- 88. before_hook + shell script Go binary https://github.com/antonbabenko/tfvars-annotations — Update values in terraform.tfvars using annotations (WIP) or take a look at modules.tf @antonbabenko
- 89. Orchestration = Terragrunt @antonbabenko subnet_id = "" # tfvars:terragrunt_output.vpc.public_subnets
- 90. Orchestration = Terragrunt @antonbabenko subnet_id = ["subnet-1a2b3c4d"] # tfvars:terragrunt_output.vpc.public_subnets
- 92. Work with stateless lists @antonbabenko
- 93. @antonbabenko Work with stateless lists
- 94. @antonbabenko Work with stateless lists
- 95. https://jsonnet.org/ Work with stateful lists @antonbabenko
- 96. Work with stateful lists @antonbabenko
- 97. Work with stateful lists @antonbabenko
- 98. Work with stateful lists @antonbabenko
- 99. Work with stateful lists @antonbabenko
- 103. Edge cases Different AWS regions (version of S3 signature, EC2 ClassicLink, IPv6) Date of creation of AWS account Limits on resources in AWS Services and features availability @antonbabenko
- 104. Avoid in Terraform Not secret arguments should not be specified as command line arguments => put them in tfvars Reduce usage of "-target" and "-parallelism" "Terraform workspaces" evil in=> separate by directories Dependency hell in modules @antonbabenko
- 106. Summary Write less and simpler (Terraform 0.12 won’t fix your code for you!) Use existing modules and utilities @antonbabenko
- 107. How to handle secrets in Terraform? • Can you accept secrets to be saved in state file in plaintext? Probably not. • AWS IAM password & access secret keys — use PGP as keybase.io • AWS RDS — set dummy password and change after DB is created • AWS RDS — use iam_database_authentication_enabled = true • EC2 instance user-data + AWS KMS • EC2 instance user-data + AWS System Manager’s Parameter Store • AWS Secrets Manager • https://github.com/opencredo/terrahelp • Other options: • Secure remote state location (S3 bucket policy, KMS key) @antonbabenko
- 108. What are the tools/solutions out there? • Terraform Registry (https://registry.terraform.io/) — collection of public Terraform modules for common infrastructure configurations for any provider. • Terraform linter to detect errors that can not be detected by `terraform plan` — https://github.com/wata727/tflint • Terraform version manager — https://github.com/kamatama41/tfenv • A web dashboard to inspect Terraform States — https://github.com/camptocamp/ terraboard • Jsonnet — The data templating language — http://jsonnet.org @antonbabenko
- 109. Atlantis — Start working on Terraform as a team A unified workflow for collaborating on Terraform through GitHub, GitLab and Bitbucket https://www.runatlantis.io @antonbabenko
- 110. Bonus
- 115. ✓ cloudcraft.co — design, plan and visualize ✓ terraform-aws-modules — building blocks of AWS infrastructure ✓ Terraform — infrastructure as code
- 116. Infrastructure as code generator — from visual diagrams to Terraform https://github.com/antonbabenko/modules.tf-lambda Demo video: https://www.youtube.com/watch?v=F1Ax1zfZbiY
- 117. 1. Go to cloudcraft.co 2. Sign up, sign in (free account) 3. Draw your AWS infrastructure 4. Click "Export" 5. Click "Terraform code export" Try it yourself!
- 118. modules.tf — generated code ✓ Potentially ready-to-use Terraform configurations ✓ Suits best for bootstrapping ✓ Enforces Terraform best-practices ✓ Batteries included (terraform-aws-modules, terragrunt, tfvars- annotations, pre-commit) ✓ 100% free and open-source (https://github.com/antonbabenko/ modules.tf-lambda) ✓ Released under MIT license