Terraforming
1 like•1,335 views
The document discusses the process of using Terraform in infrastructure projects, including the challenges of bootstrapping and managing complex project structures. It highlights the importance of modules for abstraction, reuse, and reducing complexity, and emphasizes creating a streamlined pipeline for new team members. The author also mentions leveraging serverless applications and community resources to enhance Terraform usage.
Terraforming
- 2. ! コロンビア
- 3. first time at JAWS
- 4. Tokyo
- 5. until recently Data Engineering / NLP
- 7. ..Ops + Golang + Python..
- 8. first week..started looking at projects..
- 9. first week..started looking at projects.. Github repos…
- 10. first week..started looking at projects.. Github repos… New project I had to work on…
- 13. code
- 16. lots of small services
- 19. early adopters of terraform
- 21. many different project structures
- 22. bootstrapping a project is also tough
- 23. where do I run terraform?
- 24. what role should run terraform? where do I run terraform?
- 25. what role should run terraform? where do I run terraform? ..credentials..
- 26. bootstrapping my project took longer than I wanted
- 36. api gateways
- 38. ..when it gets bigger tho..
- 42. many api gateways many lambdas many iam roles.. many everything..
- 43. many api gateways many lambdas many iam roles.. many everything.. even they are not exactly related to each other
- 44. I had to keep all pieces in my head at the same time
- 50. my_new_project/ tf /modules /stg main.tf .tfversion /prd main.tf .tfversion …just import modules..
- 51. module: should be like a function
- 52. function/module: purpose: does only one thing
- 53. module: parameters outputs inside is invisible to importer
- 55. module: abstracts complexity don’t need to know how things are connected
- 56. module: abstracts complexity don’t need to know how things are connected don't need to see all pieces
- 57. database_synchornizer: main.tf vars.tf output.tf readme.md lambda.tf iam.tf … module connect infrastructure together input parameters output parameters
- 59. database_synchornizer: main.tf vars.tf output.tf lambda.tf iam.tf … module all related less pieces
- 63. less to remember
- 64. less to remember other modules connect via output variables
- 65. we realise we could reuse modules
- 66. so we made our own GitHub repo only for modules, similar to: we realise we could share modules
- 67. github.com/terraform-community-modules github.com/segmentio/stack we realise we could reuse modules so we made our own GitHub repo only for modules, similar to:
- 68. ..looking at other projects..
- 69. ..looking at other projects.. ..how to run terraform..
- 70. what terraform version is this using?
- 71. what role should I assume to run terraform for this project?
- 72. ..Don’t forget to change environment..
- 73. project A: do this but don’t do that
- 74. project B: do this other thing but don’t do that other one
- 77. starting a new project?
- 78. starting a new project? difficult to bootstrap
- 81. bootstrapping should: take little effort be fast
- 82. bootstrapping should: take little effort be fast be easy for a new team member
- 83. bootstrapping should: take little effort be fast be easy for a new team member complexity should be hidden
- 84. what to do?
- 85. ..a module.. .. a terraform automated pipeline that takes 10 min to set up..
- 86. ..a trigger..
- 92. how do we use it?
- 93. 1. import module: terraform_pipeline 2. follow project structure
- 94. module “terraform_pipeline" { source = “git::git@:…//terraform_pipeline?ref=vx.y” notifications_channel = “slack channel” terraform_role_arn = “some::arn::role” env = "stg" name = “my_project" } importing this module
- 95. module “terraform_pipeline" { source = “git::git@:…//terraform_pipeline?ref=vx.y” notifications_channel = “slack-channel” terraform_role_arn = “some::arn::role” env = "stg" name = “my_project" } importing this module
- 96. module “terraform_pipeline" { source = “git::git@:…//terraform_pipeline?ref=vx.y” notifications_channel = “slack-channel” terraform_role_arn = “some::arn::role” env = "stg" name = “my_project" } importing this module
- 97. module “terraform_pipeline" { source = “git::git@:…//terraform_pipeline?ref=vx.y” notifications_channel = “slack-channel” terraform_role_arn = “some::arn::role” env = "stg" name = “my_project" } no need to remember this role anymore! importing this module
- 98. module “terraform_pipeline" { source = “git::git@:…//terraform_pipeline?ref=vx.y” notifications_channel = “slack-channel” terraform_role_arn = “some::arn::role” env = "stg" name = “my_project" } importing this module
- 99. module “terraform_pipeline" { source = “git::git@:…//terraform_pipeline?ref=vx.y” notifications_channel = “slack-channel” terraform_role_arn = “some::arn::role” env = "stg" name = “my_project" } importing this module
- 100. .. details are hidden for importer ..
- 101. .. details are hidden for importer .. importer does not need to worry about :
- 102. importer does not need to worry about : how to send notifications to slack .. details are hidden for importer ..
- 103. .. details are hidden for importer .. importer does not need to worry about : how to run terraform how to send notifications to slack
- 104. .. details are hidden for importer .. importer does not need to worry about : how to run terraform module knows that (inside) how to send notifications to slack
- 105. .. details are hidden for importer .. importer does not need to worry about : how to run terraform module knows that (inside) how to send notifications to slack a new team member can import it quickly
- 107. my_new_project/ tf /modules /stg .tfversion /prd .tfversion terraform ci module : 1. downloads terraform version specified in .tfversion 2. knows how to run a terraform binary 3. knows how to get git credentials to fetch to our private modules 4. knows how to send notifications to slack
- 108. my_new_project/ tf /modules /stg .tfversion /prd .tfversion want to run terraform for stg? go to stg folder
- 109. my_new_project/ tf /modules /stg .tfversion /prd .tfversion want to run terraform for stg? go to stg folder want to run terraform for prd? go to prd folder
- 114. reduced the complexity of bootstrapping a project
- 115. reduced the complexity of bootstrapping a project means a faster team
- 116. reduced the complexity of bootstrapping a project means a faster team also I am less scared to run terraform
- 117. reduced the complexity of bootstrapping a project means a faster team also I am less scared to run terraform all projects use the same pipeline
- 118. whats the next step to hide complexity?
- 119. whats the next step to hide complexity?
- 122. serverless serverless “applications” code + glue + infrastructure i.e: serverless service to get a slack bot via FaaS
- 123. serverless serverless “applications” > serverless install --url <service-github-url> > sls deploy code + glue + infrastructure i.e: serverless service to get a slack bot via FaaS
- 124. please be aware a few things have changed:
- 125. - Terraform community modules - Serverless application repository (aws)
- 126. how are you running terraform?
- 127. how are you structuring your terraform projects ? how are you running terraform?
- 128. how are you structuring your terraform projects ? how are you running terraform? my team is using terraform + github.com/serverless/serverless
- 129. how are you structuring your terraform projects ? how are you running terraform? my team is using terraform + github.com/serverless/serverless feel free to talk to me! I would like to learn more
- 130. Thanks 🙇 dav009 dav009 Q & A bit.ly/jd2018-sls