Ad
More Related Content
What's hot (20)
Similar to Terraform Abstractions for Safety and Power (20)
Ad
Recently uploaded (20)
Ad
Terraform Abstractions for Safety and Power
- 1. Terraform abstractions for safety and power Calvin French-Owen, Co-founder Segment Hashiconf, Sept 2017 @calvinfo
- 2. Terraform at Segment - Analytics API for 1000s of online businesses - 349 services - 14k containers peak - 90B msg/month - 100k rps - All AWS - ECS (# containers running)
- 3. - 2.5 years of Terraform (since v0.4!) - ~30 developers interacting with Terraform weekly - 30-50 ‘applies’ per day - Tens of thousands of AWS resources Terraform at Segment
- 4. This Talk - Why is safety such a big deal? - Some Terraform ‘nouns’ - Safety with your state - Safety with your modules - Safety elsewhere
- 5. Why is ‘safety’ such a big deal?
- 7. Developers avoid selecting tools if the … effect of the tools is unknown, and the tools have some risks. To promote development support tools, we have to suppress the risk of the tools. - Analyzing the Decision Criteria of Software Based on Prospect Theory
- 10. Terraform can be scary… But doesn’t have to be
- 14. HCL: most examples configure terraform via HCL (hashicorp configuration language), a relative of JSON
- 15. variable: a dynamically configured input
- 17. resource: configuration for a given cloud entity (instance, load balancer, image, etc)
- 18. resources take inputs as configuration, and can produce outputs once they are created in your infrastructure. they may interpolate variables
- 23. module: a reusable collection of resources that can be passed its own inputs and produce outputs
- 25. plan: the diff between what exists in your infrastructure, and the changes you would like to apply
- 26. How does it know?
- 27. How does it know? .tfstate
- 30. Terraform Workflow $ terraform plan $ terraform apply
- 31. Terraform Workflow 1. load the desired configuration 2. load the stored .tfstate file 3. calculate the diff between the current and desired states 4. use CRUD APIs to update the current state to match the desired state 5. update the state file
- 32. Terraform applies diffs in your configuration to manage your infrastructure
- 33. Part I: State
- 34. Terraform wants to manage everything
- 35. Terraform wants to manage everything How do I keep it from destroying existing infrastructure?
- 36. AWS accounts dev stage prod old prodVPC peering
- 37. AWS accounts dev stage prod old prod terraform managed VPC peering left to rot and die sunset
- 38. The advantage of states per environment?
- 39. The advantage of states per environment? safety
- 42. The one golden rule: Always use remote state or backends
- 48. - Price? S3 or Consul - Custom configuration? S3 or Consul - Out-of-the-box dashboard + changelog? TFE - Remote applies? TFE - CI Integration? TFE - Versioning? Either (with tweaks) - Locking? Either! What remote state provider should I use?
- 49. - Price? S3 or Consul - Custom configuration? S3 or Consul - Out-of-the-box dashboard + changelog? TFE - Remote applies? TFE - CI Integration? TFE - Versioning? Either (with tweaks) - Locking? Either! (at Segment, we’ve used S3 but moved to TFE) What remote state provider should I use?
- 52. →readonly→
- 53. States per team
- 55. Many ways to manage read-only state
- 59. 2. data sources
- 64. module outputs
- 65. State Safety - Separate AWS (or GCP) accounts - A state per environment - Consider states per service or per team - We use per-team states - Use a remote state manager like TFE or S3 - Limit your blast radius - Use some sort of ‘read-only’ state - We use a combination of data sources and shared outputs
- 66. Part II: Modules
- 69. terracode terracode-integrations terracode-warehouses terracode-ops terracode-site Shared modules + variables per-team repositories source = “github.com/segmentio/terracode/modules/…”
- 70. We use and re-use modules liberally
- 74. Worker: input.tf
- 76. Worker: output.tf - render JSON
- 77. Worker – understandable defaults
- 79. Worker – heredoc task definition
- 80. Worker – autoscaling rules
- 81. Worker – IAM role
- 82. Worker – in practice
- 83. Worker – in practice source
- 84. Worker – in practice image
- 85. Worker – in practice cluster to run on
- 86. Worker – in practice ‘T-shirt size’ resources
- 87. Worker – in practice auto-scaling tweaks
- 88. Worker – in practice specific configuration
- 89. A last step: adding IAM permissions
- 90. Worker - adding IAM permissions
- 91. S3 bucket permissions under the hood
- 92. S3 bucket permissions under the hood
- 93. Worker - hiding the complexity
- 96. - Modules for logical ‘units’ of resources - Simple defaults to hide complexity - Variable all the things - If you write it more than twice, make it a module - Modules can reference across repos, share them - github.com/segmentio/terraform-docs - github.com/segmentio/stack Safety with modules
- 97. Safety elsewhere
- 99. What adoption can buy you
- 100. What if you have a common substrate for defining infrastructure and cloud services?
- 101. Alerting
- 103. three-line change; zero-cost alerting
- 104. Cost management
- 105. Cost management
- 106. We need to tie parts of our infrastructure to business lines...
- 107. Cost management
- 109. One last thought...
- 110. Terraform as the common substrate
- 112. When Terraform is your substrate, any cloud is your cloud
- 113. When Terraform is your substrate, any cloud product is your cloud product
- 114. Fin @calvinfo
Editor's Notes
- #2: Hi, I’m Calvin French-Owen, CTO and Co-founder of Segment. And today, I’d like to share a few ideas for using Terraform safely _and_ powerfully.