Integrating Puppet with Cloud Infrastructures-Remco Overdijk
2 likes1,036 views
This document discusses automating cloud infrastructure using Puppet. It begins by describing issues with traditional single server infrastructure like limited scalability and redundancy. It then introduces using tools like AWS, Puppet, and Terraform to provision infrastructure in the cloud with improved scalability, isolation, and zero-downtime deployments. It discusses using Puppet and Terraform to define and provision AWS resources declaratively. It also covers bootstrapping Puppet onto new instances using techniques like autosigning, ENCs, Hiera lookups, AWS user data, and Cloud-init to automate configuration. The document concludes with a demonstration of provisioning a stack of web servers on AWS using Terraform and Puppet.
![[OpenInfra Days Korea 2018] Day 2 - E5-1: "Invited Talk: Kubicorn - Building ...](https://cdn.slidesharecdn.com/ss_thumbnails/e60955buildingsimplekubernetes-180705043459-thumbnail.jpg?width=560&fit=bounds)
Ad
More Related Content
What's hot (20)
Similar to Integrating Puppet with Cloud Infrastructures-Remco Overdijk (20)
Ad
Recently uploaded (20)
Ad
Integrating Puppet with Cloud Infrastructures-Remco Overdijk
- 1. Remco Overdijk LEAD OPERATIONS ENGINEER Automating the Cloud Integrating Puppet with Cloud Infrastructures @MaxServ @RemzJay
- 3. 3AUTOMATING THE CLOUD Traditional Website Hosting MySQL
- 4. AUTOMATING THE CLOUD Single Server Infrastructure Issues PROBLEM Limited headroom Service issues affect other services One outage means downtime Maintenance during deployment 4
- 5. Image credit: https://commons.wikimedia.org/wiki/File:Grumpy-Cat.jpg - Rjommabolli (CC 4.0)
- 6. AUTOMATING THE CLOUD Single Server Infrastructure Issues 6 Scalability Service Isolation Redundancy Zero-Downtime Deployments SOLUTIONPROBLEM Limited headroom Affected services One outage means downtime Maintenance during deployment
- 7. • AWS specific, but applies to most (if not any) Cloud platforms. AUTOMATING THE CLOUD Scope 7 • LAMP stack, but works for most stacks. • Mix & match for best results. • The method that works best depends on your own setup. • Based on Puppet Open Source. • Things may be different in Puppet Enterprise (Orchestrator). • Ready-to-go AMI’s may be faster to launch, but harder to maintain.
- 8. AUTOMATING THE CLOUD AWS Infrastructure Principles Read more https://media.amazonwebservices.com/AWS_Cloud_Best_Practices.pdf 8 • Infrastructure is failure-prone; Service robustness is achieved through redundancy. • EC2 instances should be considered volatile. • Scaling should be both vertical and horizontal. • Legacy applications aren’t magically cloud-ready. • Loose coupling improves scalability. • Isolation increases security and decreases dependencies.
- 9. AUTOMATING THE CLOUD AWS Infra: A lot of moving parts 9
- 11. • IAM Server Certificates • IAM Instance Profiles • IAM Roles • IAM Policies • IAM Role Policies • CodeDeploy Apps • CodeDeploy Deployment Groups • EC2 Instances • EC2 Elastic IP’s • CloudWatch Metrics • CloudWatch Alarms • EIP Attachments • ElastiCache Subnet Groups Provisioning Infrastructure AWS Resources • ElastiCache Clusters • ElastiCache Parameter Groups • EC2 Elastic Load Balancers • ELB Health Checks • ELB App Cookie Stickiness Policies • Key Pairs • RDS Subnet Groups • RDS Parameter Groups • RDS DB Instances • Route53 Zones • Route53 Records • S3 Buckets • S3 Policies • S3 CORS Configuration • Security Groups • SNS Topics • SNS Topic Subscriptions • VPC’s • VPC Subnets • VPC Internet Gateways • VPC (S3) Endpoints • VPC Route Tables • VPC Customer Gateways • VPC VPN Gateways • VPC DHCP Option Sets • VPC VPN Routes 11
- 13. Provisioning Infrastructure That’s a lot of clicking 13 • Manual changes are extremely error-prone. • Manual changes result in an inconsistent platform. • Collaboration is difficult; People are scared to break things. • Changes are scattered throughout the AWS console.
- 15. • Uses the same DSL as your ”regular” Puppet code. Provisioning Infrastructure Puppetlabs-AWS 15 • Most benefits from ”regular” Puppet; like relationships, defined types and the graph. • At the moment this module only supports a few of the resources in the AWS API. • Does NOT run using agents / puppet masters, but manually from your workstation using `puppet apply`. • Not as idempotent as you’d like at times. Read more https://github.com/puppetlabs/puppetlabs-aws https://forge.puppetlabs.com/puppetlabs/aws
- 16. `AWS_PROFILE=my-aws-account AWS_REGION=eu-west-1 puppet apply aws-stack.pp --templatedir ./templates/`
- 17. Provisioning Infrastructure Hashicorp Terraform • Uses HCL, which looks a lot like Puppet DSL. • Maintains a dependency graph, just like Puppet. • Runs from your workstation. • Is not AWS API feature complete, but covers most services. • Allows flexible scaling and destruction. 17 Read more https://github.com/hashicorp/terraform https://www.terraform.io/
- 19. View & try the full source https://github.com/MaxServ/Terraform-Puppet-Demo/tree/master/Terraform
- 20. Provisioning Infrastructure Puppet? 20 • Automated Infrastructure is awesome. • We like Puppet too. • Empty EC2 instances don’t serve websites. • How do we connect Puppet to the infrastructure we’ve just provisioned?
- 22. Bootstrapping Puppet Machine Configuration States 22
- 23. SLIDE CREDIT: Tim Bell, CERN – PuppetConf 2012 https://speakerdeck.com/puppetlabs/cern-accelerating-science-with-puppet https://youtu.be/-Ykb2j2ojYU?t=19m33s
- 24. Image credit: Geert Orye, https://flic.kr/p/7zDqXY (CC BY 2.0)
- 25. Bootstrapping Puppet So, Cattle. • Configuration should be at type level. Not node level. • Nodes should be replaceable. • Data on the nodes should be considered volatile. • Only versioned and cached content should be present. 25 • Provisioning should be automated; No manual intervention should be required.
- 26. Bootstrapping Puppet Puppet: Autosigning • Automates indoctrination for new nodes. • Multiple mechanisms available: Naïve, Basic and Policy-based Autosigning. 26 View & try https://github.com/MaxServ/Terraform-Puppet-Demo/blob/master/Puppet/puppet.conf Read more https://docs.puppetlabs.com/puppet/latest/reference/ssl_autosign.html Be very careful with naïve autosigning. Don’t do it in production without strict firewalls.
- 28. Bootstrapping Puppet Node Manifests using Regex 28 Read more https://docs.puppetlabs.com/puppet/latest/reference/lang_node_definitions.html#regular-expression-names
- 29. Bootstrapping Puppet Hiera • Configuration at `type` level instead of `clientcert`. • Use node-level overrides when required. • Combine with host `%H` mount points for master/slave. 29 View & try https://github.com/MaxServ/Terraform-Puppet-Demo/tree/master/Puppet/hieradata Read more https://docs.puppetlabs.com/hiera/latest/ https://docs.puppetlabs.com/guides/file_serving.html#file-server-configuration
- 31. Bootstrapping Puppet ENC’s in Foreman 31 Read more https://docs.puppetlabs.com/guides/external_nodes.html http://theforeman.org/manuals/1.10/index.html#2.Quickstart
- 32. Bootstrapping Puppet Foreman: Default Host Groups • The same idea as Autosigning in Puppet. • Uses a Foreman Plugin. • Use facts for assigning host groups. 32 Read more https://github.com/theforeman/foreman_default_hostgroup
- 33. Bootstrapping Puppet Foreman: Default Host Groups 33
- 34. Bootstrapping Puppet Assigning Configuration: Facter 34 View & try https://github.com/MaxServ/Terraform-Puppet-Demo/blob/master/Terraform/templates/userdata.tpl https://github.com/MaxServ/Terraform-Puppet-Demo/blob/master/Puppet/Vagrantfile Read more https://docs.puppetlabs.com/facter/2.4/custom_facts.html#structured-data-facts
- 35. Bootstrapping Puppet AWS: User data • User data allows you to pass configuration data to an instance during launch. • User data can be used to provide cloud-init configuration. 35
- 36. Bootstrapping Puppet AWS: User data 36
- 37. Bootstrapping Puppet Cloud-init 37 • Handles early initialization of cloud instances. • Supports EC2, CloudStack, OpenStack, OnApp, OpenNebula, RHEVm, vSphere & more. • Can install packages, do basic config and more. • Is able to bootstrap Puppet. Read more http://cloudinit.readthedocs.org/en/latest/index.html http://cloudinit.readthedocs.org/en/latest/topics/examples.html#setup-and-run-puppet
- 38. Bootstrapping Puppet Cloud-init with Puppetlabs-Aws 38 Read more https://docs.puppetlabs.com/puppet/latest/reference/lang_template_erb.html
- 39. Bootstrapping Puppet Cloud-init with Terraform 39 View & try https://github.com/MaxServ/Terraform-Puppet-Demo/blob/master/Terraform/ec2.tf Read more https://docs.puppetlabs.com/puppet/latest/reference/lang_template_erb.html
- 40. Bootstrapping Puppet Don’t forget to clean up! • Revoke Puppet-CA certificates for decommissioned nodes. • Clean up Salt keys. • Remove old reports, exported resources and catalogs from PuppetDB. • Clean connected resources like Load Balancers. 40 Sloppiness will catch up with you when it hurts the most.
- 41. Demo
- 42. Demo Terraform & Puppet THIS DEMONSTRATION INCLUDES: VIEW & TRY THE FULL SOURCE OF THIS DEMO https://github.com/MaxServ/Terraform-Puppet-Demo A B C Docker container running a Puppetmaster. AWS Stack with 4 webservers using Terraform. Webserver provisioning using Puppet based on type. 42
- 44. 44
- 45. 0416 - 30 10 00 Remco Overdijk LEAD OPERATIONS ENGINEER [email protected] Questions? @MaxServ @RemzJay