TXLF: Chef- Software Defined Infrastructure Today & Tomorrow
4 likes1,502 views
The open source configuration management and automation framework Chef is used to configure, deploy and manage infrastructure of every sort. In addition to managing Linux, Windows and many other operating systems; Chef may be used to manage network hardware and storage systems. This session will provide an overview of the concepts and capabilities of Chef and discuss upcoming projects and how they fit into the Chef ecosystem.
![package "apache2"
template "/etc/apache2/apache2.conf" do!
source "apache2.conf.erb"!
owner "root"!
group "root"!
mode "0644"!
variables(:allow_override => "All")!
notifies :reload, "service[apache2]"!
end
service "apache2" do!
action [:enable,:start]!
supports :reload => true!
end
Example Recipe](https://image.slidesharecdn.com/20140613-txlf-chefsoftwaredefinedinfrastructure-140614160031-phpapp02/85/TXLF-Chef-Software-Defined-Infrastructure-Today-Tomorrow-39-320.jpg)
![Run List
Node
Enterprise
Chef
chef-client
What policy should I follow?
"recipe[ntp::client]"
"recipe[users]"
"role[webserver]"
42](https://image.slidesharecdn.com/20140613-txlf-chefsoftwaredefinedinfrastructure-140614160031-phpapp02/85/TXLF-Chef-Software-Defined-Infrastructure-Today-Tomorrow-42-320.jpg)
![Run List
Enterprise
Chef
chef-client
What policy should I follow?
"recipe[ntp::client]"
"recipe[users]"
"role[webserver]"
43](https://image.slidesharecdn.com/20140613-txlf-chefsoftwaredefinedinfrastructure-140614160031-phpapp02/85/TXLF-Chef-Software-Defined-Infrastructure-Today-Tomorrow-43-320.jpg)
![Search for Nodes
pool_members = search("node","role:webserver")!
!
template "/etc/haproxy/haproxy.cfg" do!
source "haproxy-app_lb.cfg.erb"!
owner "root"!
group "root"!
mode 0644!
variables :pool_members => pool_members.uniq!
notifies :restart, "service[haproxy]"!
end
46](https://image.slidesharecdn.com/20140613-txlf-chefsoftwaredefinedinfrastructure-140614160031-phpapp02/85/TXLF-Chef-Software-Defined-Infrastructure-Today-Tomorrow-46-320.jpg)
![Search for Nodes
pool_members = search("node","role:webserver")!
!
template "/etc/haproxy/haproxy.cfg" do!
source "haproxy-app_lb.cfg.erb"!
owner "root"!
group "root"!
mode 0644!
variables :pool_members => pool_members.uniq!
notifies :restart, "service[haproxy]"!
end
47](https://image.slidesharecdn.com/20140613-txlf-chefsoftwaredefinedinfrastructure-140614160031-phpapp02/85/TXLF-Chef-Software-Defined-Infrastructure-Today-Tomorrow-47-320.jpg)
![Pass results into Templates
# Set up application listeners here.!
listen application 0.0.0.0:80!
balance roundrobin!
<% @pool_members.each do |member| -%>!
server <%= member[:hostname] %> <%= member[:ipaddress] %>:>
weight 1 maxconn 1 check!
<% end -%>!
<% if node["haproxy"]["enable_admin"] -%>!
listen admin 0.0.0.0:22002!
mode http!
stats uri /!
<% end -%>
48](https://image.slidesharecdn.com/20140613-txlf-chefsoftwaredefinedinfrastructure-140614160031-phpapp02/85/TXLF-Chef-Software-Defined-Infrastructure-Today-Tomorrow-48-320.jpg)
![Pass results into Templates
# Set up application listeners here.!
listen application 0.0.0.0:80!
balance roundrobin!
<% @pool_members.each do |member| -%>!
server <%= member[:hostname] %> <%= member[:ipaddress] %>:>
weight 1 maxconn 1 check!
<% end -%>!
<% if node["haproxy"]["enable_admin"] -%>!
listen admin 0.0.0.0:22002!
mode http!
stats uri /!
<% end -%>
49](https://image.slidesharecdn.com/20140613-txlf-chefsoftwaredefinedinfrastructure-140614160031-phpapp02/85/TXLF-Chef-Software-Defined-Infrastructure-Today-Tomorrow-49-320.jpg)
![# Set up application listeners here.!
listen application 0.0.0.0:80!
balance roundrobin!
<% @pool_members.each do |member| -%>!
server <%= member[:hostname] %> <%= member[:ipaddress] %>:>
weight 1 maxconn 1 check!
<% end -%>!
<% if node["haproxy"]["enable_admin"] -%>!
listen admin 0.0.0.0:22002!
mode http!
stats uri /!
<% end -%>
Pass results into Templates
50](https://image.slidesharecdn.com/20140613-txlf-chefsoftwaredefinedinfrastructure-140614160031-phpapp02/85/TXLF-Chef-Software-Defined-Infrastructure-Today-Tomorrow-50-320.jpg)
Ad
More Related Content
What's hot (20)
Viewers also liked (18)
Ad
Similar to TXLF: Chef- Software Defined Infrastructure Today & Tomorrow (20)
Ad
More from Matt Ray (20)
Recently uploaded (20)
TXLF: Chef- Software Defined Infrastructure Today & Tomorrow
- 2. Chef: Software Defined Infrastructure Today & Tomorrow Matt Ray TXLF June 14, 2014
- 3. Introductions • Matt Ray • Director of Partner Integration at Chef • [email protected] • mattray GitHub|IRC|Twitter
- 5. Items of Manipulation (Resources) • Networking • Files • Directories • Symlinks • Mounts • Registry Keys • Powershell Scripts • Users • Groups • Packages • Services • Filesystems • and so much more 5
- 6. Application A tale of growth... 6
- 7. Application Application Database Add a database 7
- 8. Application App Databases Make database redundant 8
- 9. App Servers App Databases Application server redundancy 9
- 10. App LB App Servers App Databases Add a load balancer 10
- 11. App LBs App Servers App Databases Webscale! 11
- 12. App LBs App Servers App DB Cache App DBs Now we need a caching layer 12
- 13. App LBs App Servers App DB Cache App DBs Infrastructure has a Topology 13
- 14. Round Robin DNS App Servers App DB Cache App DBs Floating IP? Your Infrastructure is a Snowflake 14
- 15. App LBs App Servers < Shiny! DB slaves Cache DB Cache DBs Complexity Increases Quickly Are we monitoring?? 15
- 16. App LBs App Servers DB slaves Cache DB Cache DBs ...and change happens! 16
- 17. App LBs App Servers DB slaves Cache DB Cache DBs ...and change happens! Add a Central Log Host Central Log Host 17
- 18. App LBs App Servers DB slaves Cache DB Cache DBs ...and change happens! Add a Central Log Host! ! Update syslog.conf on all Nodes Central Log Host 18
- 19. Chef Solves This Problem • But you already guessed that, didn’t you? 19
- 20. Chef is Infrastructure as Code • Programmatically provision and configure components http://www.flickr.com/photos/louisb/4555295187/ 20
- 21. Chef is Infrastructure as Code • Treat like any other code base http://www.flickr.com/photos/louisb/4555295187/ 21
- 22. Chef is Infrastructure as Code • Reconstruct business from code repository, data backups, and compute resources http://www.flickr.com/photos/louisb/4555295187/ 22
- 23. Chef is Infrastructure as Code • Programmatically provision and configure components • Treat like any other code base • Reconstruct business from code repository, data backup, and compute resourceshttp://www.flickr.com/photos/louisb/4555295187/ 23
- 24. Configuration Code • Chef ensures each Node complies with the policy • Policy is determined by the configurations in each Node’s run list • Reduce management complexity through abstraction • Store the configuration of your infrastructure in version control 24
- 25. Declarative Interface to Resources • You define the policy in your Chef configuration • Your policy states what state each resource should be in, but not how to get there • Chef-client will pull the policy from the Chef Server and enforce the policy on the Node 25
- 26. How does it work? http://i3.kym-cdn.com/photos/images/original/000/046/123/magnets.jpg
- 27. Managing Complexity • Organizations • Environments • Roles • Nodes • Recipes • Cookbooks • Search 27
- 28. Their Infrastructure Organizations My Infrastructure Your Infrastructure 28
- 30. Environments Define Policy • Environments may include data attributes necessary for configuring your infrastructure, e.g. • The URL of your payment service’s API • The location of your package repository • The version of the Chef configuration files that should be used 30
- 31. Roles Load Balancers Application Servers DB Cache Database 31
- 32. Roles Define Policy • Roles may include an ordered list of Chef configuration files that should be applied • This list is called a Run List • Order is always important in the Run List • Roles may include data attributes necessary for configuring your infrastructure, for example: • The port that the application server listens on • A list of applications that should be deployed 32
- 33. Nodes 33
- 34. Node • Each Node will • Belong to one Organization • Belong to one Environment • Have zero or more Roles 34
- 35. Nodes Adhere to Policy • The chef-client application runs on each node, which • Gathers the current system configuration of the node • Downloads the desired system configuration policies from the Chef server for that node • Configures the node such that it adheres to those policies 35
- 36. Resources • A Resource represents a piece of the system and its desired state • A package that should be installed • A service that should be running • A file that should be generated • A cron job that should be configured • A user that should be managed • and more 36
- 37. Resources in Recipes • Resources are the fundamental building blocks of Chef configuration • Resources are gathered into Recipes • Recipes ensure the system is in the desired state 37
- 38. Recipes • Configuration files that describe resources and their desired state • Recipes can: • Install and configure software components • Manage files • Deploy applications • Execute other recipes • and more 38
- 39. package "apache2" template "/etc/apache2/apache2.conf" do! source "apache2.conf.erb"! owner "root"! group "root"! mode "0644"! variables(:allow_override => "All")! notifies :reload, "service[apache2]"! end service "apache2" do! action [:enable,:start]! supports :reload => true! end Example Recipe
- 40. Cookbooks • Recipes are stored in Cookbooks • Cookbooks contain recipes, templates, files, custom resources, etc • Code re-use and modularity http://www.flickr.com/photos/shutterhacks/4474421855/ 40
- 41. Run List Node Enterprise Chef chef-client What policy should I follow? 41
- 42. Run List Node Enterprise Chef chef-client What policy should I follow? "recipe[ntp::client]" "recipe[users]" "role[webserver]" 42
- 43. Run List Enterprise Chef chef-client What policy should I follow? "recipe[ntp::client]" "recipe[users]" "role[webserver]" 43
- 44. Run List Specifies Policy • The Run List is an ordered collection of policies that the Node should follow • Chef-client obtains the Run List from the Chef Server • Chef-client ensures the Node complies with the policy in the Run List 44
- 45. Search • Search for nodes with Roles • Find Topology Data ! • IP addresses • Hostnames • FQDNs http://www.flickr.com/photos/kathycsus/268677262545
- 46. Search for Nodes pool_members = search("node","role:webserver")! ! template "/etc/haproxy/haproxy.cfg" do! source "haproxy-app_lb.cfg.erb"! owner "root"! group "root"! mode 0644! variables :pool_members => pool_members.uniq! notifies :restart, "service[haproxy]"! end 46
- 47. Search for Nodes pool_members = search("node","role:webserver")! ! template "/etc/haproxy/haproxy.cfg" do! source "haproxy-app_lb.cfg.erb"! owner "root"! group "root"! mode 0644! variables :pool_members => pool_members.uniq! notifies :restart, "service[haproxy]"! end 47
- 48. Pass results into Templates # Set up application listeners here.! listen application 0.0.0.0:80! balance roundrobin! <% @pool_members.each do |member| -%>! server <%= member[:hostname] %> <%= member[:ipaddress] %>:> weight 1 maxconn 1 check! <% end -%>! <% if node["haproxy"]["enable_admin"] -%>! listen admin 0.0.0.0:22002! mode http! stats uri /! <% end -%> 48
- 49. Pass results into Templates # Set up application listeners here.! listen application 0.0.0.0:80! balance roundrobin! <% @pool_members.each do |member| -%>! server <%= member[:hostname] %> <%= member[:ipaddress] %>:> weight 1 maxconn 1 check! <% end -%>! <% if node["haproxy"]["enable_admin"] -%>! listen admin 0.0.0.0:22002! mode http! stats uri /! <% end -%> 49
- 50. # Set up application listeners here.! listen application 0.0.0.0:80! balance roundrobin! <% @pool_members.each do |member| -%>! server <%= member[:hostname] %> <%= member[:ipaddress] %>:> weight 1 maxconn 1 check! <% end -%>! <% if node["haproxy"]["enable_admin"] -%>! listen admin 0.0.0.0:22002! mode http! stats uri /! <% end -%> Pass results into Templates 50
- 51. Jboss App Memcache Postgres Slaves Postgres Master NagiosGraphite So when this... 51
- 52. Jboss App Memcache Postgres Slaves Postgres Master NagiosGraphite ...becomes this 52
- 53. Memcache Postgres Slaves Postgres Master NagiosGraphite ! ! ! ! ! Jboss App Memcache Postgres Slaves Postgres Master NagiosGraphite ...this can happen automatically 53
- 54. NagiosGraphite ! ! ! ! ! NagiosGraphite Memcache Postgres Slaves • Load balancer config • Nagios host ping • Nagios host ssh • Nagios host HTTP • Nagios host app health • Graphite CPU • Graphite Memory • Graphite Disk • Graphite SNMP • Memcache firewall • Postgres firewall Postgres authZ config • 12+ resource changes for 1 node addition Count the Resources Jboss App 54
- 55. Which Operating Systems? • Many supported platforms and architectures • Relatively easy to port • Omnibus-Chef • AIX, Arch, Fedora, Gentoo, OmniOS, OpenBSD, Rasbian, SmartOS and more 55
- 56. The Chef Community • Apache License, Version 2.0 • Thousands of Individual and Corporate contributors. • Thousands of cookbooks available from the community • http://community.opscode.com
- 57. The Chef API and Server • HTTPS, RESTful API w/ JSON, RSA key auth • Infrastructure data store such as node data • Environments • Search Service • Data bags • SSH and Push jobs http://www.flickr.com/photos/core-materials/4419853626/sizes/o/in/photostream/
- 58. knife
- 59. knife - with the Chef Server • knife node • create / edit / delete • list • knife cookbook ... • knife role ... • knife environment ... 59
- 60. knife - with clouds • knife azure • knife cloudstack • knife do • knife ec2 • knife gce • knife hp • knife openstack • knife rackspace • knife vcloud • knife vsphere • ...and many others 60
- 61. Virtualization and Containers? • Docker • Hyper-V • KVM • LXC • OpenVZ • Vagrant • Virtualbox • VMware • Xen • Zones 61
- 62. Test Kitchen • Integration testing for your infrastructure code • Tests your cookbooks on all the supported platforms with “real” machines 62
- 63. ™ Desktop, Virtualization & Cloud • Vagrant • VMware • CloudStack • Eucalyptus • OpenStack • bare metal • AWS • Rackspace • HP • Google • Azure • many others
- 64. • AWS • Rackspace • HP • Google • Azure • many others Desktop, Virtualization & Cloud • Vagrant • VMware • CloudStack • Eucalyptus • OpenStack • bare metal ™
- 65. How about Storage? • Ceph • EMC • Gluster • NetApp • Nexenta • and more 65
- 66. What about Networking? • Routers and switches • Load balancers • Firewalls • SDN 66
- 68. Chef Metal • Chef recipes for deploying infrastructure • Libraries for repeatably creating machines and deployments with Chef primitives • Bootstrappers for many infrastructure types • Provisioner nodes, remote command execution 68
- 69. Chef Metal: Providers • Cloud • Digital Ocean, EC2, Fog, OpenStack • Virtualization • Vagrant (VirtualBox, Fusion), vSphere • Containers • Docker & LXC • SSH • PXE in progress 69
- 70. Chef Metal: Example Recipe machine 'mario' do! recipe 'mydb'! tag 'mydb_master'! end! num_webservers = 1! 1.upto(num_webservers) do |i|! machine "luigi#{i}" do! recipe 'mywebapp'! end! end 70
- 71. What does this all mean? •Every infrastructure is a unique snowflake •Infrastructure as Code brings transparency and traceability •Test your deployments at every stage •Use the same infrastructure code for wherever you want to deploy ™
- 72. Austin, Texas • Lots of Chef users in Austin • Austin Chef Meetup • Wednesday June 18, Maudies Triangle 8:30-10:30am • Austin DevOps Meetup • www.meetup.com/austin-devops/ • Agile Austin DevOps • Cloud, Docker, OpenStack, etc.. 72