Puppet at Bazaarvoice
3 likes1,501 views
Dave gives an overview of his experience using Puppet at previous companies. He started using Puppet in 2008 at Bioware where it configured over 14,000 nodes across 5 datacenters. He now works at Bazaarvoice where he uses Puppet in embedded DevOps approaches. Puppet is used by application teams to manage their own operations without centralized operations. Dave also discusses how Puppet is used in different teams at Bazaarvoice, including using a master/client model focused on roles rather than hostnames.
![/etc/puppet/puppet.conf
#
File
managed
by
Puppet.
[main]
vardir
=
/var/lib/puppet
logdir
=
/var/log/puppet
rundir
=
/var/run/puppet
ssldir
=
$vardir/ssl
usecacheonfailure
=
true
pluginsync
=
true
factpath
=
$vardir/lib/facter
preferred_serializaYon_format
=
yaml
[user]
vardir
=
/var/lib/puppet
logdir
=
/var/log/puppet
rundir
=
/var/run/puppet
ssldir
=
$vardir/ssl
[agent]
runinterval
=
1800
ca_server
=
<%=
ca_srv
%>
server
=
<%=
logical_srv
%>
cerYficate_revocaYon
=
False
environment
=
<%=
environment
%>
report
=
true](https://image.slidesharecdn.com/puppetatbazaarvoice-130503152853-phpapp02/85/Puppet-at-Bazaarvoice-19-320.jpg)
![/etc/puppet/env/global_hieradata/
environments.json
{
"environments":
[
{
"cert":
[
{
"modulepath":
"/etc/puppet_env/cert/modules"
},
{
"manifestdir":
"/etc/puppet_env/cert/manifests"
},
{
"manifest":
"/etc/puppet_env/cert/manifests/site.pp"
}
]
}
}](https://image.slidesharecdn.com/puppetatbazaarvoice-130503152853-phpapp02/85/Puppet-at-Bazaarvoice-20-320.jpg)
![/etc/puppet/puppetmaster.conf
[main]
vardir
=
/var/lib/puppet
logdir
=
/var/log/puppet
rundir
=
/var/run/puppet
ssldir
=
$vardir/ssl
usecacheonfailure
=
true
pluginsync
=
true
factpath
=
$vardir/lib/facter
preferred_serializaYon_format
=
yaml
syslogfacility
=
local1
[master]
certname=<%=
certname
%>
ca
=
True
cerYficate_revocaYon=False
dns_alt_names=<%=
logical_srv
%>
ssl_client_header
=
SSL_CLIENT_S_DN
ssl_client_verify_header
=
SSL_CLIENT_VERIFY
autosign
=
true
#
For
puppet
dashboard
reporYng.
reports
=
store,
datadog_reports
<%
if
store_configs
==
true
%>
#
Puppetdb.
storeconfigs
=
true
storeconfigs_backend
=
puppetdb
<%
end
%>
[user]
vardir
=
/var/lib/puppet
logdir
=
/var/log/puppet
rundir
=
/var/run/puppet
ssldir
=
$vardir/ssl
#
Environments
<%
environments.each
do
|env_val|
-‐%>
<%
env_val.keys.each
do
|env_key|
-‐%>
[<%=
env_key
-‐%>]
<%
env_val[env_key].each
do
|env_data|
-‐%>
<%
env_data.each_pair
do
|k,
v|
-‐%>
<%=
k
%>
=
<%=
v
-‐%>
<%
end
%>
<%
end
%>
<%
end
%>
<%
end
%>](https://image.slidesharecdn.com/puppetatbazaarvoice-130503152853-phpapp02/85/Puppet-at-Bazaarvoice-21-320.jpg)
Puppet at Bazaarvoice
- 2. Hi, my name is Dave. • DevOps Engineer at Bazaarvoice. • Started working with puppet in 2008 while working at Bioware. First version was 0.24. • At Bioware, puppet configured over 14k nodes that comprised of web servers, databases and game servers. • All 5 datacenters (in California, Virginia, Ireland, Australia and Texas) housed puppet managed nodes that all reported back to a centralized puppet dashboard. • My contact info: – hTp://www.linkedin.com/in/jamesbarcelo
- 3. Bazaarvoice Plug • We do embedded DevOps! • ApplicaYon teams are responsible for their applicaYon’s operaYonal success. No centralized operaYons! • 2.0 stack is 100% in Amazon! • Conferences! • Work on awesome projects with spiffy tech like Cassandra or ElasYcSearch. • Send me your resume! [email protected]
- 4. Preview presentaYon • Puppet in the legacy stack. • Puppet in the Data Infrastructure Team. – Focus on client/server. • Puppet in the Data Services Team. – Focus on masterless puppet.
- 5. Puppet in the Legacy Infrastructure • TradiYonal puppet use with client/server. • MulYple levels of inheritance using node inheritance. • Puppet managed instances are configured according to DNS naming convenYon: node /my-‐hostname/ { … … }
- 6. • Some issues encountered: – Very hard to work with. Very complex. – Large codebase. Adds to complexity. MulYple teams working with same code base. – No confidence in making changes. Side effects feared ader code change. A jinga tower of puppet code. – Too many pivot points. Many places to configure. Adds to complexity. – Lots of code rot. Had not been refactored.
- 7. Puppet master/client in Data Infrastructure teams
- 8. Architecture • Each server type we care about will be referenced by its role. We only care about roles, not hostnames. • Centered around an uber IT tools server that runs everything ops (including puppet) to do work in an environment. The Mothership. • Hiera and parameterized classes will be used to create generic puppet modules that can be reused for different roles. • Development will be centered on using puppet environments on the Mothership to protect devs from stepping on each other.
- 9. Mothership • Contains a cocktail of different applicaYon tools for doing work in the environment. Tools included: – McollecYve/AcYveMQ – Puppet server – Puppet managed operaYon scripts. • Motherships configured to be highly available in regular AWS fashion(Autoscaling, cluster mulYple acYveMQ, etc). • AdverYses mulYple puppet environments that clients can switch between via –environments.
- 10. What is a Universe?
- 11. Anatomy of a Mothership
- 12. Methods of passing in data Gejng environment data into puppet configuraYon. • Hiera datastore. • Puppet stdlib/tags.txt • Cloud formaYon parameters – Universe, VPC
- 13. Puppet Stdlib/facts.d • Bootstrap process (EC2 user data) populates / etc/facter/facts.d/tags.txt with mappings. These mappings become facters. • Example of data in tags.txt: – Universe value. – Ec2 metadata. /etc/facter/facts.d/tags.txt: universe=dev Tag_region=us-‐east-‐1
- 14. Hiera datastore • Hiera is used extensively where different data needs to be passed into puppet according to context. Different contexts would include: – Amazon region. – Instance role. – Universe. • Example usage: $app_version = hiera(‘app_version’, nil)
- 15. /etc/hiera.yaml -‐-‐ :logger: console :hierarchy: -‐ %{fqdn} -‐ 10-‐team/%{team}/10-‐region/%{tag_region}/10-‐universe/%{universe}/10-‐roles/%{role} -‐ 10-‐team/%{team}/10-‐region/%{tag_region}/10-‐universe/%{universe}/20-‐common -‐ 10-‐team/%{team}/10-‐region/%{tag_region}/20-‐roles/%{role} -‐ 10-‐team/%{team}/10-‐region/%{tag_region}/30-‐common -‐ 10-‐team/%{team}/20-‐universe/%{universe}/10-‐roles/%{role} -‐ 10-‐team/%{team}/20-‐universe/%{universe}/20-‐common -‐ 10-‐team/%{team}/30-‐roles/%{role} -‐ 10-‐team/%{team}/40-‐common -‐ 40-‐common -‐ environments :backends: -‐ yaml -‐ json :yaml: :datadir: /etc/puppet_env/%{environment}/manifests/hieradata :json: :datadir: /etc/puppet/env/global_hieradata
- 18. Puppet code on the Mothership • The files that do the things: – /etc/puppet/puppet.conf – /etc/puppet/env/global_hieradata/ environments.json – /etc/puppet/puppetmaster.conf – /etc/puppet_env/{puppet_env}/… • manifests/… • modules/…
- 19. /etc/puppet/puppet.conf # File managed by Puppet. [main] vardir = /var/lib/puppet logdir = /var/log/puppet rundir = /var/run/puppet ssldir = $vardir/ssl usecacheonfailure = true pluginsync = true factpath = $vardir/lib/facter preferred_serializaYon_format = yaml [user] vardir = /var/lib/puppet logdir = /var/log/puppet rundir = /var/run/puppet ssldir = $vardir/ssl [agent] runinterval = 1800 ca_server = <%= ca_srv %> server = <%= logical_srv %> cerYficate_revocaYon = False environment = <%= environment %> report = true
- 20. /etc/puppet/env/global_hieradata/ environments.json { "environments": [ { "cert": [ { "modulepath": "/etc/puppet_env/cert/modules" }, { "manifestdir": "/etc/puppet_env/cert/manifests" }, { "manifest": "/etc/puppet_env/cert/manifests/site.pp" } ] } }
- 21. /etc/puppet/puppetmaster.conf [main] vardir = /var/lib/puppet logdir = /var/log/puppet rundir = /var/run/puppet ssldir = $vardir/ssl usecacheonfailure = true pluginsync = true factpath = $vardir/lib/facter preferred_serializaYon_format = yaml syslogfacility = local1 [master] certname=<%= certname %> ca = True cerYficate_revocaYon=False dns_alt_names=<%= logical_srv %> ssl_client_header = SSL_CLIENT_S_DN ssl_client_verify_header = SSL_CLIENT_VERIFY autosign = true # For puppet dashboard reporYng. reports = store, datadog_reports <% if store_configs == true %> # Puppetdb. storeconfigs = true storeconfigs_backend = puppetdb <% end %> [user] vardir = /var/lib/puppet logdir = /var/log/puppet rundir = /var/run/puppet ssldir = $vardir/ssl # Environments <% environments.each do |env_val| -‐%> <% env_val.keys.each do |env_key| -‐%> [<%= env_key -‐%>] <% env_val[env_key].each do |env_data| -‐%> <% env_data.each_pair do |k, v| -‐%> <%= k %> = <%= v -‐%> <% end %> <% end %> <% end %> <% end %>
- 22. /etc/puppet_env/{env}/manifests/ site.pp import 'roles/*.pp' node default { class { 'basenode_role': } class { "$tag_role": } }
- 23. /etc/puppet_env/{env}/manifests/ role/00_basenode.pp class basenode_role { class { ‘security’: } class { ‘monitoring’: } ….. }
- 24. /etc/puppet_env/{env}/manifests/ role/mothership_role.pp class mothership_role { class { 'puppet': master => true, ca_srv => $tag_caserver, logical_srv => $tag_puppet_server, } }
- 26. Masterless Puppet in Data Services Teams
- 27. Architecture SYll keeping bits of the Mothership project: • ApplicaYons/Services scoped in zookeeper by Universe. • Emphasis is put on making things simpler. Puppet code will not be monolithic. Individual applicaYon teams will only need to maintain there own modules/manifests. • Changes to modules/manifests will not impact other teams.
- 28. Methods of passing in data • The usual suspects: – Puppet stdlib/tags.txt. – Hiera. – Cloud formaYon parameters – Universe, VPC • Some new ones: – EC2 data/metadata -‐> facter. – Zookeeper. – Cloud formaYon parameters -‐ DeployTag
- 29. getEC2data_cache.rb • Script runs out of /etc/facts.d that converts EC2 data/metadata into facts.
- 30. Zookeeper/Ostrich • Custom funcYons to pull data from zookeeper the same way applicaYons do discovery.
- 32. Puppet code in Masterless • No more Mothership. All work is done via puppet apply. – /etc/hiera.yaml – /etc/puppet/manifests/{role}.pp – /etc/puppet/manifests/00_common.pp – /etc/puppet/manifests/01_users.pp – /etc/puppet/modules/…
- 33. /etc/hiera.yaml -‐-‐ :logger: console :hierarchy: -‐ %{fqdn} -‐ 10-‐universe/%{universe}/10-‐roles/%{role} -‐ 10-‐universe/%{universe}/20-‐common -‐ 20-‐roles/%{role} -‐ 30-‐common :backends: -‐ yaml :yaml: :datadir: /etc/puppet/manifests/hieradata
- 34. /etc/puppet/manifests/{role}.pp import '00_common' node default { # This class contains common modules that should be used by all roles. class { 'common': } class { 'acYvemq’: } -‐> class { 'mcollecYve': server => true, client => true, } }
- 35. /etc/puppet/manifests/ 00_common.pp import '01_users' ##################################### # Common ##################################### class common { class { 'stdlib': } file { '/opt/bazaarvoice': ensure => directory, } # Authorized keys for project developers. class { 'user_setup': stage => setup, } host { 'internal_ip': ensure => 'present', name => $fqdn, ip => $ipaddress, } class { 'prompt': } }
- 36. /etc/puppet/manifests/01_users.pp class user_setup { include users users::user { 'dbarcelo': groups => 'wheel', sshKey => 'ssh-‐dss AAAAB3NzaC1kc3MAAACBANL1zoZdYJp/6vQ4G5iNQXjdJ7NGmK0J2eqHbztvuD0CBPyqMuEtuYKRg14tFd4iwp5EpnT4UWpv8kXF/ dkEN3b5xgN/R+1hYq7/3mnRLchMFTl0tyryLuARC9zTI003mQrXd/ W9jzXaNlCTpxh8Ihj2Ov3lvAAX65tN9nijxhCTAAAAFQCgMU0obmTLo5CRYtRwDCkj1mb2hQAAAIAiZF9axkCvMa9vwigDiAf3rNMbut1gtqtwdzux8c9T1inApKV5 sccjg5POKm+4WmWTBOtQfYR8cNot2Mn/mO+MRiKH8sYapYnU2es+KRBmhdARE+N7EqdD0WqoP7NrsNVbObHwDQBNkODuc3ZPyTQuqv/ w4poTXaS5u5M1XZbgZwAAAIEAjt4r7SN1I/m0V/ TvedTVxJvKln4wZkFxyI5CAgpsAr435kwSLM08R9Hd0/5Vy9L…YpH1aZTBaoTqmTCtnv3mp1coXoscEp5nE0llfm+4DX3YvWnR80S/OeMUe71Ucm1ORwFpST/ K4WKQoZ30TAVVsc8nYy2hyD7hyozjzsS09o= dave.barcelo@dbarcelo-‐mbpro' } users::user { 'lwadhwani': groups => 'wheel', sshKey => 'ssh-‐rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAysFCPpffw9LIOqAEFZxOOb52m2FbHhumBFc07o8sm3c4cmdLq/ bBtr5TyuQp89zVNEaTGRbw1nMpQCDno4i5ipTvCLoKkOE1PRdtyJw6PGu6VV/0U1ghK+1xmveM2jDX/otj5hjnQiRm1+Fx/ orYwNBkywDlDHZQCGxalWaFgXVyReCRUqq0jBwj3EKJfsQgoxuSrh7F6GjsQ6DUOsA3wBfewS25hPmhulEqvga4/P58BMHemL9d4Ugu98Vg7fgaur/b1adX +LzbmE6C2T4Gn1kzAOEct6bFgLPRj3n5/EaspdOsZ/Nnik0LUvIwZNHgDCLgkS0D8aMIsiUrB4OqSw== luveen@Pantalaimon' } }
- 37. /etc/puppet/modules/… • Do stuff! • Code is sYll generic but it does not have to be.