7 Tools for your Puppetized Devops stack

7 Tools for your puppetized
devops stack
Kris Buytaert
@krisbuytaert

Kris Buytaert

• I used to be a Dev,
• Then Became an Op
• Chief Trolling Officer and Open Source
Consultant @inuits.eu
• Everything is an effing DNS Problem

What's this devops thing
anyhow ?

World , 200X-2009
Patrick Debois, Gildas Le Nadan, Andrew Clay Shafer, Kris Buytaert, Jezz
Humble, Lindsay Holmwood, John Willis, Chris Read, Julian Simpson, Luke
Kanies, John Allspaw and lots of others ..

Gent , October 2009
Mountain View , June 2010

Hamburg , October 2010

Boston, March 2011

Mountain View, June 2011

Bangalore, Melbourne,

Goteborg , October 2011

Rome, Next week !
....

● Devops is a growing professional and cultural
movement
● We don't have all the answers yet
● We are reaching out to different communities
● We will point out problems we see..
● Only the name is new

While we are still working out the solutions

● Adopt the new philosophy. We are in a new economic age. Western management
must awaken to the challenge, must learn their responsibilities, and take on leadership
for change.
● Cease dependence on inspection to achieve quality. Eliminate the need for massive
inspection by building quality into the product in the first place.
● Improve constantly and forever the system of production and service, to improve
quality and productivity, and thus constantly decrease costs.
● Institute training on the job.
● Institute leadership The aim of supervision should be to help people and machines and
gadgets do a better job.
● Drive out fear, so that everyone may work effectively for the company.
● Break down barriers between departments. People in research, design, sales, and
production must work as a team, in order to foresee problems of production and
usage that may be encountered with the product or service.
● Eliminate slogans, exhortations, and targets for the work force asking for zero defects
and new levels of productivity. Such exhortations only create adversarial relationships,
as the bulk of the causes of low quality and low productivity belong to the system and
thus lie beyond the power of the work force.

●
Eliminate management by objective. Eliminate management by numbers
and numerical goals. Instead substitute with leadership.

●
Remove barriers that rob the hourly worker of his right to pride of
workmanship. The responsibility of supervisors must be changed from
sheer numbers to quality.

●
Remove barriers that rob people in management and in engineering of
their right to pride of workmanship.
● Institute a vigorous program of education and self-improvement.
● Put everybody in the company to work to accomplish the transformation. The
transformation is everybody's job.

William Edwards
Deming
1986, Out of the Crisis.

http://en.wikipedia.org/wiki/W._Edwards_Deming

CAMS
● Culture
● Automation
● Measurement
● Sharing

Damon Edwards and John Willis

Culture,
automation,

Measturement,

sharing

7 Tools for your Puppetized Devops stack

Breaking the Silos

Devs Ops Getting Along

Culture,

Automation,
Measurement,

Sharing

“Solved problems”
● Deployment automation :
●
Fai , kickstart, preseed, cobbler,
● Configuration Management
●
Puppet, chef, cfengine,

“challenges”

•
Build & Release Management
•
Package & Repository management
•
Orchestration

Jenkins
● Open Source Continuous Integration Server
● A zillion plugins (400)
● Puppet users

● Have developers build stable and deployable
code
● Test Puppet code

Jenkins
● https://github.com/UnifiedPost/puppet-jenkins
● https://github.com/rtyler/puppet-jenkins
● https://github.com/vStone/jenkins-puppet-
scripts

fpm
#packagingsucks

●
Missing upstream

●
Ancient upstream

●
Unneeded dependencies

●
Broken upstream

●
Distro Policies

● “Maximum RPM” ?

● Packaging is needed

●
Dependencies, tooling, repositories

● Anger driven development

fpm
fpm -t rpm -s dir -n hornetq -v 2.2.5 hornetq
Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.nNkVwh
+ umask 022
+ cd /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/BUILD
+ exit 0
Executing(%build): /bin/sh -e /var/tmp/rpm-tmp.yUd4MV
+ umask 022
+ tar -zxf /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/data.tar.gz
+ exit 0
Executing(%install): /bin/sh -e /var/tmp/rpm-tmp.jkpqeA
+ umask 022
+ /usr/lib/rpm/brp-compress
+ /usr/lib/rpm/brp-strip
+ /usr/lib/rpm/brp-strip-static-archive
+ /usr/lib/rpm/brp-strip-comment-note
Processing files: hornetq-2.2.5-1.x86_64
Checking for unpackaged file(s): /usr/lib/rpm/check-files /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/BUILD
Wrote: /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/SRPMS/hornetq-2.2.5-1.src.rpm
Wrote: /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/RPMS/x86_64/hornetq-2.2.5-1.x86_64.rpm
Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.z2UL3B
+ umask 022
+ rm -rf /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/BUILD
+ exit 0
Created /usr/local/hornetq-2.2.5.x86_64.rpm

fpm in action
● https://github.com/Inuits/build-gems
● Fork, pull
● Jenkins pulls , builds , pushes to repo
● (variants for Nagios Plugins / Jenkins Plugins
available)

The Marionette Collective
● Distributed ssh ++
● What version of ssh do I have installed on my
servers ?
● On what servers is XYZ running ?
● Clean al my ssl certs ?
● Restart apache on all servers with fact X

mc-service
mc-service httpd status -W /uat/

* [ ============================================================> ] 80 /
80

bleh.poc.com status=stopped
xs03.ta.google.com status=stopped
elasticsearch02.uat.com status=stopped
dbs01-.poc.com status=stopped
dino.uat.com status=stopped
dsp01.uat.com status=stopped
webserver01.uat.com status=running
dbs01-old.uat.com status=stopped
alexdb02-hdp.poc.com status=stopped
postfix.uat.com status=stopped
ns01.uat.com status=stopped
devtools03.uat.com status=running

mc-package
mc-package -W /dev/ status jdk
* [ ============================================================> ] 33 / 33

servicesdb01.dev.com version = -absent
services.dev.google.com version = jdk-1.6.0_13-fcs
drbdtest02.dev.google.com version = -absent
services3.dev.google.com version = jdk-1.6.0_20-fcs
um.dev.google.com version = jdk-1.5.0_19-fcs
devtools03.uat.com version = jdk-1.6.0_29-fcs
alexandria02.dev.google.com version = -absent
weblink01.dev.com version = -absent
wikitest.dev.google.com version = jdk-1.6.0_24-fcs
payment.dev.google.com version = jdk-1.5.0_17-fcs
tiff2pdf01.dev.com version = -absent
devdoos.dev.com version = jdk-1.6.0_30-fcs
wiki.dev.google.com version = jdk-1.6.0_24-fcs
reporting01.dev.com version = -absent
devtools01-dev.uat.com version = jdk-1.6.0_23-fcs
devtools02.uat.com version = jdk-1.6.0_29-fcs
drbdtest01.dev.google.com version = -absent

---- package agent summary ----
Nodes: 33/33
Versions: 1 * 1.5.0_17-fcs, 1 * 1.5.0_19-fcs, 1 * 1.6.0_13-fcs, 1 * 1.6.0_20-fcs, 1 * 1.6.0_23-fcs, 2 * 1.6.0_24-fcs, 2 * 1.6.0_29-fcs
Elapsed Time: 1.73 s

Culture,

Automation,

Measurement,
Sharing

#monitoringsucks
Monitoring is AWESOME. Metrics are AWESOME. I love it. Here's what I don't love:

● Having my hands tied with the model of host and service bindings.

● Having to set up "fake" hosts just to group arbitrary metrics together

● Having to either collect metrics twice - once for alerting and another for trending

● Only being able to see my metrics in 5 minute intervals

● Having to chose between shitty interface but great monitoring or shitty monitoring but
great interface

● Dealing with a monitoring system that thinks IT is the system of truth for my
environment

● Not actually having any real choices

John Vincent (@lusis) on his blog http://lusislog.blogspot.com/2011/06/why-
monitoring-sucks.html

A sub movement
● #monitoringsucks trending
● https://github.com/monitoringsucks/
● 2008 Study :Nagios + Friends
● 2011 Conclusion : Nagios/Icinga are the only
automatable alternatives
● Monitoring and trending at scale , new kids
Graphite, Icinga, flapjack, etc
● Old Cool Kids, Ganglia

Logstash
● Not your average centralized logging tool
● Elasticsearch backed

● Shipper
● Indexer
● Web

Logstash

● Collect from
anywhere
● Filter
● Send anywhere

● Queuing support

Puppet and Logstash
● https://github.com/KrisBuytaert/vagrant-
puppet-logstash
● Includes Logstash / Kibana / Grok / ES modules
as submodules !

● git clone
● git submodule init
● git submodule update
● vagrant up

Graphite
● Graphing at Scale
● Graphing at Ease
● Any metric is a graph

● echo "somestring $somevalue $timestamp" |
nc <%= graphitehost %> 2003

Friends of Graphite
● Collection :
●
Statsd,
●
Collectd + Carbonplugin
●
Jmxtrans
●
Logster
● Dashboards
●
Tattle
●
Gdash
●
...

Puppet and Graphite
● https://github.com/KrisBuytaert/vagrant-
graphite/
● Includes Graphite / Gdash / Jmxtrans /
Logster / Collectd / Statsd / Tattle and more
modules as submodules !

● git clone
● git submodule init
● git submodule update
● vagrant up

Culture,

Automation,

Measurement,

Sharing

● Open Source
● Github
● Talk about Experiences
● Open Spaces

Sharing environments

● Build identical environments
● Share code
● Shared ownership of content , code and
configuration

Vagrant
● Abstraction layer for VirtualBox
● Integrates well with Puppet/Chef
● Project =
●
Vagrantfile
●
Manifests / Cookbooks
● Portable, Small , Versionable
● Use veewee to build your boxen

Vagrantfile
Vagrant::Config.run do |config|
# All Vagrant configuration is done here. The most common configuration
# options are documented and commented below. For a complete reference,
# please see the online documentation at vagrantup.com.

config.vm.define :mongo1 do |mongo1_config|
mongo1_config.ssh.max_tries = 100
mongo1_config.vm.box = "MyCentOS2"
mongo1_config.vm.network("192.168.99.101")
mongo1_config.vm.host_name = "mongo1"
mongo1_config.vm.provision :puppet do |mongo1_puppet|
mongo1_puppet.pp_path = "/tmp/vagrant-puppet"
mongo1_puppet.manifests_path = "manifests"
mongo1_puppet.module_path = "modules"
mongo1_puppet.manifest_file = "site.pp"
end
end
config.vm.define :mongo2 do |mongo2_config|
mongo2_config.ssh.max_tries = 100
mongo2_config.vm.box = "MyCentOS2"
mongo2_config.vm.network("192.168.99.102")
mongo2_config.vm.host_name = "mongo2"
mongo2_config.vm.provision :puppet do |mongo2_puppet|
mongo2_puppet.pp_path = "/tmp/vagrant-puppet"
mongo2_puppet.manifests_path = "manifests"
mongo2_puppet.module_path = "modules"
mongo2_puppet.manifest_file = "site.pp"
end
end

Vagrant Rocks
● Vagrant init
● Vagrant up
● Vagrant provision
● Vagrant down
● Vagrant destroy

Quiz Time :
Which tool did I forget ?

Links
http://puppetlabs.com/mcollective/introduction/
https://github.com/jordansissel/fpm
http://logstash.net/
https://github.com/rashidkpc/Kibana
http://graphite.wikidot.com/
https://github.com/wayfair/Graphite-Tattle
http://vagrantup.com/
https://github.com/KrisBuytaert

Contact
Kris Buytaert
Kris.Buytaert@inuits.be

Further Reading
@krisbuytaert
http://www.krisbuytaert.be/blog/
http://www.inuits.be/

Inuits

Duboistraat 50
2060 Antwerpen
Belgium
891.514.231

+32 475 961221

7 Tools for your Puppetized Devops stack

Recommended

More Related Content

Viewers also liked (19)

Similar to 7 Tools for your Puppetized Devops stack (20)

More from Kris Buytaert (20)

Recently uploaded (20)

7 Tools for your Puppetized Devops stack