Chef Cookbook Testing and Continuous Integration

Chef Cookbook Testing and
Continuous Integration
Julian C. Dunn
Senior Consultant, Opscode, Inc.
<jdunn@opscode.com>
Friday, June 14, 13

Why Write Cookbook Tests?
• Catch development errors
• Catch regressions
• Code to requirements (TDD, BDD)
• Smaller batch sizes == lower blast radius
• Continuous delivery of cookbooks!
• Build confidence that your infrastructure is
working correctly
Friday, June 14, 13

Software Development
Lifecycle for
Sysadmins
Friday, June 14, 13

Devops Is A Two-Way Street
• It’s great when
developers care
about
• uptime!
• scaling!
• deployment!
• Put them on call!
etc. etc. etc.
Friday, June 14, 13

Devops Is A Two-Way Street
• Systems Administrators
also have as much or more to
learn from developers as well!
Friday, June 14, 13

Source
Code
Compiler
Artifact
Test
Typical Development Workflow
• Typical Software Development Workflow:
• Write Source
• Compile Source
• Deploy Artifacts
• Write Tests
• Run Tests
Friday, June 14, 13

Source
Code
Compiler
Artifact
Test
• Source Code
• The recipe for a computer program
• Edited directly
• Managed via Source Control Software
Friday, June 14, 13

Source
Code
Compiler
Artifact
Test
• Compiler
• Takes source code and converts it into
executable programs
Friday, June 14, 13

Source
Code
Compiler
Artifact
Test
• Artifact
• Artifacts are executable programs
created by compilers.
• Compiled artifacts cannot be edited
directly. Source code must be changed
and re-compiled to produce a new build
artifact.
Friday, June 14, 13

Source
Code
Compiler
Artifact
Test
• Testing
• Write tests to verify that code works as
intended
• Run tests at different stages of the code
lifecycle to ensure correctness
Friday, June 14, 13

Source
Code
Compiler
Artifact
Test
• When developing software, most time
isn’t actually spent coding
• 10-second changes to source code can
take minutes to vet
• Compiling code
• Deploying code
• Writing & Running tests
Friday, June 14, 13

Source
Code
Compiler
Artifact
Test
Fun!
Boring!
• Things that are fun:
• Designing programs!
• Writing source code!
• Things that are boring:
• Compiling code
• Deploying artifacts
• Running tests
Friday, June 14, 13

Look familiar to any Chefs?Friday, June 14, 13

Worksta(on:+
Knife+cookbook+
create+
Worksta(on:+
Edit+cookbook+
Worksta(on:+
Knife+cookbook+
upload+
Provision+target+
Bootstrap+
target+
Worksta(on:+
Edit+target+run+
list+
ssh+target+
Target:+Run+
chef>client+
Traditional Cookbook Development: First Chef Run
Friday, June 14, 13

Worksta(on:+
edit+cookbook+
Worksta(on:+
knife+cookbook+
upload+
Target:+Run+
chef9client+
Traditional Cookbook Development: Subsequent Runs
Friday, June 14, 13

Source'Code'
Compiler'
Ar/fact'
Test'
Boooooooring
• Too much time doing
“paperwork”
• vi recipes/something.rb
• knife cookbook upload
• sudo pkill -USR1 chef-
client
• #%$#%$ something
broke, let me do that all
again
• Not enough time doing fun
stuff!
• Writing recipes
Less Fun!
More Boring!
X
Friday, June 14, 13

Source'Code'
Compiler'
Ar/fact'
Test'
Boooooooring
• Too much time doing
“paperwork”
• vi recipes/something.rb
• knife cookbook upload
• sudo pkill -USR1 chef-
client
• #%$#%$ something
broke, let me do that all
again
• Not enough time doing fun
stuff!
• Writing recipes
Less Fun!
More Boring!
XTH
IS
SUCKS!
Friday, June 14, 13

Chefs on a Plane
What if...
Friday, June 14, 13

Chefs on a Plane
What if...Worksta(on:+
Knife+cookbook+
create+
Worksta(on:+
Edit+cookbook+
Worksta(on:+
Knife+cookbook+
upload+
Provision+target+
Bootstrap+
target+
Worksta(on:+
Edit+target+run+
list+
ssh+target+
Target:+Run+
chef>client+
Target:(Run(
chef/client(
Worksta6on:(
edit(cookbook(
Worksta6on:(
knife(cookbook(
upload(
...we could
automate all
of this...
Friday, June 14, 13

Chefs on a Plane
What if...Worksta(on:+
Knife+cookbook+
create+
Worksta(on:+
Edit+cookbook+
Worksta(on:+
Knife+cookbook+
upload+
Provision+target+
Bootstrap+
target+
Worksta(on:+
Edit+target+run+
list+
ssh+target+
Target:+Run+
chef>client+
Target:(Run(
chef/client(
Worksta6on:(
edit(cookbook(
Worksta6on:(
knife(cookbook(
upload(
...we could
automate all
of this...
...to run entirely on this...
Friday, June 14, 13

Chefs on a Plane
What if...
...even while
aboard this?
Worksta(on:+
Knife+cookbook+
create+
Worksta(on:+
Edit+cookbook+
Worksta(on:+
Knife+cookbook+
upload+
Provision+target+
Bootstrap+
target+
Worksta(on:+
Edit+target+run+
list+
ssh+target+
Target:+Run+
chef>client+
Target:(Run(
chef/client(
Worksta6on:(
edit(cookbook(
Worksta6on:(
knife(cookbook(
upload(
...we could
automate all
of this...
...to run entirely on this...
Friday, June 14, 13

Source'Code'
Compiler'
Ar/fact'
Test'
Developing for Chef: Rapid Iteration
• Less time waiting around
for cookbook deploys and
Chef runs
• More frequent testing
• Better code
• Business needs met
more quickly
More Fun!
Less Boring!
X
Friday, June 14, 13

Code Can (Help)
Friday, June 14, 13

The Toolchain
Friday, June 14, 13

The Toolchain
• You’re becoming a developer!
• Reasonably powerful computer
• Good editor
Friday, June 14, 13

Have a Good Computer
• Virtualization is used to run
acceptance tests
• Running on a real virtualized
“node”
• Lots of memory (4GB+, 8GB
recommended)
• Fast disk (SSD)
• Good processor (Intel i5+)
• Modern OS
X
Friday, June 14, 13

Tools
• Chef (obviously)
• Virtualization provider
• Virtualization automation
(Vagrant)
• Cookbook dependency
manager (Berkshelf,
Librarian)
• Unit and acceptance testing
frameworks
Friday, June 14, 13

Software Testing
Terminology and Chef
Friday, June 14, 13

Different Types of Testing
• There are formal
Software Engineering
definitions for testing
Friday, June 14, 13

• Unit test
Friday, June 14, 13

• Unit test
• Integration test
Friday, June 14, 13

• Unit test
• Acceptance test
Friday, June 14, 13

• Unit test
• Acceptance test
• An easy explanation for
SAs is ...
Friday, June 14, 13

• Unit Test: Signal Input
What and When To Test
Flickr user: Rain Rabbit
Friday, June 14, 13

• Did I send Chef the correct
command?
Friday, June 14, 13

command?
• Signal Processing
Friday, June 14, 13

command?
• Did Chef interpret my command
correctly?
Friday, June 14, 13

command?
correctly?
• Acceptance Test: Signal Output
Friday, June 14, 13

command?
correctly?
• Acceptance Test: Signal Output
• Did my expressed intent, executed
by Chef, achieve the desired
result?
Friday, June 14, 13

Chef Testing Tools
Friday, June 14, 13

Chef Testing Tools
• Signal Input
• ChefSpec
Friday, June 14, 13

Chef Testing Tools
• Signal Input
• ChefSpec
• Signal out
• Chef Minitests & Minitest Handler
Friday, June 14, 13

Chef Testing Tools
• Signal Input
• ChefSpec
• Signal out
• Chef Minitests & Minitest Handler
• Signal processing
• Who cares; that’s what Opscode’s own tests for
Chef are for!
Friday, June 14, 13

General Philosophy for Chef Testing
Friday, June 14, 13

• Unit tests
• ChefSpec assertion for every resource of interest
• Each recipe should get its own test file
Friday, June 14, 13

• Unit tests
• Acceptance tests
• Anything that you couldn’t test in a unit test
Friday, June 14, 13

• Unit tests
• Acceptance tests
• Anything that you couldn’t test in a unit test
• Don’t repeat yourself!
Friday, June 14, 13

Unit Testing: ChefSpec
Friday, June 14, 13

ChefSpec Overview
• Built on top of RSpec
• Converge a Chef run in memory
• Overrides all providers to take no action
• Make assertions about the in-memory Chef run
• Mock Ohai data (automatic attributes) with Fauxhai
Friday, June 14, 13

Example Recipe Part One
include_recipe “java”
user node['sauceproxy']['server']['user'] do
  comment "SauceLabs Proxy User"
  system true
  action :create
end
directory node['sauceproxy']['server']['install_dir'] do
  owner node['sauceproxy']['server']['user']
  mode 00755
  action :create
end
# Can't assume we have unzip
package "unzip" do
  action :install
end
execute "unzip-saucelabs-proxy" do
  cwd node['sauceproxy']['server']['install_dir']
  command "unzip -o #{Chef::Config[:file_cache_path]}/#{node['sauceproxy']['server']['zipfile']}"
  action :nothing
  notifies :restart, "service[sauceproxy]"
end
https://github.com/juliandunn/sauceproxy/blob/master/recipes/server.rb
Friday, June 14, 13

Example Recipe Part Two
remote_file "#{Chef::Config[:file_cache_path]}/#{node['sauceproxy']['server']['zipfile']}" do
source "#{node['sauceproxy']['server']['download_url']}/#{node['sauceproxy']['server']['zipfile']}"
  action :create_if_missing
  notifies :run, "execute[unzip-saucelabs-proxy]", :immediately
end
template "/etc/init.d/sauceproxy" do
  source "sauceproxy.init.erb"
  mode 00755
  owner "root"
  group "root"
  action :create
end
template "/etc/sysconfig/sauceproxy" do
  source "sauceproxy.sysconfig.erb"
  mode 00644
  owner "root"
  group "root"
  action :create
end
service "sauceproxy" do
  supports :restart => true
  action [:enable, :start]
end
Friday, June 14, 13

Example ChefSpec Test: Setup
describe 'sauceproxy::server' do
let (:chef_run) do
runner = ChefSpec::ChefRunner.new(
platform: 'centos',
version: '6.3'
)
runner.node.set['sauceproxy']['server']['user'] = 'fake'
runner.node.set['sauceproxy']['server']['install_dir'] =
'/tmp/fake'
runner.node.set['sauceproxy']['server']['version'] = '3.14159'
runner.converge('sauceproxy::server')
end
Friday, June 14, 13

Example ChefSpec Test: Expectations
it 'should create a sauceproxy directory with the right
ownership' do
expect(chef_run).to create_directory('/tmp/fake')
expect(chef_run.directory('/tmp/fake')).to
be_owned_by('fake')
end
it 'should start a service called sauceproxy' do
expect(chef_run).to start_service('sauceproxy')
expect(chef_run).to
set_service_to_start_on_boot('sauceproxy')
end
end
Friday, June 14, 13

ChefSpec In Action
Friday, June 14, 13

ChefSpec: Expectations
• Many expectations (assertions) possible
• https://github.com/acrmp/chefspec#making-
assertions
• Don’t (just) restate static resources
• Mock out data
• Static resource assertions: good for regressions
• Whoops, I deleted the service resource
Friday, June 14, 13

Acceptance Testing:
MiniTest
Friday, June 14, 13

Chef Client Run
Friday, June 14, 13

Acceptance Tests as Report Handler
• Acceptance testing as a Chef Report Handler
• Many tools (serverspec, bats, minitest)
• I’ll demo Chef-MiniTest-Handler; most widely used
Friday, June 14, 13

MiniTest Handler Cookbook
• http://community.opscode.com/cookbooks/minitest-
handler
• Installs Minitest Gems
• Installs Chef Minitest Gems
• Installs the Chef-Minitest-Handler Notification
Handler for Chef-Client
• Places test files from cookbooks on the target node
as part of a Chef-client run
Friday, June 14, 13

MiniTest Files
• Each recipe gets an individual test file.
• recipes/server.rb
• files/default/test/server_test.rb
• Tests for each recipe are automatically loaded by the
handler
Friday, June 14, 13

Example MiniTest
require 'minitest/spec'
describe_recipe 'sauceproxy::server' do
it 'runs as a daemon' do
service('sauceproxy').must_be_running
end
end
http://tinyurl.com/minitest-examples
for many more example tests
Friday, June 14, 13

Make this go on a
with virtualization!
Friday, June 14, 13

Using Vagrant & Berkshelf to Iterate
• Vagrant is virtualization middleware
• Driven from Vagrantfile
• Defines VM images, customization parameters,
provisioners (Chef in our case)
• Berkshelf is a cookbook dependency manager
• Get dependent cookbooks from community API
• Feed them to Vagrant Chef provisioner
Friday, June 14, 13

Example Vagrantfile
Vagrant.configure("2") do |config|
config.vm.hostname = "sauceproxy-berkshelf"
config.vm.box = "opscode-centos-6.4"
config.vm.box_url = "https://opscode-vm.s3.amazonaws.com/vagrant/
opscode_centos-6.4_chef-11.4.4.box"
config.ssh.max_tries = 40
config.ssh.timeout = 120
config.vm.provision :chef_solo do |chef|
chef.json = {}
chef.run_list = [
"recipe[minitest-handler]",
"recipe[sauceproxy::server]"
]
end
end
Friday, June 14, 13

% vagrant up
Bringing Up Vagrant
borkbork:~/Dropbox/devel/github/juliandunn/sauceproxy (master)$ vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
[default] Importing base box 'centos-6.4'...
[default] Matching MAC address for NAT networking...
[default] Setting the name of the VM...
[default] Clearing any previously set forwarded ports...
[Berkshelf] Updating Vagrant's berkshelf: '/Users/juliandunn/.berkshelf/vagrant/
berkshelf-20130611-20810-rt7k01'
[Berkshelf] Using sauceproxy (0.1.8) at path: '/Users/juliandunn/Dropbox/devel/github/juliandunn/
sauceproxy'
[Berkshelf] Using minitest-handler (0.2.1)
[Berkshelf] Using java (1.11.4)
[Berkshelf] Using windows (1.8.10)
[Berkshelf] Using chef_handler (1.1.4)
[default] Creating shared folders metadata...
[default] Clearing any previously set network interfaces...
[default] Preparing network interfaces based on configuration...
[default] Forwarding ports...
[default] -- 22 => 2222 (adapter 1)
[default] Booting VM...
[default] Waiting for VM to boot. This can take a few minutes.
Friday, June 14, 13

Vagrant Run Continued
[default] Waiting for VM to boot. This can take a few minutes.
[default] VM booted and ready for use!
GuestAdditions 4.2.12 running --- OK.
[default] Setting hostname...
[default] Configuring and enabling network interfaces...
[default] Mounting shared folders...
[default] -- /vagrant
[default] -- /tmp/vagrant-chef-1/chef-solo-1/cookbooks
[default] Running provisioner: chef_solo...
Generating chef JSON and uploading...
Running chef-solo...
[2013-06-12T02:52:26+00:00] INFO: *** Chef 11.4.4 ***
[2013-06-12T02:52:26+00:00] INFO: Setting the run_list to ["recipe[minitest-
handler]", "recipe[sauceproxy::server]"] from JSON
[2013-06-12T02:52:26+00:00] INFO: Run List is [recipe[minitest-handler],
recipe[sauceproxy::server]]
[2013-06-12T02:52:26+00:00] INFO: Run List expands to [minitest-handler,
sauceproxy::server]
[2013-06-12T02:52:26+00:00] INFO: Starting Chef Run for sauceproxy-berkshelf
Friday, June 14, 13

Vagrant Run Continued
[2013-06-12T02:55:22+00:00] INFO: Chef Run complete in 175.788656866 seconds
[2013-06-12T02:55:22+00:00] INFO: Running report handlers
Run options: -v --seed 31883
# Running tests:
recipe::java::openjdk#test_0001_installs the correct version of the jdk =
0.11 s = .
recipe::java::openjdk#test_0002_properly sets JAVA_HOME environment variable =
0.04 s = .
recipe::sauceproxy::server#test_0001_runs as a daemon =
0.09 s = .
Finished tests in 0.244690s, 12.2604 tests/s, 12.2604 assertions/s.
3 tests, 3 assertions, 0 failures, 0 errors, 0 skips
[2013-06-12T02:55:22+00:00] INFO: Report handlers complete
Friday, June 14, 13

Minitests from Java cookbook ran too!
[2013-06-12T02:55:22+00:00] INFO: Chef Run complete in 175.788656866 seconds
[2013-06-12T02:55:22+00:00] INFO: Running report handlers
Run options: -v --seed 31883
# Running tests:
recipe::java::openjdk#test_0001_installs the correct version of the jdk =
0.11 s = .
recipe::java::openjdk#test_0002_properly sets JAVA_HOME environment variable =
0.04 s = .
recipe::sauceproxy::server#test_0001_runs as a daemon =
0.09 s = .
Finished tests in 0.244690s, 12.2604 tests/s, 12.2604 assertions/s.
3 tests, 3 assertions, 0 failures, 0 errors, 0 skips
[2013-06-12T02:55:22+00:00] INFO: Report handlers complete
Friday, June 14, 13

New Test/Dev Cycle
Write Unit Tests
Worked?
Run ChefSpec
No
Write Recipe
Code
Write Acceptance
Tests
Yes
Worked?
Run Vagrant + Minitest Handler
No
vagrant destroy
Yes Commit/Tag Code,
etc.
Friday, June 14, 13

CI Pipeline Integration
Friday, June 14, 13

Continuous Integration Pipeline
• Run unit tests in your pipeline
• Run acceptance tests in your
pipeline
• Drive cookbook uploads as the
output!
Friday, June 14, 13

Travis-CI Example for SauceProxy
language: ruby
gemfile:
  - gemfiles/travis.gemfile
rvm:
  - "1.9.2"
  - "1.9.3"
script: bundle exec rake test:syntax test:lint test:spec
notifications:
  email:
    - jdunn@opscode.com
Friday, June 14, 13

Rakefile Example
• Too long to post here
• Rake tasks for Foodcritic
and ChefSpec
• https://github.com/
juliandunn/sauceproxy/
blob/master/Rakefile
Friday, June 14, 13

Travis-CI Example Dashboard
Friday, June 14, 13

^#$%(#$ I broke the build
Friday, June 14, 13

Test Kitchen (alpha)
• Run acceptance test suite on
multiple OSes
• Different fixtures for each, if desired
• Different drivers (vagrant, ec2, lxc,
etc.)
• Hook up to Jenkins or other CI
system if desired
Friday, June 14, 13

Test Kitchen Config File
---
driver_plugin: vagrant
platforms:
- name: centos-6.4
driver_config:
box: opscode-centos-6.4
box_url: https://opscode-vm.s3.amazonaws.com/vagrant/opscode_centos-6.4_provisionerless.box
require_chef_omnibus: true
- name: centos-5.9
driver_config:
box: opscode-centos-5.9
box_url: https://opscode-vm.s3.amazonaws.com/vagrant/opscode_centos-5.9_provisionerless.box
require_chef_omnibus: true
suites:
- name: default
run_list: ["recipe[minitest-handler]", "recipe[sauceproxy::server]"]
attributes: {}
Friday, June 14, 13

Test Kitchen Demo
Friday, June 14, 13

Resources
• ChefSpec: https://github.com/acrmp/chefspec
• Chef Minitest Handler: https://github.com/calavera/
minitest-chef-handler
• Berkshelf: http://berkshelf.com/
• Vagrant: http://vagrantup.com/
• Test Kitchen: https://github.com/opscode/test-kitchen
• Bento: https://github.com/opscode/bento
• SauceProxy Sample Cookbook: https://github.com/
juliandunn/sauceproxy
Friday, June 14, 13

More on Testing
• Chef-NYC: The Hows and Whys
of Cookbook Testing
• Seth Vargo (Author of ChefSpec,
Fauxhai, many others)
• http://www.meetup.com/Chef-
NYC/events/122219772/
• June 25th in Manhattan
Friday, June 14, 13

Chef Fundamentals Training
• Boston, July 11-12
(CompuWorks, 263
Summer St.)
• eventbrite.com/event/
6652057483
• Use code BOSTON-
MEETUP to save 25%!
Friday, June 14, 13

Questions and Prizes
Also, we’re hiring! Automate all the things for fun and profit!
http://www.opscode.com/blog/careers/
(and/or see me after)
Friday, June 14, 13

Chef Cookbook Testing and Continuous Integration

Recommended

More Related Content

Viewers also liked (20)

Similar to Chef Cookbook Testing and Continuous Integration (20)

More from Julian Dunn (20)

Recently uploaded (20)

Chef Cookbook Testing and Continuous Integration