Anatomy of a Build Pipeline

Sam Brown
samuel.brown@excella.com
November 7, 2012

Thanks to Mike McGarr and
Excella Consulting for hosting!!

Sam Brown
 11+ Years as a Java developer with commercial and
federal clients
 Practicing continuous integration/continuous delivery
for ~6 years
 DevOps Evangelist at Excella (www.excella.com)
 Certified Scrum Master
 Puppet Certified Professional

Basic components of an automated enterprise
 Continuous integration
 Dependency management
 Automated build tools

to build...
 Shared API libraries
 Custom web applications
 Products

“The purpose of a pipeline is to transport some resource
from point A to point B quickly and effectively with minimal
upkeep or attention required once built” – me

So how did „pipelines‟ get applied to software? Let‟s
try a few changes to this statement...

“The purpose of a pipeline is to transport from
to quickly and
effectively with minimal upkeep or attention required once
built” – me

Build pipelines require measurements and verification of
the code to ensure:
 Adherence to standards
 Quality proven through testing
 A product that meets user‟s needs

The purpose is not just transport, but to ensure that our
product is high-quality, prepared for the environment it will
reach, and satisfies the end-user.

“An automated manifestation of the process required to get your
team’s application code to the end-user, typically implemented via
continuous integration server, with emphasis on eliminating
defects” – me (again)

 …in fact, NONE ARE!
 Build pipelines will vary as much as
applications
 Different teams have different needs
 Simplicity is key

One Size
Fits All!

Repeatable, automated, process to ensure that application code is
tested, analyzed, and packaged for deployment.

 System of record
 Just do it!
 Take advantage of commit hooks
 Build from trunk and reduce server-side
branches
 Tag often
 Don‟t check in broken code!

Purpose: Integrate, build and unit test code for quick
feedback
 Best Practices
 Runs in under 10 minutes (rapid feedback)
 Unit tests do not require external resources
 Run on EVERY developer check-in
 Fixing broken builds is the top priority!
 Gamification to drive adoption
 80% test coverage or BETTER
 Challenges
 LOTS of builds
 False sense of security
 Writing tests is hard

Purpose: Test component and/or external resource
integration
 Best Practices
 Test connectivity with external resources
 Test frameworks load correctly
 Test application components work together
 Test configuration
 Fewer integration tests than unit tests
 Challenges
 External resources may not be available in all environments
○ Mock locally
 Can be time consuming
○ Use local resources
○ Separate short/long running tests

Purpose: Use automated tools to inspect code
 Best Practices
 Check syntax
 Find security vulnerabilities
 Record test coverage
 Discover complexity
 Optional: Fail based on a metric
 Optional: View technical debt
 Challenges
 Not all code analysis tools are free
 Learning/installing new tools

Purpose: Label code and package as
deployable
Best Practices
 Labeling allows you to go back in time
 Packaging code for deployment
 Reduce complexity by combining steps
 NO configuration in package -> Package once,
deploy multiple
 Challenges
 Labeling can be resource intensive
 Many packaging options

Purpose: Make artifacts available for
deployment or available to other teams
 Best Practices
 Publish a versioned artifact
 Make repository available
 Reduce complexity by combining steps
 Challenges
 Requires initial complex setup
 Security requirements around exposing artifacts
○ Use a tool with security built-in like Nexus

Repeatable, automated, process to ensure that our target environment
is properly constructed for our application(s).

Purpose: Check syntax and compile prior to
application
 Puppet Lint – Static format checker for
Puppet manifests
 No-op Test Run – Ensure that manifest
compiles
 Challenges
 Puppet-lint requires a ruby-based environment
 No-op test needs production-like VM
 Long feedback loop

Purpose: Test infrastructure in a prod-like environment
 Puppet Apply –Puppet application against VM that
mimics DEV/TEST/PROD
 Infrastructure Tests – Test your environment!
 Example tests:
 Users and groups created
 Packages installed
 Services running
 Firewall configured
 Challenges
 Long feedback loop
 Yet another language (cucumber/rspec/other)
 VM must be up to date with DEV/TEST/PROD

cucumber-puppet rspec-puppet

Feature: Services require 'spec_helper'

Scenario Outline: Service should be running and bind to describe 'logrotate::rule' do
port let(:title) { 'nginx' }
When I run `lsof -i :<port>`
Then the output should match /<service>.*<user>/ it { should include_class('logrotate::rule') }

Examples: it do
| service | user | port | should contain_file('/etc/logrotate.d/nginx').with({
| master | root | 25 | 'ensure' => 'present',
| apache2 | www-data | 80 | 'owner' => 'root',
| dovecot | root | 110 | 'group' => 'root',
| mysqld | mysql | 3306 | 'mode' => '0444',
})
end
http://projects.puppetlabs.com/projects/cucu end
mber-puppet/wiki

http://rspec-puppet.com/

Repeatable, automated, process to ensure that our application is
properly installed in the target environment and that the application
meets acceptance criteria.

Purpose: Test acceptance criteria in a prod-like
environment
 Puppet Apply – Apply Puppet manifests including
deploying application
 Run Acceptance Tests – “End-to-end” testing
 End-user perspective
 Meets user-defined acceptance criteria
 Possible tools: Cucumber, Selenium, Geb, Sikuli
 Challenges
 Maintain a production-like VM
 Acceptance tests brittle
○ Test at the right level
 Acceptance tests long running
○ Run nightly

Purpose: Label application and infrastructure
code, deploy to DEV environment
 Label Release Candidate – Known
“accepted” versions will be deployed
together
 Deploy to DEV – Automated deployment
 Infrastructure AND application
 Challenges
 DEV updating, not deployed from scratch
○ Create tests for ALL possible scenarios
 Security
○ Work with security early and often!

Simplified process to support streamlined deployments to TEST
and PRODUCTION

Purpose: Enable the test team to pull the
latest code
 Pull-based deployment
 Manual Testing/Approval
 Challenges
 Enabling test team is a paradigm shift
 Producing changes too fast
○ Create good release notes
○ Not every build needs manual testing

Purpose: Enable operations team to pull the
latest code into production
 “Push-button” deployment to production
 Requires testing approval
 Challenges
 Audit/security check before deployment
○ Discuss with operations
○ Automate as much as possible and prudent
 Paradigm shift for operations, TOO EASY!
○ Engage the operations team as early and often
 Rollback/Roll forward strategy
○ Easier with RPM‟s, I prefer roll forward

 Remove human error
 Repeatability tests and improves the
process
 Visibility from code to deployment
 Baked-in quality
 Metrics, metrics, metrics
 Rapid and constant feedback
 Releases are non-events

Why do we store old/obsolete versions?
 Rollback
 Auditing
 History?
 Any other reason?

My view: Store only the latest build and current production release
 Bugs fixed in latest version
 (Almost) impossible to reproduce environments
 Version control has history

Exception: Other teams dependent on a previous version
 Store major/minor revisions

Reasoning: In a continuous delivery environment, delivering frequently
allows you to keep moving forward with new features AND bug fixes!

 Put EVERYTHING in version control
 Start simple, up your unit test coverage.
 Analyze your code in order to focus
 Install CI and start with two build steps
 Start and maintain a wiki
 And lastly…

 samuel.brown@excella.com

 @SamuelBrownIV

 http://github.com/samueltbrown

 http://www.linkedin.com/pub/samuel-brown/3/715/352

Anatomy of a Build Pipeline

Recommended

More Related Content

What's hot (20)

Similar to Anatomy of a Build Pipeline (20)

Recently uploaded (20)

Anatomy of a Build Pipeline

Editor's Notes