DevOps and the Importance of Single Source Code Repos

@RealGeneKim
Session ID:
Gene Kim
DevOps,
Dynamic, Learning Organizations
And Shared Repos:
My Fifteen Year Journey Studying
High Performing IT Organizations

@RealGeneKim
The Downward
Spiral…

@RealGeneKim
IT Ops And Dev At War
5

@RealGeneKim
There Is A Better Way…

@RealGeneKim
Google, Amazon, Netflix,
Spotify, Etsy, Spotify, Twitter,
Facebook…

@RealGeneKim
Walmart, Verizon, Raytheon,
Target, Nordstrom,
U.S. Dept of Homeland
Security…

@RealGeneKim
10 deploys per day
Dev & ops cooperation at Flickr
John Allspaw & Paul Hammond
Velocity 2009
Source: John Allspaw (@allspaw) and Paul Hammond (@ph)

@RealGeneKim
Making Changes When It Matters Most
“By installing a rampant innovation culture,
we performed 165 experiments in the peak three
months of tax season.”
–Scott Cook, Intuit Founder
“Our business result? Conversion rate of the
website is up 50 percent. Employee result?
Everyone loves it, because now their ideas can
make it to market.”

@RealGeneKim
Who Is Doing DevOps?
 Google, Amazon, Netflix, Etsy, Spotify, Twitter, Facebook …
 Microsoft, IBM, CA, SAP, HP, Red Hat, …
 GE Capital, Capital One, Nationwide, BNP Paribas, Amex,
BNY Mellon, MasterCard, Paychex, Intuit, …
 The Gap, Nordstrom, Macy’s, Williams-Sonoma, Target …
 General Motors, Raytheon, LEGO, Bosche …
 UK Government, US Department of Homeland Security …
 Kansas State University…
Who else?

@RealGeneKim
High Performers Are More Agile
30x 200x
more frequent
deployments
faster lead times
than their peers
Source: Puppet Labs 2013 State Of DevOps: http://puppetlabs.com/2013-state-of-devops-infographic

@RealGeneKim
High Performers Are More Reliable
60x 168x
the change
success rate
faster mean time
to recover (MTTR)
Source: Puppet Labs 2013 State Of DevOps: http://puppetlabs.com/2013-state-of-devops-infographic

@RealGeneKim
High Performers Win In The Marketplace
2x 50%more likely to
exceed profitability,
market share &
productivity goals
higher market
capitalization growth
over 3 years*
Source: Puppet Labs 2014 State Of DevOps

@RealGeneKim
“This book will have a profound effect on IT,
just as The Goal did for manufacturing.”
–Jez Humble,
co-author Continuous Delivery
“This is the IT swamp draining manual for
anyone who is neck deep in alligators.”
–Adrian Cockroft,
Cloud Architect at Netflix
“This is The Goal for our decade,
and is for any IT professional who wants
their life back.”
–Charles Betz, IT architect, author
“Architecture and Patterns for IT”

@RealGeneKim
The First Way: Flow

@RealGeneKim
“deploys per day”
vs.
“lead time”

@RealGeneKim
“What is your lead time
for changes?”
“How long does it take to go from
code committed to code successfully
running in production?”

@RealGeneKim
Lead Time = 9 months
48
Source: Damon Edwards (@damonedwards)

@RealGeneKim
Lead Time = 9 months
49
Source: Damon Edwards (@damonedwards)

@RealGeneKim
Everything In Version Control and
Environments Available On-Demand
 Make environments available early in the
Development process
 Make sure Dev builds the code and environment
at the same time
 Create a common Dev, QA and Production
environment creation process

@RealGeneKim
Deploy Smaller Changes, More Frequently *
Source: http://www.facebook.com/note.php?note_id=14218138919

@RealGeneKim
“As a lifelong Ops practitioner, I know
we need DevOps to make our work
humane.
In the past, I’ve worked every holiday, on
my birthday, my spouse’s birthday, and
even on the day my son was born.”
Nathan Shimek
Engineering Manager, New Context
@nathan_shimek

@RealGeneKim
CSG: Reducing Batch Size By 50%
Source: Scott Prugh, Chief Architect, CSG, Inc.
And the customer got the feature in
half the time!
 Apps supporting bill printing and
customer care for 100K+ customer
service reps, supporting 50MM telco
customers, 6B transactions per month
 20 technology platforms, including
mainframe VSAM and DB2, Java,
desktop client
 Moved from 2 to 4 releases per year
 Shared Operations Team performed
daily deployments to UAT

@RealGeneKimSource: John Jenkins, Amazon.com (2011)
Amazon 2010:
~15K deploys/day

@RealGeneKim
Amazon 2015:
136K deploys/day
Source: Ken Exner, Director of Dev Resources, Amazon.com (2015)

@RealGeneKim
“deploys / day”
“deploys / day / dev”

@RealGeneKim
High (linear)
Low
Med
Source: Puppet Labs 2015 State Of DevOps: https://puppetlabs.com/2015-devops-report
deploys/day
# of developers

@RealGeneKim
The First Way: Outcomes
 Creating single repository for code and environments
 All Ops artifacts in version control
 Determinism in the release process
 Consistent Dev, Test and Production environments, all properly
built before deployment begins
 Developers checking in code daily, being productive
 Features being deployed daily without catastrophic failures
 Decreased lead time
 Faster cycle time and release cadence

@RealGeneKim
The Second Way: Feedback

@RealGeneKim
How many times per day is the andon cord
pulled in a typical day at a Toyota
manufacturing plant?
3,500 times per day
Source: http://www.gembapantarei.com/2008/04/how_many_times_do_you_pull_the_andon_cord_each_day.html

@RealGeneKim
Why would Toyota do something so disruptive as
stopping production thousands of times per day?
“It’s the only way we can build 2,000 vehicles
per day – that’s one completed vehicle every
55 seconds.”

@RealGeneKim
Swarming Solve Problems And Creates
Knowledge
 Swarming and solving problems as they are seen
to build new knowledge
 Problems that are seen are solved so that new
knowledge is built quickly
 Improvement of daily work is prioritized above daily
work
Stopping work when builds, tests, deployments and services break,
enabling fast feedback loops, especially to Dev…
Source: Dr. Steven Spear

@RealGeneKim
"Automated tests transform fear into boredom."
-- Eran Messeri, Google
Google Dev And Ops (2013)
 15,000 engineers, working on 4,000+ projects
 All code is checked into one source tree
(billions of files!)
 5,500 code commits/day
 75 million test cases are run daily

@RealGeneKim
The Importance Of Shared Repositories
 Spreading new knowledge throughout the
organization
 The new discovery of local knowledge and
improvements are turned into global improvements,
shared throughout the organization
 Learning is fed back into the system to prevent future
failures
High trust culture, blameless post-mortems when things go wrong,
single source code repositories enterprise-wide, …
Source: Dr. Steven Spear

@RealGeneKim
Developers Carry Pagers
“We found that when we woke up developers at
2am, defects got fixed faster than ever”
– Patrick Lightbody,
VP Prod Mgmt, New Relic
“You build it, you run it.”
– Werner Vogels
CTO, Amazon

@RealGeneKim
“As a developer, the most satisfying
points in my career?
“It’s when I wrote the code, pushed the
button to deploy it, watched the metrics
to see if it actually worked in production,
and fixed it when it broke.”
Tim Tischler
Director of Operations Engineering
Nike, Inc.

@RealGeneKim
Pervasive Production Telemetry
“Having a
developer add a
monitoring metric
shouldn’t feel like
a schema
change.”
– John Allspaw,
SVP Tech Ops,
Etsy

@RealGeneKim
89
People actually look at the logs!
(Mention Verizon PCI Data Breach Study)

@RealGeneKim
Capital One: DevOpsSec
Source: Tapabrata Pal, Capital One

@RealGeneKim
One Of The Highest Predictors Of
Performance
Source: Typology Of Organizational Culture (Westrum, 2004)

@RealGeneKim
New engineer to John Allspaw:
“Is it okay for me to make this change?”
John Allspaw:
“I don’t know. Is it?”

@RealGeneKim
The Second Way: Outcomes
 Peer review of code and environment changes
 Disciplined automated testing enabling many simultaneous
small, agile teams to work productively
 Proactive monitoring of the production environment
 Defects and security issues getting fixed faster than ever
 High trust culture
 All groups communicating and coordinating better
 Everybody is getting more work done

@RealGeneKim
The Third Way:
Continual Experimentation And Learning

@RealGeneKim
Break Things Early And Often
“Do painful things more frequently, so you can
make it less painful… We don’t get pushback
from Dev, because they know it makes rollouts
smoother.”
– Adrian Cockcroft,
Former Architect, Netflix
(Now Technology Fellow,
Battery Ventures)

@RealGeneKim
Inject Failures Often

@RealGeneKim
You Don’t Choose Chaos Monkey…
Chaos Monkey Chooses You

@RealGeneKim
“[The result of Admiral Rickover creating this
capability] was that a young crew and their officers
setting out for their first cruise on a US naval
vessel benefits from the collective experience
gained from over 5,700 reactor-years of
experience behind them.”
— Dr. Steven Spear

@RealGeneKim
Allocate 20% Of Cycles To Technical
Debt Reduction

@RealGeneKim
DevOps Enterprise Summit
Learnings
Planning and Organizing
Initiatives DevOps

@RealGeneKim
DevOps Enterprise: Lessons Learned
 On Oct 19-21, we held the second DevOps
Enterprise Summit, a conference for horses, by
horses
 Speakers included fifty leaders from:
 Macy’s, Disney, Target, GE Capital, Western Union, Sherwin
Williams, Blackboard, Nordstrom, Telstra, US Department of
Homeland Security, CSG, Raytheon, IBM, Ticketmaster,
MITRE, Marks and Spencer, Barclays Capital, Microsoft,
Nationwide Insurance, Capital One, Gov.UK, Fidelity, Rally
Software, Neustar, Walmart, PNC, ADP, …

@RealGeneKim
Observations
 They were using the same technical practices and getting
the same sort of metrics as the unicorns
 Target: 100+ deploys per week, < 10 incidents per month, enabled
53 business initiatives
 Capital One: 100s of deploys per day, lead time of minutes
 Macy’s: 1,500 manual tests every 10 days, now 100Ks automated
tests run daily
 Disney: Has embedded nearly 100 Ops engineers into LOB teams
across the enterprise
 Nationwide Insurance: Retirement Plans app (COBOL on
mainframe)
 Raytheon: testing and certification from months to a day
 US CIS: security and compliance testing run every code commit

@RealGeneKim
Observations
 The transformation stories are among the most
courageous I’ve ever heard –
 Often the transformation leader was putting themselves
in personal jeopardy
 Why? Absolute clarity and conviction that it was the
right thing for the organization

@RealGeneKim
Why Do I Think This Is
Important?

@RealGeneKim
Want More Learn More?
To receive the following:
 A copy of this presentation
 The 140 page excerpt of The Phoenix Project
 Videos and slides from DevOps Enterprise 2014 & 2015
 Link to the DevOps Audit Defense Toolkit
 One hour excerpt of The Phoenix Project audiobook
 See early drafts of our upcoming DevOps Handbook
Just pick up your phone, and send an email:
To: realgenekim@SendYourSlides.com
Subject: devops
realgenekim@SendYourSlides.com
devops

DevOps and the Importance of Single Source Code Repos

Recommended

More Related Content

What's hot (20)

Viewers also liked (10)

Similar to DevOps and the Importance of Single Source Code Repos (20)

More from Perforce (20)

Recently uploaded (20)

DevOps and the Importance of Single Source Code Repos

Editor's Notes