SlideShare a Scribd company logo
Engineering Velocity: Continuous Delivery at Netflix
Dianne Marsh
SATURN 2014
en-gi-neer-ing + ve-loc-i-ty
!
applying science and technology to designing and building speed
into a system
Availability vs. Rate of ChangeAvailablity(in9’s)
0
1
2
3
4
5
6
Rate of Change
0 10 100 1000
Shift the CurveAvailablity(in9’s)
0
1
2
3
4
5
6
Rate of Change
0 10 100 1000 10000
http://www.slideshare.net/reed2001/culture-1798664
Manager’s Role
Context, not Control
Loosely coupled, Tightly aligned
And hire well!
Get out of the
Way
Freedom to Innovate
Support
Experimentation
!
How We Built a
Predictive
Autoscaling
Engine
http://techblog.netflix.com/2013/11/scryer-netflixs-predictive-auto-scaling.html
Support Independent Paths of Exploration
Don’t Prematurely Optimize!
Blameless Culture
Developers Deploy Their
Code
Run What You Wrote
!
• Rapid Innovation
• Rapid Detection
• Rapid Response
!
= Freedom + Responsibility
Support with Tools
Jenkins Job DSL
Configuration as Code
Groovy Script
Scripts go in Version Control
http://www.slideshare.net/quidryan/configuration-as-code
Aminator
Create AMI from Base AMI
Image contains service and everything needed to run it
Unit of Deployment for Test and Prod
Abstracts Cloud Details
http://techblog.netflix.com/2013/03/ami-creation-with-aminator.html
Asgard
Deploys Netflix to the Cloud
Red/Black push
Developed to address delays in rollback
http://www.infoq.com/presentations/asgard
Red/Black Push
!
• Scale up new instances
• Run canary analysis
• Turn on traffic to new ASG
• Turn off traffic to old ASG
• Wait … analyze … continue
Workflow
Continuous Delivery Engine
Judges between Stages
Represent Best Practices
http://techblog.netflix.com/2013/09/glisten-groovy-way-to-use-amazons.html
One Click Deployment?
Regional Isolation
Limit Impact of Human Error
!
• Stagger Deployments?
• Canary Testing per Region?
!
Know your Service!
Multi-Region Consistency
Build Tooling to:
!
• Schedule Deployments
• Prefer Off-Peak
• Choose Next Available
Region
• Provide Visibility by Region
Simian Army
• Chaos Monkey
• Latency Monkey
• Conformity Monkey
• Janitor Monkey 

(and more)
http://www.infoq.com/presentations/netflix-resiliency-failure-cloud
Chaos Monkey
Kills Running Instances
• Simulates failures inherent to
running in the cloud
• In Production
Latency Monkey
Introduces Latency
between services
Conformity Monkey
Have Deployments Diverged?
• Balance Regional
Consistency with Regional
Isolation
• Build Best Practices into
Tooling and Reporting
Janitor Monkey
Reduce Cognitive Load and
Cost
• Remove unused instances
• Uniform way to clean up
Shifting the Curve with
Tooling
• Value Self-Service
• Test Everywhere
• Awareness of Multiple Regions
• Best Practices Represented in Tooling
• Recover Quickly and Easily
• Be Cloud Native
Shifting the Curve with
Culture
• Context not Control
• Freedom to Experiment
• Blameless Culture
ArsTechnica, November 2012
“As the number of applications and the scale of
the campaign's AWS infrastructure use
climbed, the DevOps team shifted to using
Asgard—an open-source tool developed by
Netflix to manage cloud deployments.”
Thanks!
Dianne Marsh (@dmarsh)
dmarsh@netflix.com

More Related Content

PDF
How Netflix thinks of DevOps. Spoiler: we don’t.
Dianne Marsh
 
PPTX
Open Business Conference: Continuous Delivery At Netflix -- Powered by Open S...
Dianne Marsh
 
PDF
Engineering Velocity: Shifting the Curve at Netflix
Dianne Marsh
 
PPTX
Release the Monkeys ! Testing in the Wild at Netflix
Gareth Bowles
 
PPTX
I Don't Test Often ...
Gareth Bowles
 
PPTX
Understand Immutable infrastructure - at Build Stuff Kiev 2016
Quentin Adam
 
PPT
Devops at Netflix (re:Invent)
Jeremy Edberg
 
PDF
KubeCon/Cloud Native Keynote December 2017
Dianne Marsh
 
How Netflix thinks of DevOps. Spoiler: we don’t.
Dianne Marsh
 
Open Business Conference: Continuous Delivery At Netflix -- Powered by Open S...
Dianne Marsh
 
Engineering Velocity: Shifting the Curve at Netflix
Dianne Marsh
 
Release the Monkeys ! Testing in the Wild at Netflix
Gareth Bowles
 
I Don't Test Often ...
Gareth Bowles
 
Understand Immutable infrastructure - at Build Stuff Kiev 2016
Quentin Adam
 
Devops at Netflix (re:Invent)
Jeremy Edberg
 
KubeCon/Cloud Native Keynote December 2017
Dianne Marsh
 

What's hot (19)

PPTX
AWS Summit - Trends in Advanced Monitoring for AWS environments
Andreas Grabner
 
PDF
Scrum Control or Kanban Agility? You Can Have both, Using Metrics
Atlassian
 
PDF
Scaling Without Expanding: a DevOps Story
Atlassian
 
PDF
Let's Program The Cloud
Stephane Woillez
 
PPTX
Infrastructure as Code (BBWorld/DevCon13)
Mike McGarr
 
PDF
Security as Code: DOES15
Ed Bellis
 
PDF
Bitbucket Pipelines: Serverless CI/CD That Will Save Your Life
Atlassian
 
PDF
Serverless is more FinDev than DevOps
Yan Cui
 
PDF
Tracking Huge Files with Git LFS
Atlassian
 
PDF
Make Data Smart Again 2018 - Building a Smart Security Camera with Raspberry ...
Mark West
 
PDF
Introducing Change while Preserving Engineering Velocity
Dianne Marsh
 
PPTX
DevOps Days Toronto: From 6 Months Waterfall to 1 hour Code Deploys
Andreas Grabner
 
PDF
Demystifying Web Performance
Atlassian
 
PDF
Serverless 101 in Montreal
Aaron Williams
 
PDF
Debunking serverless myths
Yan Cui
 
PDF
OWASP DefectDojo - Open Source Security Sanity
Matt Tesauro
 
PPTX
Riga Dev Days: Building a Smart Security Camera with Raspberry Pi Zero, Node....
Mark West
 
PPTX
GeeCon 2017 : Building a Smart Security Camera with Raspberry Pi Zero, Node.j...
Mark West
 
PDF
Saving Money by Optimizing Your Cloud Add-On Infrastructure
Atlassian
 
AWS Summit - Trends in Advanced Monitoring for AWS environments
Andreas Grabner
 
Scrum Control or Kanban Agility? You Can Have both, Using Metrics
Atlassian
 
Scaling Without Expanding: a DevOps Story
Atlassian
 
Let's Program The Cloud
Stephane Woillez
 
Infrastructure as Code (BBWorld/DevCon13)
Mike McGarr
 
Security as Code: DOES15
Ed Bellis
 
Bitbucket Pipelines: Serverless CI/CD That Will Save Your Life
Atlassian
 
Serverless is more FinDev than DevOps
Yan Cui
 
Tracking Huge Files with Git LFS
Atlassian
 
Make Data Smart Again 2018 - Building a Smart Security Camera with Raspberry ...
Mark West
 
Introducing Change while Preserving Engineering Velocity
Dianne Marsh
 
DevOps Days Toronto: From 6 Months Waterfall to 1 hour Code Deploys
Andreas Grabner
 
Demystifying Web Performance
Atlassian
 
Serverless 101 in Montreal
Aaron Williams
 
Debunking serverless myths
Yan Cui
 
OWASP DefectDojo - Open Source Security Sanity
Matt Tesauro
 
Riga Dev Days: Building a Smart Security Camera with Raspberry Pi Zero, Node....
Mark West
 
GeeCon 2017 : Building a Smart Security Camera with Raspberry Pi Zero, Node.j...
Mark West
 
Saving Money by Optimizing Your Cloud Add-On Infrastructure
Atlassian
 
Ad

Similar to Saturn 2014. Engineering Velocity: Continuous Delivery at Netflix (20)

PPTX
From Code to the Monkeys: Continuous Delivery at Netflix
Dianne Marsh
 
PDF
Qconny2014dmarsh 140613080328-phpapp02
재구 김
 
PDF
Continuous Delivery with NetflixOSS
Daniel Woods
 
KEY
Real World Cloud Application Security
Jason Chan
 
PPTX
Surge 2013: Maximizing Scalability, Resiliency, and Engineering Velocity in t...
Coburn Watson
 
PPTX
Dystopia as a Service
Adrian Cockcroft
 
PPTX
Engineering Tools at Netflix: Enabling Continuous Delivery
Mike McGarr
 
PPTX
Resilience and Security @ Scale: Lessons Learned
Jason Chan
 
PDF
Netflix Architecture and Open Source
All Things Open
 
PPTX
Netflix Cloud Architecture and Open Source
aspyker
 
PDF
20140708 - Jeremy Edberg: How Netflix Delivers Software
DevOps Chicago
 
PPTX
Engineering Netflix Global Operations in the Cloud
Josh Evans
 
PPTX
Gluecon 2013 - NetflixOSS Cloud Native Tutorial Introduction
Adrian Cockcroft
 
PPTX
Yow Conference Dec 2013 Netflix Workshop Slides with Notes
Adrian Cockcroft
 
PDF
Mini-Training: Netflix Simian Army
Betclic Everest Group Tech Team
 
PDF
DevOps, Common use cases, Architectures, Best Practices
Shiva Narayanaswamy
 
PDF
QConSF 2014 - How we learned to stop worrying and start deploying the Netflix...
Sangeeta Narayanan
 
PDF
Triangle Devops Meetup 10/2015
aspyker
 
PPTX
Cloud @ Scania - Södertälje Science Week
Anders Lundsgård
 
PPTX
CMG2013 Workshop: Netflix Cloud Native, Capacity, Performance and Cost Optimi...
Adrian Cockcroft
 
From Code to the Monkeys: Continuous Delivery at Netflix
Dianne Marsh
 
Qconny2014dmarsh 140613080328-phpapp02
재구 김
 
Continuous Delivery with NetflixOSS
Daniel Woods
 
Real World Cloud Application Security
Jason Chan
 
Surge 2013: Maximizing Scalability, Resiliency, and Engineering Velocity in t...
Coburn Watson
 
Dystopia as a Service
Adrian Cockcroft
 
Engineering Tools at Netflix: Enabling Continuous Delivery
Mike McGarr
 
Resilience and Security @ Scale: Lessons Learned
Jason Chan
 
Netflix Architecture and Open Source
All Things Open
 
Netflix Cloud Architecture and Open Source
aspyker
 
20140708 - Jeremy Edberg: How Netflix Delivers Software
DevOps Chicago
 
Engineering Netflix Global Operations in the Cloud
Josh Evans
 
Gluecon 2013 - NetflixOSS Cloud Native Tutorial Introduction
Adrian Cockcroft
 
Yow Conference Dec 2013 Netflix Workshop Slides with Notes
Adrian Cockcroft
 
Mini-Training: Netflix Simian Army
Betclic Everest Group Tech Team
 
DevOps, Common use cases, Architectures, Best Practices
Shiva Narayanaswamy
 
QConSF 2014 - How we learned to stop worrying and start deploying the Netflix...
Sangeeta Narayanan
 
Triangle Devops Meetup 10/2015
aspyker
 
Cloud @ Scania - Södertälje Science Week
Anders Lundsgård
 
CMG2013 Workshop: Netflix Cloud Native, Capacity, Performance and Cost Optimi...
Adrian Cockcroft
 
Ad

More from Dianne Marsh (7)

PDF
Velocity 2017 Keynote: Looking Back to Move Forward
Dianne Marsh
 
PDF
OSS Collaboration & Contribution: How Netflix Drives Industry Engagement
Dianne Marsh
 
PDF
The Paved Road at Netflix
Dianne Marsh
 
PDF
Walking the Tightrope: Balancing Bias to Action and Planning
Dianne Marsh
 
PDF
ScalaDays 2015 Keynote
Dianne Marsh
 
PDF
Dr scalalove: How I learned to stop worrying and love the functions
Dianne Marsh
 
PPT
Sneaking Scala through the Back Door
Dianne Marsh
 
Velocity 2017 Keynote: Looking Back to Move Forward
Dianne Marsh
 
OSS Collaboration & Contribution: How Netflix Drives Industry Engagement
Dianne Marsh
 
The Paved Road at Netflix
Dianne Marsh
 
Walking the Tightrope: Balancing Bias to Action and Planning
Dianne Marsh
 
ScalaDays 2015 Keynote
Dianne Marsh
 
Dr scalalove: How I learned to stop worrying and love the functions
Dianne Marsh
 
Sneaking Scala through the Back Door
Dianne Marsh
 

Recently uploaded (20)

PPTX
Explanation about Structures in C language.pptx
Veeral Rathod
 
PDF
Jenkins: An open-source automation server powering CI/CD Automation
SaikatBasu37
 
PPTX
Presentation about variables and constant.pptx
safalsingh810
 
PDF
49785682629390197565_LRN3014_Migrating_the_Beast.pdf
Abilash868456
 
PDF
Download iTop VPN Free 6.1.0.5882 Crack Full Activated Pre Latest 2025
imang66g
 
PDF
New Download FL Studio Crack Full Version [Latest 2025]
imang66g
 
PPTX
Presentation about variables and constant.pptx
kr2589474
 
PDF
Enhancing Healthcare RPM Platforms with Contextual AI Integration
Cadabra Studio
 
PPTX
Role Of Python In Programing Language.pptx
jaykoshti048
 
PPT
Activate_Methodology_Summary presentatio
annapureddyn
 
PPTX
The-Dawn-of-AI-Reshaping-Our-World.pptxx
parthbhanushali307
 
PDF
lesson-2-rules-of-netiquette.pdf.bshhsjdj
jasmenrojas249
 
PDF
MiniTool Power Data Recovery Crack New Pre Activated Version Latest 2025
imang66g
 
PDF
An Experience-Based Look at AI Lead Generation Pricing, Features & B2B Results
Thomas albart
 
PPTX
AI-Ready Handoff: Auto-Summaries & Draft Emails from MQL to Slack in One Flow
bbedford2
 
PDF
Salesforce Implementation Services Provider.pdf
VALiNTRY360
 
PPTX
Contractor Management Platform and Software Solution for Compliance
SHEQ Network Limited
 
PDF
Applitools Platform Pulse: What's New and What's Coming - July 2025
Applitools
 
PDF
Teaching Reproducibility and Embracing Variability: From Floating-Point Exper...
University of Rennes, INSA Rennes, Inria/IRISA, CNRS
 
PDF
Exploring AI Agents in Process Industries
amoreira6
 
Explanation about Structures in C language.pptx
Veeral Rathod
 
Jenkins: An open-source automation server powering CI/CD Automation
SaikatBasu37
 
Presentation about variables and constant.pptx
safalsingh810
 
49785682629390197565_LRN3014_Migrating_the_Beast.pdf
Abilash868456
 
Download iTop VPN Free 6.1.0.5882 Crack Full Activated Pre Latest 2025
imang66g
 
New Download FL Studio Crack Full Version [Latest 2025]
imang66g
 
Presentation about variables and constant.pptx
kr2589474
 
Enhancing Healthcare RPM Platforms with Contextual AI Integration
Cadabra Studio
 
Role Of Python In Programing Language.pptx
jaykoshti048
 
Activate_Methodology_Summary presentatio
annapureddyn
 
The-Dawn-of-AI-Reshaping-Our-World.pptxx
parthbhanushali307
 
lesson-2-rules-of-netiquette.pdf.bshhsjdj
jasmenrojas249
 
MiniTool Power Data Recovery Crack New Pre Activated Version Latest 2025
imang66g
 
An Experience-Based Look at AI Lead Generation Pricing, Features & B2B Results
Thomas albart
 
AI-Ready Handoff: Auto-Summaries & Draft Emails from MQL to Slack in One Flow
bbedford2
 
Salesforce Implementation Services Provider.pdf
VALiNTRY360
 
Contractor Management Platform and Software Solution for Compliance
SHEQ Network Limited
 
Applitools Platform Pulse: What's New and What's Coming - July 2025
Applitools
 
Teaching Reproducibility and Embracing Variability: From Floating-Point Exper...
University of Rennes, INSA Rennes, Inria/IRISA, CNRS
 
Exploring AI Agents in Process Industries
amoreira6
 

Saturn 2014. Engineering Velocity: Continuous Delivery at Netflix