SlideShare a Scribd company logo
A Story of Continuous Integration
jcharlety@salesforce.com
cmourot@salesforce.com
Your speakers
C. Mourot
20142005
JB Charlety
20132004
We want to tell you a success story
​ This story is a fiction. Any resemblance to actual events is purely coincidental.
6 months
12 months
18 months
3 years
5 years
10 years
time
Once upon a time…
•  Class Project
•  2 people
•  USB keys / shared drive
•  < 100 tests
•  because the teachers wanted some tests
•  coded the last day
•  manual execution
At the beginning…
​ How effectively share the code
•  Use a Version Control
System (VCS)
​ How often run the tests?
•  Let’s nominate a “build
master”!
​ Let’s meet your uncle to get some love money ($)
​ x100
​ x3
I’m not able to start
the server!
Who has got the last
version of the code?
!
Sharing the code
mainlinea1 a3
a1
a
a2
a3
integration
commit, checkin
build ok
build: compile+test
file
sync, update
codeline
At Salesforce
more than 1000 engineers
one codeline (almost)
150+ scrum teams
more than 15 locations
Demo to a business angel ($$)
6 months later…
​ x500
I worked all night to fix
your $@!#% bugs!
​ x10
​ x1
It has not been
compiling for days
on my machine! Why is my feature not
in the demo?
I checked in yesterday!
mainline
Do we have tests?
6 months later… the day after
​ What happened? Integration chaos
​ Let’s empower the build master!
mainline
asap!
12 months later…
​ x1000
​ x15
I need to fix a bug
asap for a customer
​ x10
Damned!
But I don’t want to ship all
those new features
mainline
Let’s branch!
Versioning the code
mainline
v1.0
backport?
Be careful when choosing your branching pattern!
Versioning the code
http://www.bradapp.com/acme/branching/branch-structs.html
1.0-release
mainline
1.0-staging
2.0-release
2.0-staging
At Salesforce
​ v4 shipped in production
18 months later…
​ x2000
Customer Z is very angry with
the last version: it’s buggy!
​ x20
​ x10
Every morning, I fix
the build on my
machine
mainline
I spend all my time
fixing the build
Everybody is committing a
lot of code, the day before
the demo
Build ok!
Do we have tests?
18 months later… the day after
Automatic build
•  Build server
•  Run after each checkin
•  Compile + Tests + Alert
Everyone is responsible for the
build
•  Don’t break the build!
•  No more buildmaster
Scope/Time/Quality
mainline
At Salesforce
​ Quality : the triangle
​ Time and Quality are always fixed
•  Milestone/Release dates never change
•  Feature & Release Signoff requirements are quantifiable and deterministic
​ Scope is the only thing that you can change
•  Team cuts features to meet FF/RF signoff requirements
Scope
Time Quality
Planned Scope
Actual Scope
3 years later…
​ x5000
​ x35
Build last more than 2
hours !
​ x100
mainline
3 days to get the build
results! Are you kidding me?
What? I thought we had
continuous integration!
The build is broken
all the time!
•  How to speed up the build?
•  Buy a faster server
•  Buy more servers
•  Break in different chunks
•  Build stage
•  Test types
•  Choice
•  After each commit, compile only
•  Run the basic tests twice a day
•  Run the extended each night
Let’s scale
At Salesforce
​ As of Q2 2013
(20%)
UI Tests
Functional Tests
(70%)
Unit Tests (5%)
Type Of Test	
   Number of Tests	
  
Unit Test	
   12 000+	
  
Basic Functional	
   20 000+	
  
Full Functional	
   140 000+	
  
Extended Functional	
   25 000+	
  
Basic UI	
   500+	
  
Full UI	
   40 000+	
  
Extended UI	
   10 000+	
  
•  How to not break the build?
•  Unbreakable build
•  How to speed up test execution?
•  Add more servers to run in //
•  Who broke the tests?
•  Nominate a buildmaster to create bugs
and assign them to the suspects
​ Just bought another company in the US
5 years later…
​ x20k
​ x110
I’m always fixing the build
because the other team broke it!
​ x300
When is the
night?
The nightly build
lasts 3 days
More than 2000 tests are
failing!
•  automatic compilation of changes before check-in
•  basic tests run for each checkin
•  extended tests run on a regular basis
​ 
Precheckin
The Unbreakable Build
- UNBREAKABLE -
mainline
•  Who broke the tests?
•  Who is responsible to fix the tests?
•  How to handle flappy tests?
It’s time to automate the tests failures
process!
​ Growing, growing, growing!
10 years later…
​ x200k
​ x1000
My team delivers
new features,
doesn’t fix test!
More than 8000 tests are
failing!​ x300
Tests are ok on my machine,
they are failing on the servers
At Salesforce
​ TFMS: Automation of the tests failures process
​ Basics
•  Auto-create a bug for new test failures
•  Auto-close Test Failure bug
•  Re-open unresolved Test Failure
•  Detect duplicate Test Failure bug records
​ Advanced functionality
•  Suspect identification
•  Lock the line
•  Flapper detection
LucasfilmLtd.
At Salesforce
​ Suspect Change Identification
Full
TF
TF
TF
TF
CL1
CL2
CL5
CL4
CL3
At Salesforce
​ Code Check-in & Build Pipeline
Code
Review
Submit
to
Pre-
Checkin
Commit
to
perforce
Repo
Build
Run
Tests
Report
Test
Results
File
Bugs for
Test
Failures
PRECHECKIN AUTOBUILD TFMS
At Salesforce
​ Let’s recap
Code
•  Multi-million line code base
•  Several major branches
•  Hundreds of checkin per day
Tests
•  More than 200,000 tests
•  3000 hours serialized runtime
•  Number of Full Test Runs Processed – 1700/day
Continuous Integration
•  Number of remote instances (VMs) – 1300
•  Prod-like test environments
Bonus: a portfolio of patents related to testing
•  See google patents: salesforce testing
15 years later, what’s next?
•  From Continuous Integration to Continuous Delivery
•  Release more often, release smaller chunks
•  No more release freeze?
•  DevOps movement
•  cloud architecture
•  0 downtime
​ Ship features in production
15 years later, what’s next?
•  distributed testing
•  testing for UI - devices and browsers
​ Huge challenges
Key takeaways
​ Shorten the feedback loops
•  Compilation
•  Tests
•  Customers
​ Key principles of continuous integration
•  Version Control everything
•  If it hurts, do it more frequently
•  Automate everything
​ A solution for each scale
Thank you
Resources
​ http://www.martinfowler.com/articles/continuousIntegration.html
​ http://continuousdelivery.com/
Ad

More Related Content

What's hot (18)

The definitive guide to salesforce sandbox flosum
The definitive guide to salesforce sandbox flosumThe definitive guide to salesforce sandbox flosum
The definitive guide to salesforce sandbox flosum
Flosum
 
Team Development on Force.com with Github and Ant
Team Development on Force.com with Github and AntTeam Development on Force.com with Github and Ant
Team Development on Force.com with Github and Ant
Salesforce Developers
 
Testing in a DevOps team
Testing in a DevOps teamTesting in a DevOps team
Testing in a DevOps team
Laurent PY
 
Shift Left Mobile Application Testing
Shift Left Mobile Application TestingShift Left Mobile Application Testing
Shift Left Mobile Application Testing
Josiah Renaudin
 
Streamline Selenium Testing with Page Flow Navigation
Streamline Selenium Testing with Page Flow NavigationStreamline Selenium Testing with Page Flow Navigation
Streamline Selenium Testing with Page Flow Navigation
Salesforce Developers
 
Shift Left Quality Assurance: How to do it. Why it matters.
Shift Left Quality Assurance: How to do it. Why it matters.Shift Left Quality Assurance: How to do it. Why it matters.
Shift Left Quality Assurance: How to do it. Why it matters.
Worksoft
 
Automating the Impossible: End to End Team Development for ISVs (October 14, ...
Automating the Impossible: End to End Team Development for ISVs (October 14, ...Automating the Impossible: End to End Team Development for ISVs (October 14, ...
Automating the Impossible: End to End Team Development for ISVs (October 14, ...
Salesforce Partners
 
Fun with Jenkins & Salesforce
Fun with Jenkins & SalesforceFun with Jenkins & Salesforce
Fun with Jenkins & Salesforce
Abhinav Gupta
 
Webcast: TFS 2015 & Why Upgrade
Webcast: TFS 2015 & Why UpgradeWebcast: TFS 2015 & Why Upgrade
Webcast: TFS 2015 & Why Upgrade
InCycleSoftware
 
[Webinar] Test First, Fail Fast - Simplifying the Tester's Transition to DevOps
[Webinar] Test First, Fail Fast - Simplifying the Tester's Transition to DevOps[Webinar] Test First, Fail Fast - Simplifying the Tester's Transition to DevOps
[Webinar] Test First, Fail Fast - Simplifying the Tester's Transition to DevOps
KMS Technology
 
Implementing Test Automation: What a Manager Should Know
Implementing Test Automation: What a Manager Should KnowImplementing Test Automation: What a Manager Should Know
Implementing Test Automation: What a Manager Should Know
SoftServe
 
DevOps and Continuous Delivery with Visual Studio 2015 and VSTS
DevOps and Continuous Delivery with Visual Studio 2015 and VSTSDevOps and Continuous Delivery with Visual Studio 2015 and VSTS
DevOps and Continuous Delivery with Visual Studio 2015 and VSTS
Solidify
 
Automation testing strategy, approach & planning
Automation testing  strategy, approach & planningAutomation testing  strategy, approach & planning
Automation testing strategy, approach & planning
SivaprasanthRentala1975
 
Relieveing the Testing Bottle Neck - Webinar
Relieveing the Testing Bottle Neck - WebinarRelieveing the Testing Bottle Neck - Webinar
Relieveing the Testing Bottle Neck - Webinar
Cprime
 
Apex Trigger Debugging: Solving the Hard Problems
Apex Trigger Debugging: Solving the Hard ProblemsApex Trigger Debugging: Solving the Hard Problems
Apex Trigger Debugging: Solving the Hard Problems
Salesforce Developers
 
Technical Capabilities as enabler for Agile and DevOps
Technical Capabilities as enabler for Agile and DevOpsTechnical Capabilities as enabler for Agile and DevOps
Technical Capabilities as enabler for Agile and DevOps
Nelis Boucké
 
DevOps - Agile on Steroids by Tom Clement Oketch and Augustine Kisitu
DevOps - Agile on Steroids by Tom Clement Oketch and Augustine KisituDevOps - Agile on Steroids by Tom Clement Oketch and Augustine Kisitu
DevOps - Agile on Steroids by Tom Clement Oketch and Augustine Kisitu
Thoughtworks
 
SF1 - Apex Development Best Practises
SF1 - Apex Development Best PractisesSF1 - Apex Development Best Practises
SF1 - Apex Development Best Practises
Sebastian Wagner
 
The definitive guide to salesforce sandbox flosum
The definitive guide to salesforce sandbox flosumThe definitive guide to salesforce sandbox flosum
The definitive guide to salesforce sandbox flosum
Flosum
 
Team Development on Force.com with Github and Ant
Team Development on Force.com with Github and AntTeam Development on Force.com with Github and Ant
Team Development on Force.com with Github and Ant
Salesforce Developers
 
Testing in a DevOps team
Testing in a DevOps teamTesting in a DevOps team
Testing in a DevOps team
Laurent PY
 
Shift Left Mobile Application Testing
Shift Left Mobile Application TestingShift Left Mobile Application Testing
Shift Left Mobile Application Testing
Josiah Renaudin
 
Streamline Selenium Testing with Page Flow Navigation
Streamline Selenium Testing with Page Flow NavigationStreamline Selenium Testing with Page Flow Navigation
Streamline Selenium Testing with Page Flow Navigation
Salesforce Developers
 
Shift Left Quality Assurance: How to do it. Why it matters.
Shift Left Quality Assurance: How to do it. Why it matters.Shift Left Quality Assurance: How to do it. Why it matters.
Shift Left Quality Assurance: How to do it. Why it matters.
Worksoft
 
Automating the Impossible: End to End Team Development for ISVs (October 14, ...
Automating the Impossible: End to End Team Development for ISVs (October 14, ...Automating the Impossible: End to End Team Development for ISVs (October 14, ...
Automating the Impossible: End to End Team Development for ISVs (October 14, ...
Salesforce Partners
 
Fun with Jenkins & Salesforce
Fun with Jenkins & SalesforceFun with Jenkins & Salesforce
Fun with Jenkins & Salesforce
Abhinav Gupta
 
Webcast: TFS 2015 & Why Upgrade
Webcast: TFS 2015 & Why UpgradeWebcast: TFS 2015 & Why Upgrade
Webcast: TFS 2015 & Why Upgrade
InCycleSoftware
 
[Webinar] Test First, Fail Fast - Simplifying the Tester's Transition to DevOps
[Webinar] Test First, Fail Fast - Simplifying the Tester's Transition to DevOps[Webinar] Test First, Fail Fast - Simplifying the Tester's Transition to DevOps
[Webinar] Test First, Fail Fast - Simplifying the Tester's Transition to DevOps
KMS Technology
 
Implementing Test Automation: What a Manager Should Know
Implementing Test Automation: What a Manager Should KnowImplementing Test Automation: What a Manager Should Know
Implementing Test Automation: What a Manager Should Know
SoftServe
 
DevOps and Continuous Delivery with Visual Studio 2015 and VSTS
DevOps and Continuous Delivery with Visual Studio 2015 and VSTSDevOps and Continuous Delivery with Visual Studio 2015 and VSTS
DevOps and Continuous Delivery with Visual Studio 2015 and VSTS
Solidify
 
Automation testing strategy, approach & planning
Automation testing  strategy, approach & planningAutomation testing  strategy, approach & planning
Automation testing strategy, approach & planning
SivaprasanthRentala1975
 
Relieveing the Testing Bottle Neck - Webinar
Relieveing the Testing Bottle Neck - WebinarRelieveing the Testing Bottle Neck - Webinar
Relieveing the Testing Bottle Neck - Webinar
Cprime
 
Apex Trigger Debugging: Solving the Hard Problems
Apex Trigger Debugging: Solving the Hard ProblemsApex Trigger Debugging: Solving the Hard Problems
Apex Trigger Debugging: Solving the Hard Problems
Salesforce Developers
 
Technical Capabilities as enabler for Agile and DevOps
Technical Capabilities as enabler for Agile and DevOpsTechnical Capabilities as enabler for Agile and DevOps
Technical Capabilities as enabler for Agile and DevOps
Nelis Boucké
 
DevOps - Agile on Steroids by Tom Clement Oketch and Augustine Kisitu
DevOps - Agile on Steroids by Tom Clement Oketch and Augustine KisituDevOps - Agile on Steroids by Tom Clement Oketch and Augustine Kisitu
DevOps - Agile on Steroids by Tom Clement Oketch and Augustine Kisitu
Thoughtworks
 
SF1 - Apex Development Best Practises
SF1 - Apex Development Best PractisesSF1 - Apex Development Best Practises
SF1 - Apex Development Best Practises
Sebastian Wagner
 

Viewers also liked (7)

Automating Deployment Between Orgs Using Git & Continuous Integration
Automating Deployment Between Orgs Using Git & Continuous IntegrationAutomating Deployment Between Orgs Using Git & Continuous Integration
Automating Deployment Between Orgs Using Git & Continuous Integration
Sebastian Wagner
 
How to Setup Continuous Integration With Git, Jenkins, and Force.com
How to Setup Continuous Integration With Git, Jenkins, and Force.comHow to Setup Continuous Integration With Git, Jenkins, and Force.com
How to Setup Continuous Integration With Git, Jenkins, and Force.com
Salesforce Developers
 
2016 ISBG - Enterprise integration done right with Salesforce Lightning, IBM ...
2016 ISBG - Enterprise integration done right with Salesforce Lightning, IBM ...2016 ISBG - Enterprise integration done right with Salesforce Lightning, IBM ...
2016 ISBG - Enterprise integration done right with Salesforce Lightning, IBM ...
René Winkelmeyer
 
Introduction to Enterprise-Release Engineering on the Salesforce Platform
Introduction to Enterprise-Release Engineering on the Salesforce PlatformIntroduction to Enterprise-Release Engineering on the Salesforce Platform
Introduction to Enterprise-Release Engineering on the Salesforce Platform
Salesforce Developers
 
Gitlab flow solo
Gitlab flow soloGitlab flow solo
Gitlab flow solo
viniciusban
 
Building a Center of Excellence for your Salesforce crm team
Building a Center of Excellence for your Salesforce crm teamBuilding a Center of Excellence for your Salesforce crm team
Building a Center of Excellence for your Salesforce crm team
Buyan Thyagarajan
 
Full stackagile - Squads Chapters Tribes and Guilds
Full stackagile - Squads Chapters Tribes and GuildsFull stackagile - Squads Chapters Tribes and Guilds
Full stackagile - Squads Chapters Tribes and Guilds
Ashley-Christian Hardy
 
Automating Deployment Between Orgs Using Git & Continuous Integration
Automating Deployment Between Orgs Using Git & Continuous IntegrationAutomating Deployment Between Orgs Using Git & Continuous Integration
Automating Deployment Between Orgs Using Git & Continuous Integration
Sebastian Wagner
 
How to Setup Continuous Integration With Git, Jenkins, and Force.com
How to Setup Continuous Integration With Git, Jenkins, and Force.comHow to Setup Continuous Integration With Git, Jenkins, and Force.com
How to Setup Continuous Integration With Git, Jenkins, and Force.com
Salesforce Developers
 
2016 ISBG - Enterprise integration done right with Salesforce Lightning, IBM ...
2016 ISBG - Enterprise integration done right with Salesforce Lightning, IBM ...2016 ISBG - Enterprise integration done right with Salesforce Lightning, IBM ...
2016 ISBG - Enterprise integration done right with Salesforce Lightning, IBM ...
René Winkelmeyer
 
Introduction to Enterprise-Release Engineering on the Salesforce Platform
Introduction to Enterprise-Release Engineering on the Salesforce PlatformIntroduction to Enterprise-Release Engineering on the Salesforce Platform
Introduction to Enterprise-Release Engineering on the Salesforce Platform
Salesforce Developers
 
Gitlab flow solo
Gitlab flow soloGitlab flow solo
Gitlab flow solo
viniciusban
 
Building a Center of Excellence for your Salesforce crm team
Building a Center of Excellence for your Salesforce crm teamBuilding a Center of Excellence for your Salesforce crm team
Building a Center of Excellence for your Salesforce crm team
Buyan Thyagarajan
 
Full stackagile - Squads Chapters Tribes and Guilds
Full stackagile - Squads Chapters Tribes and GuildsFull stackagile - Squads Chapters Tribes and Guilds
Full stackagile - Squads Chapters Tribes and Guilds
Ashley-Christian Hardy
 
Ad

Similar to A Story of Continuous Integration (20)

BTD2015 - Your Place In DevTOps is Finding Solutions - Not Just Bugs!
BTD2015 - Your Place In DevTOps is Finding Solutions - Not Just Bugs!BTD2015 - Your Place In DevTOps is Finding Solutions - Not Just Bugs!
BTD2015 - Your Place In DevTOps is Finding Solutions - Not Just Bugs!
Andreas Grabner
 
Performance Metrics for your Build Pipeline - presented at Vienna WebPerf Oct...
Performance Metrics for your Build Pipeline - presented at Vienna WebPerf Oct...Performance Metrics for your Build Pipeline - presented at Vienna WebPerf Oct...
Performance Metrics for your Build Pipeline - presented at Vienna WebPerf Oct...
Andreas Grabner
 
Continuous delivery @wcap 5-09-2013
Continuous delivery   @wcap 5-09-2013Continuous delivery   @wcap 5-09-2013
Continuous delivery @wcap 5-09-2013
David Funaro
 
Sailing through devlopment with legacy code
Sailing through devlopment with legacy codeSailing through devlopment with legacy code
Sailing through devlopment with legacy code
prasadkunte
 
Bigger Unit Test Are Better
Bigger Unit Test Are BetterBigger Unit Test Are Better
Bigger Unit Test Are Better
Peter Schuler
 
Using MLOps to Bring ML to Production/The Promise of MLOps
Using MLOps to Bring ML to Production/The Promise of MLOpsUsing MLOps to Bring ML to Production/The Promise of MLOps
Using MLOps to Bring ML to Production/The Promise of MLOps
Weaveworks
 
An Introduction To Software Development - Final Review
An Introduction To Software Development - Final ReviewAn Introduction To Software Development - Final Review
An Introduction To Software Development - Final Review
Blue Elephant Consulting
 
Road to Continuous Delivery - Wix.com
Road to Continuous Delivery - Wix.comRoad to Continuous Delivery - Wix.com
Road to Continuous Delivery - Wix.com
Aviran Mordo
 
Automated testing in javascript
Automated testing in javascriptAutomated testing in javascript
Automated testing in javascript
Michael Yagudaev
 
Performance Tuning in the Trenches
Performance Tuning in the TrenchesPerformance Tuning in the Trenches
Performance Tuning in the Trenches
Donald Belcham
 
Continuous Integration Testing for SAP
Continuous Integration Testing for SAPContinuous Integration Testing for SAP
Continuous Integration Testing for SAP
Worksoft
 
5 Steps to Detecting Issues Earlier in Your Release Cycles
 5 Steps to Detecting Issues Earlier in Your Release Cycles 5 Steps to Detecting Issues Earlier in Your Release Cycles
5 Steps to Detecting Issues Earlier in Your Release Cycles
Perfecto by Perforce
 
Cross functional peer review preso 10-01-2013
Cross functional peer review preso 10-01-2013Cross functional peer review preso 10-01-2013
Cross functional peer review preso 10-01-2013
SmartBear
 
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Wojciech Seliga
 
Making a Mock by Kelsey Shannahan
Making a Mock by Kelsey ShannahanMaking a Mock by Kelsey Shannahan
Making a Mock by Kelsey Shannahan
QA or the Highway
 
Escaping Test Hell - ACCU 2014
Escaping Test Hell - ACCU 2014Escaping Test Hell - ACCU 2014
Escaping Test Hell - ACCU 2014
Wojciech Seliga
 
Test parallelization using Jenkins
Test parallelization using JenkinsTest parallelization using Jenkins
Test parallelization using Jenkins
Rogue Wave Software
 
Rsqrd AI: How to Design a Reliable and Reproducible Pipeline
Rsqrd AI: How to Design a Reliable and Reproducible PipelineRsqrd AI: How to Design a Reliable and Reproducible Pipeline
Rsqrd AI: How to Design a Reliable and Reproducible Pipeline
Sanjana Chowdhury
 
Scaling for Success: Lessons from handling peak loads on Azure with NServiceBus
Scaling for Success: Lessons from handling peak loads on Azure with NServiceBusScaling for Success: Lessons from handling peak loads on Azure with NServiceBus
Scaling for Success: Lessons from handling peak loads on Azure with NServiceBus
Particular Software
 
Continuous integration, delivery & deployment
Continuous integration,  delivery & deploymentContinuous integration,  delivery & deployment
Continuous integration, delivery & deployment
Martijn van der Kamp
 
BTD2015 - Your Place In DevTOps is Finding Solutions - Not Just Bugs!
BTD2015 - Your Place In DevTOps is Finding Solutions - Not Just Bugs!BTD2015 - Your Place In DevTOps is Finding Solutions - Not Just Bugs!
BTD2015 - Your Place In DevTOps is Finding Solutions - Not Just Bugs!
Andreas Grabner
 
Performance Metrics for your Build Pipeline - presented at Vienna WebPerf Oct...
Performance Metrics for your Build Pipeline - presented at Vienna WebPerf Oct...Performance Metrics for your Build Pipeline - presented at Vienna WebPerf Oct...
Performance Metrics for your Build Pipeline - presented at Vienna WebPerf Oct...
Andreas Grabner
 
Continuous delivery @wcap 5-09-2013
Continuous delivery   @wcap 5-09-2013Continuous delivery   @wcap 5-09-2013
Continuous delivery @wcap 5-09-2013
David Funaro
 
Sailing through devlopment with legacy code
Sailing through devlopment with legacy codeSailing through devlopment with legacy code
Sailing through devlopment with legacy code
prasadkunte
 
Bigger Unit Test Are Better
Bigger Unit Test Are BetterBigger Unit Test Are Better
Bigger Unit Test Are Better
Peter Schuler
 
Using MLOps to Bring ML to Production/The Promise of MLOps
Using MLOps to Bring ML to Production/The Promise of MLOpsUsing MLOps to Bring ML to Production/The Promise of MLOps
Using MLOps to Bring ML to Production/The Promise of MLOps
Weaveworks
 
An Introduction To Software Development - Final Review
An Introduction To Software Development - Final ReviewAn Introduction To Software Development - Final Review
An Introduction To Software Development - Final Review
Blue Elephant Consulting
 
Road to Continuous Delivery - Wix.com
Road to Continuous Delivery - Wix.comRoad to Continuous Delivery - Wix.com
Road to Continuous Delivery - Wix.com
Aviran Mordo
 
Automated testing in javascript
Automated testing in javascriptAutomated testing in javascript
Automated testing in javascript
Michael Yagudaev
 
Performance Tuning in the Trenches
Performance Tuning in the TrenchesPerformance Tuning in the Trenches
Performance Tuning in the Trenches
Donald Belcham
 
Continuous Integration Testing for SAP
Continuous Integration Testing for SAPContinuous Integration Testing for SAP
Continuous Integration Testing for SAP
Worksoft
 
5 Steps to Detecting Issues Earlier in Your Release Cycles
 5 Steps to Detecting Issues Earlier in Your Release Cycles 5 Steps to Detecting Issues Earlier in Your Release Cycles
5 Steps to Detecting Issues Earlier in Your Release Cycles
Perfecto by Perforce
 
Cross functional peer review preso 10-01-2013
Cross functional peer review preso 10-01-2013Cross functional peer review preso 10-01-2013
Cross functional peer review preso 10-01-2013
SmartBear
 
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Wojciech Seliga
 
Making a Mock by Kelsey Shannahan
Making a Mock by Kelsey ShannahanMaking a Mock by Kelsey Shannahan
Making a Mock by Kelsey Shannahan
QA or the Highway
 
Escaping Test Hell - ACCU 2014
Escaping Test Hell - ACCU 2014Escaping Test Hell - ACCU 2014
Escaping Test Hell - ACCU 2014
Wojciech Seliga
 
Test parallelization using Jenkins
Test parallelization using JenkinsTest parallelization using Jenkins
Test parallelization using Jenkins
Rogue Wave Software
 
Rsqrd AI: How to Design a Reliable and Reproducible Pipeline
Rsqrd AI: How to Design a Reliable and Reproducible PipelineRsqrd AI: How to Design a Reliable and Reproducible Pipeline
Rsqrd AI: How to Design a Reliable and Reproducible Pipeline
Sanjana Chowdhury
 
Scaling for Success: Lessons from handling peak loads on Azure with NServiceBus
Scaling for Success: Lessons from handling peak loads on Azure with NServiceBusScaling for Success: Lessons from handling peak loads on Azure with NServiceBus
Scaling for Success: Lessons from handling peak loads on Azure with NServiceBus
Particular Software
 
Continuous integration, delivery & deployment
Continuous integration,  delivery & deploymentContinuous integration,  delivery & deployment
Continuous integration, delivery & deployment
Martijn van der Kamp
 
Ad

More from Salesforce Engineering (20)

Locker Service Ready Lightning Components With Webpack
Locker Service Ready Lightning Components With WebpackLocker Service Ready Lightning Components With Webpack
Locker Service Ready Lightning Components With Webpack
Salesforce Engineering
 
Scaling HBase for Big Data
Scaling HBase for Big DataScaling HBase for Big Data
Scaling HBase for Big Data
Salesforce Engineering
 
Techniques to Effectively Monitor the Performance of Customers in the Cloud
Techniques to Effectively Monitor the Performance of Customers in the CloudTechniques to Effectively Monitor the Performance of Customers in the Cloud
Techniques to Effectively Monitor the Performance of Customers in the Cloud
Salesforce Engineering
 
Predictive System Performance Data Analysis
Predictive System Performance Data AnalysisPredictive System Performance Data Analysis
Predictive System Performance Data Analysis
Salesforce Engineering
 
Apache HBase State of the Project
Apache HBase State of the ProjectApache HBase State of the Project
Apache HBase State of the Project
Salesforce Engineering
 
Hit the Trail with Trailhead
Hit the Trail with TrailheadHit the Trail with Trailhead
Hit the Trail with Trailhead
Salesforce Engineering
 
HBase/PHOENIX @ Scale
HBase/PHOENIX @ ScaleHBase/PHOENIX @ Scale
HBase/PHOENIX @ Scale
Salesforce Engineering
 
Scaling up data science applications
Scaling up data science applicationsScaling up data science applications
Scaling up data science applications
Salesforce Engineering
 
Containers and Security for DevOps
Containers and Security for DevOpsContainers and Security for DevOps
Containers and Security for DevOps
Salesforce Engineering
 
Aspect Oriented Programming: Hidden Toolkit That You Already Have
Aspect Oriented Programming: Hidden Toolkit That You Already HaveAspect Oriented Programming: Hidden Toolkit That You Already Have
Aspect Oriented Programming: Hidden Toolkit That You Already Have
Salesforce Engineering
 
Monitoring @ Scale in Salesforce
Monitoring @ Scale in SalesforceMonitoring @ Scale in Salesforce
Monitoring @ Scale in Salesforce
Salesforce Engineering
 
Performance Tuning with XHProf
Performance Tuning with XHProfPerformance Tuning with XHProf
Performance Tuning with XHProf
Salesforce Engineering
 
A Smarter Pig: Building a SQL interface to Pig using Apache Calcite
A Smarter Pig: Building a SQL interface to Pig using Apache CalciteA Smarter Pig: Building a SQL interface to Pig using Apache Calcite
A Smarter Pig: Building a SQL interface to Pig using Apache Calcite
Salesforce Engineering
 
Implementing a Content Strategy Is Like Running 100 Miles
Implementing a Content Strategy Is Like Running 100 MilesImplementing a Content Strategy Is Like Running 100 Miles
Implementing a Content Strategy Is Like Running 100 Miles
Salesforce Engineering
 
Salesforce Cloud Infrastructure and Challenges - A Brief Overview
Salesforce Cloud Infrastructure and Challenges - A Brief OverviewSalesforce Cloud Infrastructure and Challenges - A Brief Overview
Salesforce Cloud Infrastructure and Challenges - A Brief Overview
Salesforce Engineering
 
Koober Preduction IO Presentation
Koober Preduction IO PresentationKoober Preduction IO Presentation
Koober Preduction IO Presentation
Salesforce Engineering
 
Finding Security Issues Fast!
Finding Security Issues Fast!Finding Security Issues Fast!
Finding Security Issues Fast!
Salesforce Engineering
 
Microservices
MicroservicesMicroservices
Microservices
Salesforce Engineering
 
Global State Management of Micro Services
Global State Management of Micro ServicesGlobal State Management of Micro Services
Global State Management of Micro Services
Salesforce Engineering
 
The Future of Hbase
The Future of HbaseThe Future of Hbase
The Future of Hbase
Salesforce Engineering
 
Locker Service Ready Lightning Components With Webpack
Locker Service Ready Lightning Components With WebpackLocker Service Ready Lightning Components With Webpack
Locker Service Ready Lightning Components With Webpack
Salesforce Engineering
 
Techniques to Effectively Monitor the Performance of Customers in the Cloud
Techniques to Effectively Monitor the Performance of Customers in the CloudTechniques to Effectively Monitor the Performance of Customers in the Cloud
Techniques to Effectively Monitor the Performance of Customers in the Cloud
Salesforce Engineering
 
Predictive System Performance Data Analysis
Predictive System Performance Data AnalysisPredictive System Performance Data Analysis
Predictive System Performance Data Analysis
Salesforce Engineering
 
Aspect Oriented Programming: Hidden Toolkit That You Already Have
Aspect Oriented Programming: Hidden Toolkit That You Already HaveAspect Oriented Programming: Hidden Toolkit That You Already Have
Aspect Oriented Programming: Hidden Toolkit That You Already Have
Salesforce Engineering
 
A Smarter Pig: Building a SQL interface to Pig using Apache Calcite
A Smarter Pig: Building a SQL interface to Pig using Apache CalciteA Smarter Pig: Building a SQL interface to Pig using Apache Calcite
A Smarter Pig: Building a SQL interface to Pig using Apache Calcite
Salesforce Engineering
 
Implementing a Content Strategy Is Like Running 100 Miles
Implementing a Content Strategy Is Like Running 100 MilesImplementing a Content Strategy Is Like Running 100 Miles
Implementing a Content Strategy Is Like Running 100 Miles
Salesforce Engineering
 
Salesforce Cloud Infrastructure and Challenges - A Brief Overview
Salesforce Cloud Infrastructure and Challenges - A Brief OverviewSalesforce Cloud Infrastructure and Challenges - A Brief Overview
Salesforce Cloud Infrastructure and Challenges - A Brief Overview
Salesforce Engineering
 
Global State Management of Micro Services
Global State Management of Micro ServicesGlobal State Management of Micro Services
Global State Management of Micro Services
Salesforce Engineering
 

Recently uploaded (20)

Salesforce AI Associate 2 of 2 Certification.docx
Salesforce AI Associate 2 of 2 Certification.docxSalesforce AI Associate 2 of 2 Certification.docx
Salesforce AI Associate 2 of 2 Certification.docx
José Enrique López Rivera
 
Datastucture-Unit 4-Linked List Presentation.pptx
Datastucture-Unit 4-Linked List Presentation.pptxDatastucture-Unit 4-Linked List Presentation.pptx
Datastucture-Unit 4-Linked List Presentation.pptx
kaleeswaric3
 
Build Your Own Copilot & Agents For Devs
Build Your Own Copilot & Agents For DevsBuild Your Own Copilot & Agents For Devs
Build Your Own Copilot & Agents For Devs
Brian McKeiver
 
Splunk Security Update | Public Sector Summit Germany 2025
Splunk Security Update | Public Sector Summit Germany 2025Splunk Security Update | Public Sector Summit Germany 2025
Splunk Security Update | Public Sector Summit Germany 2025
Splunk
 
"Client Partnership — the Path to Exponential Growth for Companies Sized 50-5...
"Client Partnership — the Path to Exponential Growth for Companies Sized 50-5..."Client Partnership — the Path to Exponential Growth for Companies Sized 50-5...
"Client Partnership — the Path to Exponential Growth for Companies Sized 50-5...
Fwdays
 
Automation Dreamin' 2022: Sharing Some Gratitude with Your Users
Automation Dreamin' 2022: Sharing Some Gratitude with Your UsersAutomation Dreamin' 2022: Sharing Some Gratitude with Your Users
Automation Dreamin' 2022: Sharing Some Gratitude with Your Users
Lynda Kane
 
Cyber Awareness overview for 2025 month of security
Cyber Awareness overview for 2025 month of securityCyber Awareness overview for 2025 month of security
Cyber Awareness overview for 2025 month of security
riccardosl1
 
Drupalcamp Finland – Measuring Front-end Energy Consumption
Drupalcamp Finland – Measuring Front-end Energy ConsumptionDrupalcamp Finland – Measuring Front-end Energy Consumption
Drupalcamp Finland – Measuring Front-end Energy Consumption
Exove
 
Manifest Pre-Seed Update | A Humanoid OEM Deeptech In France
Manifest Pre-Seed Update | A Humanoid OEM Deeptech In FranceManifest Pre-Seed Update | A Humanoid OEM Deeptech In France
Manifest Pre-Seed Update | A Humanoid OEM Deeptech In France
chb3
 
TrustArc Webinar: Consumer Expectations vs Corporate Realities on Data Broker...
TrustArc Webinar: Consumer Expectations vs Corporate Realities on Data Broker...TrustArc Webinar: Consumer Expectations vs Corporate Realities on Data Broker...
TrustArc Webinar: Consumer Expectations vs Corporate Realities on Data Broker...
TrustArc
 
Into The Box Conference Keynote Day 1 (ITB2025)
Into The Box Conference Keynote Day 1 (ITB2025)Into The Box Conference Keynote Day 1 (ITB2025)
Into The Box Conference Keynote Day 1 (ITB2025)
Ortus Solutions, Corp
 
"PHP and MySQL CRUD Operations for Student Management System"
"PHP and MySQL CRUD Operations for Student Management System""PHP and MySQL CRUD Operations for Student Management System"
"PHP and MySQL CRUD Operations for Student Management System"
Jainul Musani
 
Rusty Waters: Elevating Lakehouses Beyond Spark
Rusty Waters: Elevating Lakehouses Beyond SparkRusty Waters: Elevating Lakehouses Beyond Spark
Rusty Waters: Elevating Lakehouses Beyond Spark
carlyakerly1
 
UiPath Community Berlin: Orchestrator API, Swagger, and Test Manager API
UiPath Community Berlin: Orchestrator API, Swagger, and Test Manager APIUiPath Community Berlin: Orchestrator API, Swagger, and Test Manager API
UiPath Community Berlin: Orchestrator API, Swagger, and Test Manager API
UiPathCommunity
 
The Evolution of Meme Coins A New Era for Digital Currency ppt.pdf
The Evolution of Meme Coins A New Era for Digital Currency ppt.pdfThe Evolution of Meme Coins A New Era for Digital Currency ppt.pdf
The Evolution of Meme Coins A New Era for Digital Currency ppt.pdf
Abi john
 
Electronic_Mail_Attacks-1-35.pdf by xploit
Electronic_Mail_Attacks-1-35.pdf by xploitElectronic_Mail_Attacks-1-35.pdf by xploit
Electronic_Mail_Attacks-1-35.pdf by xploit
niftliyevhuseyn
 
Role of Data Annotation Services in AI-Powered Manufacturing
Role of Data Annotation Services in AI-Powered ManufacturingRole of Data Annotation Services in AI-Powered Manufacturing
Role of Data Annotation Services in AI-Powered Manufacturing
Andrew Leo
 
Hands On: Create a Lightning Aura Component with force:RecordData
Hands On: Create a Lightning Aura Component with force:RecordDataHands On: Create a Lightning Aura Component with force:RecordData
Hands On: Create a Lightning Aura Component with force:RecordData
Lynda Kane
 
Special Meetup Edition - TDX Bengaluru Meetup #52.pptx
Special Meetup Edition - TDX Bengaluru Meetup #52.pptxSpecial Meetup Edition - TDX Bengaluru Meetup #52.pptx
Special Meetup Edition - TDX Bengaluru Meetup #52.pptx
shyamraj55
 
What is Model Context Protocol(MCP) - The new technology for communication bw...
What is Model Context Protocol(MCP) - The new technology for communication bw...What is Model Context Protocol(MCP) - The new technology for communication bw...
What is Model Context Protocol(MCP) - The new technology for communication bw...
Vishnu Singh Chundawat
 
Salesforce AI Associate 2 of 2 Certification.docx
Salesforce AI Associate 2 of 2 Certification.docxSalesforce AI Associate 2 of 2 Certification.docx
Salesforce AI Associate 2 of 2 Certification.docx
José Enrique López Rivera
 
Datastucture-Unit 4-Linked List Presentation.pptx
Datastucture-Unit 4-Linked List Presentation.pptxDatastucture-Unit 4-Linked List Presentation.pptx
Datastucture-Unit 4-Linked List Presentation.pptx
kaleeswaric3
 
Build Your Own Copilot & Agents For Devs
Build Your Own Copilot & Agents For DevsBuild Your Own Copilot & Agents For Devs
Build Your Own Copilot & Agents For Devs
Brian McKeiver
 
Splunk Security Update | Public Sector Summit Germany 2025
Splunk Security Update | Public Sector Summit Germany 2025Splunk Security Update | Public Sector Summit Germany 2025
Splunk Security Update | Public Sector Summit Germany 2025
Splunk
 
"Client Partnership — the Path to Exponential Growth for Companies Sized 50-5...
"Client Partnership — the Path to Exponential Growth for Companies Sized 50-5..."Client Partnership — the Path to Exponential Growth for Companies Sized 50-5...
"Client Partnership — the Path to Exponential Growth for Companies Sized 50-5...
Fwdays
 
Automation Dreamin' 2022: Sharing Some Gratitude with Your Users
Automation Dreamin' 2022: Sharing Some Gratitude with Your UsersAutomation Dreamin' 2022: Sharing Some Gratitude with Your Users
Automation Dreamin' 2022: Sharing Some Gratitude with Your Users
Lynda Kane
 
Cyber Awareness overview for 2025 month of security
Cyber Awareness overview for 2025 month of securityCyber Awareness overview for 2025 month of security
Cyber Awareness overview for 2025 month of security
riccardosl1
 
Drupalcamp Finland – Measuring Front-end Energy Consumption
Drupalcamp Finland – Measuring Front-end Energy ConsumptionDrupalcamp Finland – Measuring Front-end Energy Consumption
Drupalcamp Finland – Measuring Front-end Energy Consumption
Exove
 
Manifest Pre-Seed Update | A Humanoid OEM Deeptech In France
Manifest Pre-Seed Update | A Humanoid OEM Deeptech In FranceManifest Pre-Seed Update | A Humanoid OEM Deeptech In France
Manifest Pre-Seed Update | A Humanoid OEM Deeptech In France
chb3
 
TrustArc Webinar: Consumer Expectations vs Corporate Realities on Data Broker...
TrustArc Webinar: Consumer Expectations vs Corporate Realities on Data Broker...TrustArc Webinar: Consumer Expectations vs Corporate Realities on Data Broker...
TrustArc Webinar: Consumer Expectations vs Corporate Realities on Data Broker...
TrustArc
 
Into The Box Conference Keynote Day 1 (ITB2025)
Into The Box Conference Keynote Day 1 (ITB2025)Into The Box Conference Keynote Day 1 (ITB2025)
Into The Box Conference Keynote Day 1 (ITB2025)
Ortus Solutions, Corp
 
"PHP and MySQL CRUD Operations for Student Management System"
"PHP and MySQL CRUD Operations for Student Management System""PHP and MySQL CRUD Operations for Student Management System"
"PHP and MySQL CRUD Operations for Student Management System"
Jainul Musani
 
Rusty Waters: Elevating Lakehouses Beyond Spark
Rusty Waters: Elevating Lakehouses Beyond SparkRusty Waters: Elevating Lakehouses Beyond Spark
Rusty Waters: Elevating Lakehouses Beyond Spark
carlyakerly1
 
UiPath Community Berlin: Orchestrator API, Swagger, and Test Manager API
UiPath Community Berlin: Orchestrator API, Swagger, and Test Manager APIUiPath Community Berlin: Orchestrator API, Swagger, and Test Manager API
UiPath Community Berlin: Orchestrator API, Swagger, and Test Manager API
UiPathCommunity
 
The Evolution of Meme Coins A New Era for Digital Currency ppt.pdf
The Evolution of Meme Coins A New Era for Digital Currency ppt.pdfThe Evolution of Meme Coins A New Era for Digital Currency ppt.pdf
The Evolution of Meme Coins A New Era for Digital Currency ppt.pdf
Abi john
 
Electronic_Mail_Attacks-1-35.pdf by xploit
Electronic_Mail_Attacks-1-35.pdf by xploitElectronic_Mail_Attacks-1-35.pdf by xploit
Electronic_Mail_Attacks-1-35.pdf by xploit
niftliyevhuseyn
 
Role of Data Annotation Services in AI-Powered Manufacturing
Role of Data Annotation Services in AI-Powered ManufacturingRole of Data Annotation Services in AI-Powered Manufacturing
Role of Data Annotation Services in AI-Powered Manufacturing
Andrew Leo
 
Hands On: Create a Lightning Aura Component with force:RecordData
Hands On: Create a Lightning Aura Component with force:RecordDataHands On: Create a Lightning Aura Component with force:RecordData
Hands On: Create a Lightning Aura Component with force:RecordData
Lynda Kane
 
Special Meetup Edition - TDX Bengaluru Meetup #52.pptx
Special Meetup Edition - TDX Bengaluru Meetup #52.pptxSpecial Meetup Edition - TDX Bengaluru Meetup #52.pptx
Special Meetup Edition - TDX Bengaluru Meetup #52.pptx
shyamraj55
 
What is Model Context Protocol(MCP) - The new technology for communication bw...
What is Model Context Protocol(MCP) - The new technology for communication bw...What is Model Context Protocol(MCP) - The new technology for communication bw...
What is Model Context Protocol(MCP) - The new technology for communication bw...
Vishnu Singh Chundawat
 

A Story of Continuous Integration

  • 3. We want to tell you a success story ​ This story is a fiction. Any resemblance to actual events is purely coincidental. 6 months 12 months 18 months 3 years 5 years 10 years time
  • 4. Once upon a time… •  Class Project •  2 people •  USB keys / shared drive •  < 100 tests •  because the teachers wanted some tests •  coded the last day •  manual execution
  • 5. At the beginning… ​ How effectively share the code •  Use a Version Control System (VCS) ​ How often run the tests? •  Let’s nominate a “build master”! ​ Let’s meet your uncle to get some love money ($) ​ x100 ​ x3 I’m not able to start the server! Who has got the last version of the code? !
  • 6. Sharing the code mainlinea1 a3 a1 a a2 a3 integration commit, checkin build ok build: compile+test file sync, update codeline
  • 7. At Salesforce more than 1000 engineers one codeline (almost) 150+ scrum teams more than 15 locations
  • 8. Demo to a business angel ($$) 6 months later… ​ x500 I worked all night to fix your $@!#% bugs! ​ x10 ​ x1 It has not been compiling for days on my machine! Why is my feature not in the demo? I checked in yesterday! mainline Do we have tests?
  • 9. 6 months later… the day after ​ What happened? Integration chaos ​ Let’s empower the build master! mainline asap!
  • 10. 12 months later… ​ x1000 ​ x15 I need to fix a bug asap for a customer ​ x10 Damned! But I don’t want to ship all those new features mainline
  • 11. Let’s branch! Versioning the code mainline v1.0 backport?
  • 12. Be careful when choosing your branching pattern! Versioning the code http://www.bradapp.com/acme/branching/branch-structs.html
  • 14. ​ v4 shipped in production 18 months later… ​ x2000 Customer Z is very angry with the last version: it’s buggy! ​ x20 ​ x10 Every morning, I fix the build on my machine mainline I spend all my time fixing the build Everybody is committing a lot of code, the day before the demo Build ok! Do we have tests?
  • 15. 18 months later… the day after Automatic build •  Build server •  Run after each checkin •  Compile + Tests + Alert Everyone is responsible for the build •  Don’t break the build! •  No more buildmaster Scope/Time/Quality mainline
  • 16. At Salesforce ​ Quality : the triangle ​ Time and Quality are always fixed •  Milestone/Release dates never change •  Feature & Release Signoff requirements are quantifiable and deterministic ​ Scope is the only thing that you can change •  Team cuts features to meet FF/RF signoff requirements Scope Time Quality Planned Scope Actual Scope
  • 17. 3 years later… ​ x5000 ​ x35 Build last more than 2 hours ! ​ x100 mainline 3 days to get the build results! Are you kidding me? What? I thought we had continuous integration! The build is broken all the time!
  • 18. •  How to speed up the build? •  Buy a faster server •  Buy more servers •  Break in different chunks •  Build stage •  Test types •  Choice •  After each commit, compile only •  Run the basic tests twice a day •  Run the extended each night Let’s scale
  • 19. At Salesforce ​ As of Q2 2013 (20%) UI Tests Functional Tests (70%) Unit Tests (5%) Type Of Test   Number of Tests   Unit Test   12 000+   Basic Functional   20 000+   Full Functional   140 000+   Extended Functional   25 000+   Basic UI   500+   Full UI   40 000+   Extended UI   10 000+  
  • 20. •  How to not break the build? •  Unbreakable build •  How to speed up test execution? •  Add more servers to run in // •  Who broke the tests? •  Nominate a buildmaster to create bugs and assign them to the suspects ​ Just bought another company in the US 5 years later… ​ x20k ​ x110 I’m always fixing the build because the other team broke it! ​ x300 When is the night? The nightly build lasts 3 days More than 2000 tests are failing!
  • 21. •  automatic compilation of changes before check-in •  basic tests run for each checkin •  extended tests run on a regular basis ​  Precheckin The Unbreakable Build - UNBREAKABLE - mainline
  • 22. •  Who broke the tests? •  Who is responsible to fix the tests? •  How to handle flappy tests? It’s time to automate the tests failures process! ​ Growing, growing, growing! 10 years later… ​ x200k ​ x1000 My team delivers new features, doesn’t fix test! More than 8000 tests are failing!​ x300 Tests are ok on my machine, they are failing on the servers
  • 23. At Salesforce ​ TFMS: Automation of the tests failures process ​ Basics •  Auto-create a bug for new test failures •  Auto-close Test Failure bug •  Re-open unresolved Test Failure •  Detect duplicate Test Failure bug records ​ Advanced functionality •  Suspect identification •  Lock the line •  Flapper detection LucasfilmLtd.
  • 24. At Salesforce ​ Suspect Change Identification Full TF TF TF TF CL1 CL2 CL5 CL4 CL3
  • 25. At Salesforce ​ Code Check-in & Build Pipeline Code Review Submit to Pre- Checkin Commit to perforce Repo Build Run Tests Report Test Results File Bugs for Test Failures PRECHECKIN AUTOBUILD TFMS
  • 26. At Salesforce ​ Let’s recap Code •  Multi-million line code base •  Several major branches •  Hundreds of checkin per day Tests •  More than 200,000 tests •  3000 hours serialized runtime •  Number of Full Test Runs Processed – 1700/day Continuous Integration •  Number of remote instances (VMs) – 1300 •  Prod-like test environments Bonus: a portfolio of patents related to testing •  See google patents: salesforce testing
  • 27. 15 years later, what’s next? •  From Continuous Integration to Continuous Delivery •  Release more often, release smaller chunks •  No more release freeze? •  DevOps movement •  cloud architecture •  0 downtime ​ Ship features in production
  • 28. 15 years later, what’s next? •  distributed testing •  testing for UI - devices and browsers ​ Huge challenges
  • 29. Key takeaways ​ Shorten the feedback loops •  Compilation •  Tests •  Customers ​ Key principles of continuous integration •  Version Control everything •  If it hurts, do it more frequently •  Automate everything ​ A solution for each scale