Build & Release Engineering

ì

Build
&
Release
Engineering

By:
Pranesh
Vi.al
CG

h.p://linkedin.com/in/praneshvi.al

Main
Agenda
of
this
session

h"p://linkedin.com/in/praneshvi"al

2

Key
Takeaways
from
this
session

ì  So<ware
Release
Principles
&
its
importance.

ì  Not
to
expect
results
from
“Day
1”.
It’s
a
slow
process.
Takes
months
or
at

Lmes
even
years
to
reach
perfecLon
levels.

ì  What
is
“ConLnuous
IntegraLon”?

ì  Steps
associated
with
“ConLnuous
IntegraLon”?

ì  What
is
a
“ConLnuous
Delivery”?

ì  Steps
associated
with
“ConLnuous
Delivery”?

ì  What
is
a
“ConLnuous
Deployment”?

ì  Release
Engineering
Steps.

ì  Main
Focus
of
Release
Engineering.


3

Key
Takeaways
from
this
session

ì  ConﬁguraLon
Management.

ì  Principles
of
Managing
ApplicaLon
ConﬁguraLon.

ì  Its
not
just
about
the
tools.
Its
about
the
big
“Cultural
Change”
that

everyone
has
to
go
through
.

ì  Release
Engineering
Architecture.

ì  Deep
dive
into
commit
/
component
builds.

ì  AnL-‐Pa.erns
in
commit
/
component
builds.

ì  Aim
of
Deployment
Pipeline.

ì  AnL-‐Pa.erns
of
Deployment
Pipeline.

ì  Best
qualiLes
of
a
Deployment
Pipeline.


4

Key
Takeaways
from
this
session

ì  Deployment
on
Large
Scale
ProducLon
Environments.

ì  Why
Automated
Deployment
is
an
indispensable
goal
???

ì  Deep
dive
into
TesLng
Quadrant.

ì  Deep
Dive
into
Automated
Acceptance
Tests.

ì  Few
more
Generic
AnL-‐Pa.erns.

ì  AdapLon
of
Release
Engineering.

ì  Tools
Used
at
each
of
the
stages.

ì  Managing
Environments.

ì  Managing
Releases
of
complex
systems.


5

Key
Takeaways
from
this
session

ì  Steps
involved
to
build
a
deployment
pipeline.

ì  Everyone
is
an
equal
contributor
to
reach
“ConLnuous
Deployment”
stage.

ì  MulL-‐Tenancy
Cloud.

ì  MulL-‐Instance
ApplicaLons.

ì  Zero
DownLme
Deployment.

ì  ExtracLng
the
server
logs.

ì  Next
generaLon
tools
in
the
DevOps
world.

ì  PreparaLon
for
the
Release.

ì  Maturity
of
a
“Deployment
Pipeline”.


6

Target
Audience

ì  Product
Managers
/
Business
Analysts.

ì  Engineering
Managers.

ì  Development
Teams.

ì  Quality
Engineering
Teams.

ì  System
Admins.

ì  Service
Engineering
/
ProducLon
Engineering
Teams.

ì  On-‐site
Coordinators.


7

Software
Release
Principles
&
its
importance.

ì  Should
be
Fast
/
Predictable
/
Repeatable.

ì  Automate
almost
everything.

ì  Keep
everything
in
Version
Control.

ì  If
it
hurts,
do
it
more
o<en.

ì  Improve
the
quality
of
the
Builds.
Push
the
boundaries
of
test

automaLon
so
that
quality
is
not
compromised.

ì  Done
means
the
so<ware
is
released
and
has
reached
the

producLon.


8

What
is
Deployment?

ì  Run
the
same
commands
in
several
hosts.

ì  Install
the
required
so<ware
from
a
central
repository
so
that

the
end
state
of
every
host
is
same.

ì  Based
on
the
host-‐type,
install
the
required
packages.

ì  End
goal
is
to
bring
all
the
hosts
of
the
respecLve
host-‐type
in

the
same
state.


9

What
is
“Continuous
Integration”?

ì  Building,
deploying
and
tesLng
the
applicaLon.


10

Steps
associated
with
“Continuous

Integration”?

ì  Integrate
all
the
components
at
regular
intervals.

ì  Binaries
built
only
once
and
used
in
all
the
environments.

ì  Execute
Unit
Tests
before
the
packages
are
generated.

ì  Deploy
the
latest
packages
onto
an
INT
environment.

ì  Run
the
automated
tests,
validate
the
test
results.

ì  Track
the
Code
quality
criteria
such
as
staLc
code
analysis,
test
coverage.

ì  Deploying
to
other
environments
with
a
pre-‐deﬁned
frequency.

ì  Repeat
the
process.

ì  It’s
a
pracGce,
not
a
tool


11

What
is
a
“Continuous
Delivery”?

ì  Building,
deploying,
tesLng,
promoLon
of
the
applicaLon
to
the
next

environment.

ì  Image
credit:
h.p://www.axonivy.com/


12

Steps
associated
with
“Continuous

Delivery”?

ì  Steps
followed
in
“ConLnuous
IntegraLon”.

ì  AutomaLc
promoLon
/
deployment
to
other
environments
(QA,

Staging,
Performance
etc…)
upon
successful
execuLon
of

automated
tests.

ì  ApplicaLon
is
always
ready
to
deploy
to
producLon
through
a

largely
automated
process.


13

What
is
a
“Continuous
Deployment”?

ì  Building,
deploying,
tesLng,
promoLon
of
the
applicaLon
to
the

producLon.

ì  Image
credit:
Yassal
Sundman


14

Release
Engineering
Principles
&
its
importance.

ì  Minimize
Cycle
Time
from
check-‐in
to
producLon
push.

ì  Everybody
is
responsible
for
the
Delivery
Process.

ì  ConLnuous
Improvement.

ì  AcLviLes
such
as
SCM,
Release
planning,
Automated

Deployment,
Acceptance
TesLng,
Performance
TesLng
are
NOT

SECONDARY.

ì  Generates
no
revenue
Lll
its
in
the
hands
of
the
end-‐users.

ì  Customer
SaLsfacLon
through
early
&
conLnuous
delivery
of

so<ware.


15

Release
Engineering
Steps

ì  SCM
(Git,
SVN,
Clearcase).

ì  Automated
Commit
/
Component
Builds
(Generate
Binaries).

ì  Generate
Code
Quality
Metrics.

ì  Build
Deployment
Pipeline.

ì  Automated
Acceptance
TesLng.

ì  Automated
PromoLon
to
subsequent
Environments.

ì  Single
click
or
automated
push
to
producLon.


16

Main
Focus
of
Release
Engineering

ì  Build
-‐>
Deploy
-‐>
Test
-‐>
Release

ì  Every
single
check-‐in
should
potenLally
be
in
a
releasable
state.


17

Configuration
Management

ì  Using
Version
Control
and
commit
everything.

ì  Check-‐in
the
changes
at
regular
frequency.

ì  Use
meaningful
messages
for
every
commit.

ì  Managing
Dependencies
(External
Libraries,
Components).

ì  Managing
the
Infrastructure
(Consistent
network
topology,
firewall,

OS
configuraLon,
patches
etc...
across
all
the
environments).

ì  Managing
the
Environments
&
the
tools
to
be
used
for
the
same.

ì  Managing
the
Change
Process.


18

Principles
of
Managing
Application

Configuration

ì  Good
understanding
of
the
stage
in
which
the
configuraLon
should

be
injected
(Assembly,
Deployment,
Restart
etc…)

sefngs
for
the
applicaLon
to
be
in
the
project’s
root

directory.

ì  Should
always
be
automated
and
values
checked
into
repository.

ì  Use
clear
naming
convenLon.

ì  Avoid
Over-‐engineering.
Keep
it
as
simple
as
possible.

ì  Ensure
that
the
configuraLon
is
tested
properly
a<er
deployment.


19

Release
Engineering
Architecture

ì  Its
all
about
building
the
“Build
&
Release”
pipeline.

ì  Built
using
Jenkins
or
similar
ConLnuous
IntegraLon
Tools.

ì  IllustraLon
of
a
“Delivery
Pipeline”
(Image
Courtesy:

“ConLnuous
Delivery”
by
Jez
Humble
&
David
Farley).


20

Deep
dive
into
commit
/
component
builds:

ì  Aim:

ì  Eliminate
builds
that
are
unﬁt
for
producLon.

ì  Steps:

ì  Compile
the
code.

ì  Run
a
set
of
commit
tests.

ì  Run
Lint
based
tests
or
some
of
the
basic
staLc
code
analysis
tasks.

ì  Publish
the
results.

ì  Once
above
steps
are
done,
build
the
package
and
upload
the
binary
to

a
centralized
repository.

ì  Add
tests
on
an
ongoing
basis.

ì  Keep
a
watch
on
the
build
execuLon
Lme
and
opLmize
frequently.

ì  Explore
the
‘Pre-‐Commit’
or
‘Pre-‐Flight’
build
opLons
provided
by
some

of
the
CI
tools.


21

Anti-‐Patterns
in
commit
/
component
builds:

ì  Don’t
check-‐in
new
code
on
a
broken
build.
The
only
fix
that
has
to
go
are
the
changes

that
fix
the
broken
build.

ì  Always
run
commit
tests
locally
before
the
commit.

ì  Always
wait
for
commit
tests
to
pass
before
next
check-‐in.

ì  Developers
who
have
commi.ed
their
code
are
responsible
for
the
build
process
to

succeed.

ì  Never
go
home
on
a
broken
build.

ì  Time-‐box
during
every
failed
build.
Give
about
10-‐20
minutes
to
fix,
else
rollback
the

changes.

ì  Don’t
EVER
comment
failing
tests.

ì  Take
ownership
for
all
the
breakages
that
result
from
your
changes
and
fix
them

accordingly.


22

Aim
of
Deployment
Pipeline

ì  Every
Step
should
be
visible
(Results
in
be.er
collaboraLon).

ì  Deliver
useful,
working
so<ware
to
the
users
as
early
as
possible.

ì  Early
feedback
(IdenLfy
&
resolve
issues
in
the
early
stages).

ì  Enable
teams
to
deploy
&
release
any
version
of
their
so<ware
at
any
point
to

any
of
the
environments.

ì  Avoid
last
day
heroics
on
the
release
day.

ì  Release
in
small
chunks
(Avoid
“Big
Bang”
release).

ì  Should
be
driven
by
tools
&
not
by
sviduals.

ì  Ability
for
every
change
to
be
transparent
&
propagate
them
through
the

pipeline.

ì  Ability
to
roll-‐back
to
a
previous
stable
state
in
case
of
any
failures.


23

Anti-‐Patterns
of
Deployment
Pipeline

ì  Manual
Deployment
of
So<ware.

ì  Deployment
performed
by
mulLple
teams.

ì  The
order
of
steps
are
not
deﬁned.

ì  No
proper
“Run
books”
for
failures.

ì  Too
much
of
reliance
of
manual
tesLng.

ì  CorrecLon
to
the
release
process
during
the
actual
producLon
release.

across
all
the
environments
varies
to
a
large
extent.

ì  Manual
ConﬁguraLon
management
of
ProducLon
Environment.

ì  Deployment
to
a
producLon-‐like
environment
is
done
only
development
is

100%
complete.


24

Best
qualities
of
a
Deployment
Pipeline

ì  Deployments
should
be
automated.

ì  Deployments
should
be
done
frequently
(If
possible,
for
every

single
check-‐in).

ì  Provide
early
feedback
so
that
the
development
team
can
act

on
the
failures
in
a
faster
way.

ì  Good
AutomaLon
Coverage
to
test
early.

ì  Should
be
scalable.

ì  Should
enable
one-‐click
deployment.


25

Deployment
on
Large
Scale
Production

Environments

ì  What
are
host-‐types?

ì  What
are
recipes
/
cookbooks?

ì  Tools
like
Pogo
OR
other
agent
based
deployment
tools.

$
ssh
<target-‐host-‐name>
'whoami;’

praneshv


26

Why
Automated
Deployment
is
an
indispensable
goal
???

ì  On
most
occasions,
the
deployment
documentaLon
is
outdated.

ì  Logging
all
the
steps
in
the
form
of
scripts
is
very
beneficial
for
book
keeping

purposes.

ì  Works
seamlessly
if
all
the
steps
are
taken
care
of.

ì  Even
non-‐experts
should
be
able
to
deploy
effortlessly.

ì  Every
team
using
automated
deployment
scripts
results
in
its
maturity
&
less

prone
to
errors.

ì  Take
off
the
dependency
from
the
deployment
expert
and
uLlize
them
for
more

challenging
tasks.

ì  Its
fast,
cheap.
Facilitates
in
early
tesLng
&
faster
feedback
cycles.

ì  Deployment
Logs
are
auditable
for
future
references.


27

Deep
dive
into
Testing
Quadrant

Image
Credit:
Concept
deﬁned
by
‘Brian
Marick’
(Diagram
obtained
from
“ConLnuous
Delivery”
by

‘Jez
Humble’
&
‘David
Farley’.


28

Deep
Dive
into
Automated
Acceptance
Tests

ì  Aim:

ì  Avoid
manual
tests
to
whatever
extent
possible.

ì  Steps
to
be
followed:

ì  Tests
to
be
performed
in
producLon-‐like
environment.

ì  If
sefng
up
environment
is
expensive,
use
a
scaled-‐down

environment.

ì  Setup
the
actual
user’s
environment
on
a
grid.

ì  Use
Business
language
in
the
scripts
instead
of
technical
jargon

(‘Place
Order’
instead
of
‘Clicking
on
bu.on
XYZ’).

ì  Well-‐deﬁned
exit
criteria
for
Pass
/
Fail
of
tests.

ì  Don’t
fail
the
test
due
to
minor
issues.


29

Deep
Dive
into
Automated
Acceptance
Tests

continued…

ì  Steps
to
be
followed:

ì  Include
a
few
tests
to
assert
external
systems.

ì  If
you
don’t
care
about
a
parLcular
ﬁeld,
don’t
bother
tesLng
it.

ì  These
tests
should
be
run
a<er
every
deployment.

ì  Block
the
pipeline
when
there
are
massive
failures.

ì  Parallelize
the
tests
to
whatever
extent
possible.

ì  Outcome
of
this
step
are
the
so<ware
packages
that
have
fought

against
all
the
tests
&
challenges
like
a
mythical
hero
and
ready

to
take
on
the
world
!!!


30

Few
more
Generic
Anti-‐Patterns

ì  TesLng
done
on
developer
machines.

ì  Service
Engineering
team
hasn’t
even
seen
the
applicaLon
Lll
the

actual
release
date.

ì  Installers,
ConfiguraLons
etc…
not
done
Lll
the
release
date.

ì  No
collaboraLon
between
development
team
&
SE
teams.

ì  Late
setup
of
Staging
Environment.

ì  Release
too
many
changes,
unable
to
figure
out
what
caused
the

failure.

ì  Changing
the
configuraLons
directly
on
ProducLon
o<en
resulLng
in

disasters.


31

Adaption
of
Release
Engineering

ì  Change
in
the
mindset
of
the
team
members.

ì  All
aspects
of
development,
tesLng,
staging,
producLon
environments
(most

importantly
conﬁguraLon
management)
to
be
managed
through
a
VCS.

ì  Integrate
development,
tesLng,
release
teams
as
a
part
of
the
development

process.

ì  Manage
the
Infrastructure
to
build
environments
in
no
Lme.

ì  Beef
up
the
test
automaLon
coverage
so
that
there’s
not
much
reliance
on

Manual
tesLng.

ì  Rehearse
the
release
&
rollback
process
so
that
its
easy
to
do
it
Lme
&

again.

ì  Reach
a
stage
wherein
“So<ware
Release”
is
in
self-‐service
mode.


32

Tools
Used
at
each
of
the
stages.

ì  Version
control
–
Git

ì  Build
Tools
–
Ant,
Nant,
MSBuild,
gmake,
Maven,
Buildr,
Psake.

ì  Agile
/
Scrum
–
Jira.

ì  StaLc
validaLon
–
Coverity,
SonarQube

ì  Unit
tesLng
–
junit

ì  Test
automaLon
–
Protractor,
Selenium

ì  ConLnuous
IntegraLon
–
Jenkins,
Atlassian
Bamboo,
Thoughtworks
Go,
UrbanCode

AntHillPro

ì  Deployment
–
Pogo,
Chef

ì  Environment
provisioning
–
Puppet,
Chef

ì  IAAS,
PAAS
–
AWS,
Azure,
Docker,
Vagrant


33

Tools
Used
at
each
of
the
stages.

ì  Image
Credit
:
h.p://maestrodev.com


34

Managing
Environments

ì  No
ApplicaLon
is
an
Island.

ì  Poor
ConﬁguraLon
Management
results
in
signiﬁcant
waste
of
Lme,

addiLonal
costs,
tech
debt
etc…

ì  Should
always
be
cheaper
to
create
a
new
environment
than
repairing
a

broken
environment.

ì  Few
of
the
names
used
for
Environments

ì  Development

ì  IntegraLon

ì  QA
/
TesLng

ì  Staging
/
Pre-‐ProducLon

ì  Performance

ì  Canary
/
Bucket
TesLng

ì  ProducLon


35

Managing
Releases
of
complex
systems


36

Image
Credit:
Slide
deck
available
at
h.ps://learn.chef.io/

Managing
Releases
of
complex
systems


37

OrchestraLon:

ì  Automated
arrangement,
coordinaLon,
and
management
of
complex
computer
systems,

middleware,
and
services.

ì  Aligning
the
business
request
with
the
applicaLons,
data,
and
infrastructure

ì  Deployment
Rhythm

ì  Its
about
gefng
the
perfect
configuraLon,
checking
in
these
values
and
automate
the

release
using
these
values.

ì  Have
the
right
sequence
of
steps
in
the
configuraLon
file.

ì  Have
the
right
kind
of
checks
&
balances
at
every
stage.

ì  PracLce
roll-‐backs
in
case
of
issues.

ì  Implement
fail-‐safe
methodologies
in
case
of
issues.

ì  Build
environments
wherein
the
ProducLon
traffic
can
be
replayed
(Use
of
access
logs
to

simulate
user
acLons).

Steps
involved
to
build
a
deployment
pipeline

ì  Model
your
value
stream
&
create
a
walking
skeleton.

ì  Automate
the
build
&
deployment
process.

ì  Automate
unit
tests
&
code
analysis.

ì  Automate
Acceptance
Tests.

ì  Automate
Releases.


38

Multi-‐Tenant
Applications

ì  Customers
share
the
same
cloud
plavorm
and
infrastructure
and
their
data

is
commingled.

ì  Single
code-‐base
for
the
applicaLon.

ì  Upgrades
are
executed
easily
by
the
SaaS
provider.

ì  All
the
tenants
are
using
the
same
version.

ì  Data
of
different
tenants
is
stored
in
the
same
place,
however,
measures

taken
to
make
it
inaccessible
to
other
tenants.

ì  BuckeLze
the
access
based
on
the
IP
Range
/
Cookie
range
/
Header

InformaLon
etc...

ì  Toggle
a
Feature
using
configuraLon.

ì  Hard
to
maintain
different
versions
for
different
tenants.


39

Multi-‐Instance
Applications

ì  MulLple
customers
run
their
own
separate
instance
of
an

applicaLon
and
OS
running
on
a
separate
VM.

ì  Avoid
comingling
of
data.

ì  Develop
&
use
their
own
logical
piece
of
the
cloud
service.

ì  Allows
for
greater
ﬂexibility
and
control
of
conﬁguraLon,

customizaLon,
updates
and
upgrades.

ì  Ability
to
migrate
instance
to
an
on-‐premise
server,
or
to

another
cloud
hosLng
provider


40

Zero
Downtime
Deployment

ì  What
is
‘Zero
DownLme
Deployment’?

ì  What
is
‘Out
Of
RotaLon’
(OOR)
?

ì  What
is
‘Concurrency’
?

ì  Colo
(Data-‐Center)
based
deployments.


41

Extracting
the
server
logs

ì  Standardize
the
logging
messages.
Involves
lot
of
eﬀort
from

the
development
teams.

ì  Setup
Splunk
alerts
for
‘FATAL’
errors
in
the
code.


42

Next
generation
tools
in
the
DevOps
world

ì  Chef

ì  Puppet

ì  Cfengine

ì  Salt


43

Preparation
for
the
Release

ì  Any
issues
during
the
release,
stop
or
postpone
the
release.

Stability
takes
the
highest
priority.

ì  Prepare
Release
Plan
or
Runbook
with
correct
sets
and
if

possible
automate
them.

ì  IdenLfy
the
error-‐prone
steps
and
automate
them.

ì  Perform
the
drill
for
performing
rolling
back
of
the
releases.

ì  Should
be
as
simple
as
selecLng
a
value
and
clicking
a
bu.on.

ì  Let
one
team
perform
all
the
releases.
(Too
many
cooks
spoil

the
broth).


44

Maturity
of
a
“Deployment
Pipeline”
?


45

ì  Image
Credit:
h.p://www.praqma.com/

Q
&
A


46

Thank
You


47

References:

ì  Details
about
each
of
the
phases
from
“ConLnuous
Delivery”
by
Jez
Humble,

David
Farley

ì  Source
of
some
of
the
generic
images
used
–
“Unknown”.

ì  RespecLve
credits
given
to
the
images
used
in
the
same
slide.

ì  h.ps://gigaom.com/2014/01/26/mulL-‐tenant-‐or-‐mulL-‐instance-‐cloud-‐lets-‐
do-‐both/

ì  h.p://diginomica.com/2013/12/20/mulL-‐tenant-‐mulL-‐instance-‐saas-‐
spectrum/

ì  h.ps://msdn.microso<.com/en-‐us/library/hh534478.aspx

ì  h.ps://www.youtube.com/watch?v=CE4hLYhfmBE
-‐
Deployment
using

pogo.


48

Build & Release Engineering

Recommended

More Related Content

What's hot (20)

Viewers also liked (10)

Similar to Build & Release Engineering (20)

Recently uploaded (20)

Build & Release Engineering