Testing in Production, Deploy on Fridays

技術在我們手上世界就在我們手上
Testing in Production,
Deploy on Fridays
曾義峰 (Ant)
yftzeng@gmail.com

曾義峰 (aka Ant)
➔
LeadBest Consulting Group 首席執行顧問
➔
TGO 台北分會學習委員
➔
臺灣資安社群 CHROOT 成員
➔
曾任資安顧問及電子票證公司顧問
➔
開源人年會 (COSCUP) 2009 及 2012 講師
➔
臺灣駭客年會 (HITCON) 2008 及 2009 講師
➔
臺灣 Modern Web 2015/2016/2017/2018/2019 講師

3/111
Deploy on ... Fridays

4/111
No
NoNoNO
Don't
Do
that
Image Credit : https://kknews.cc/pet/pqbxb8.html
Font Credit : https://www.dafont.com/theme.php?cat=109 (Defused)

5/111
Image Credit : https://www.getdigital.eu/deploy-on-friday.html
我應該在星期五下午 5 點部署嗎？
我只是想要 ( 做一點小變更 )...
不行
不行就是不行 ~

6/111
Image Credit : https://medium.com/openclassrooms-product-design-and-engineering/do-not-deploy-on-friday-92b1b46ebfe6

7/111
Image Credit : https://medium.com/openclassrooms-product-design-and-engineering/do-not-deploy-on-friday-92b1b46ebfe6
他們說在星期五進行部署一切都會沒事的 ...

8/111
Credit : https://twitter.com/kvlly/status/1116672656781266944

9/111
Credit : https://twitter.com/kvlly/status/1116672656781266944
I am not to blame, it is about (team) culture

10/111
Image Credit : https://thecodinglove.com/wishing-you-all-a-nice-friday
Team Discipline

11/111

12/111
Why

13/111
Confidence vs. Risk
&Trust
Why

14/111
Confidence vs. Risk
&Trust
Why
如果團隊害怕在星期五部署，是不是意謂著星期一、二、三、四的部署也同樣令人不安
如果你提供的是 24/7 全天候服務，那麼星期五與其他時間應該沒有什麼不同！

15/111
Confidence vs. Risk
&Trust
Why
如果團隊害怕在星期五部署，是不是意謂著星期一、二、三、四的部署也同樣令人不安
如果你提供的是 24/7 全天候服務，那麼星期五與其他時間應該沒有什麼不同！
團隊對部署有沒有信心，這才是重要的問題

Progressive Deployment & NoDeploy
曾義峰 (Ant)
yftzeng@gmail.com
2019-08-28
Credit : https://www.slideshare.net/yftzeng/progressive-deployment-nodeploy

17/111
Credit : https://www.slideshare.net/yftzeng/progressive-deployment-nodeploy
13,692 views

18/111
~Dark 1980s 2008 2014
ITIL
Design for Anti-FragilityDesign for Robustness Design for Resilience
DevOps
堅實性設計抗脆性設計復原性設計
精心設計。
嚴格管控。
故障被設計為需要異常
處理。
將開發及維運為一體。
視故障為正常狀態。
為故障發生設計隔離。
將維運視為工程。
最小化人為錯誤。
為故障復原至正常狀態。
轉型
轉型
進化
SRE Dev(ops)
Design for Deployless
Partial Credit : https://noops.connpass.com
NoDeploy
2019
想像
想像進化
無部署設計

19/111
Code Test Acceptance Deploy Operate
Continuous Integration / 持續整合
Continuous Delivery / 持續交付
Continuous Deployment / 持續部署
DevOps / SRE
Commit Dev Test Staging Test Deploy Prod
Continuous Deployment
Local Dev Staging Prod
Environments
Chaos
Test

20/111
DevOps / SRE
Image Credit : https://software.af.mil/training/devops/

21/111
DevOps / SRE
Deploy on Fridays

22/111
DevOps / SRE

23/111
Image Credit : https://www.freepik.com/free-icon/silo_869863.htm
Sales
Marketing
Dev Ops Compliance
(Security)
Value
DevOps
Cost Cost

24/111

25/111

26/111
Weekends

27/111
Weekends
20% of the week

28/111
No-Deploy-Fridays

29/111
No-Deploy-Fridays
No-Merge-Fridays

30/111
No-Deploy-Fridays
No-Merge-Fridays
Conflicts ⬆

31/111
No-Deploy-Fridays
No-Merge-Fridays
Conflicts ⬆
Trunk
Trunk
PUSH
⬇️
No-Merge-Fridays
PUSH
⬇️
PUSH
⬇️
PUSH
⬇️
PUSH
⬇️
PUSH
⬇️
PUSH
⬇️
PUSH
⬇️
PUSH
⬇️
time

32/111
No-Deploy-Fridays
No-Merge-Fridays
Conflicts ⬆
Image Credit : https://www.git-tower.com/help/guides/branches-and-tags/merge-conflicts/windows

33/111
No-Deploy-Fridays
No-Merge-Fridays
Conflicts ⬆ Lead Time ⬆

34/111
No-Deploy-Fridays
No-Merge-Fridays
Image Credit : https://www.qualysoft.com/en/blog/tech-blog-introduction-part-2-cicd-patterns-and-best-practices

35/111
No-Deploy-Fridays
No-Merge-Fridays
Image Credit : https://www.qualysoft.com/en/blog/tech-blog-introduction-part-2-cicd-patterns-and-best-practices

36/111
No-Deploy-Fridays
No-Merge-Fridays
Conflicts ⬆ Lead Time ⬆ Problems Debug⬆ ⬇

37/111
No-Deploy-Fridays
No-Merge-Fridays
Conflicts ⬆ Lead Time ⬆ Problems Debug⬆ ⬇
Small
Commits
Release
Big
Commits
Release
Vs.

38/111
Image Credit : https://doc.oroinc.com/cloud/environments/
Image Credit : https://oroinc.com/b2b-ecommerce/blog/testing-and-staging-environments-in-ecommerce-implementation/

39/111
Image Credit : https://doc.oroinc.com/cloud/environments/
Image Credit : https://oroinc.com/b2b-ecommerce/blog/testing-and-staging-environments-in-ecommerce-implementation/

40/111
Image Credit : https://www.slideshare.net/cPrime/kanban-testing-54239643
Image Credit : https://jaxenter.com/deployment-best-practices-172662.html
Image Credit : https://www.slideshare.net/LaunchDarkly/the-staging-server-is-dead-long-live-the-staging-server

41/111

42/111

43/111
這些網站都告訴我們應該這麼做，好像世界就應該要長成這個樣子

44/111
這些網站都告訴我們應該這麼做，好像世界就應該要長成這個樣子
缺點是什麼？

45/111
Staging
Staging 環境必須與 Production 環境幾乎
相同，這十分困難，否則測試就不準了。
Trying to mirror your staging
environment to production is a fool’s
errand. Just give up.
嘗試將 Staging 視為 Production 是一個愚蠢的事。放棄吧。
Partial Credit : https://thenewstack.io/honeycombs-charity-majors-go-ahead-test-in-production/

46/111
Credit : https://www.infoq.cn/article/b9lG4O8oJ8bWcFFQfFKE
Staging

47/111
Image Credit : Progressive Deployment, Experimentation, Multitenancy, No Downtime, Cloud Security (2018-02-23).pdf
Microsoft/Azure

48/111Image Credit : https://jaxenter.com/deployment-best-practices-172662.html
Image Credit : https://www.reddit.com/r/ProgrammerHumor/comments/ing3jx/works_on_my_machine/
Image Credit : https://www.pngitem.com/middle/hwwmRmi_dove-clipart-flame-clipart-flame-png-transparent-png/
在我電腦環境就沒問題

50/111
大膽的想法
純真的想法

51/111
Testing in Production
在 Microservices 環境中，尤其是 FaaS 環境中，
Testing in production 是一種實際的需要。

52/111
Chaos Engineering @ Netflix
這些是需要整合測試的故障模式示例，因為它們涉及服務之間的互動。在某
些情況下，可能只能在 Production 進行完全整合測試。在 Netflix ，根本不
可能完全重現整個架構並運行端到端測試。
當可以在 Testing 環境中重現整個系統時，我們仍然相信需要在 Production
運行實驗，因為在 Testing 環境中永遠不可能完全重現系統的所有方面。總
是存在差異，例如合併客戶端與真實客戶端相比的行為方式或 DNS 配置問
題。
Credit : Chaos Engineering IEEE Software (2016-05-05).pdf

53/111
“
If you aren't testing in prod
you aren't testing in reality
”
如果不在正式環境測試，
你的測試就不會是真實的
Credit : https://twitter.com/mipsytipsy/status/1147863838647185408
Image Credit : https://store.steampowered.com/app/561970/REALITY/

54/111
“
Dev will match Prod,
if you develop on production
”
如果在正式環境上進行開發，
開發環境就會與正式環境相稱
Credit : Blue is the new green (2019-03-26).pdf
Image Credit : https://tw.voicetube.com/videos/62711

55/111
Your aim won’t be perfect,
control over the blast radius.
你的瞄準並不完美，
請控制好爆炸半徑。
Image Credit : https://www.behance.net/gallery/57558081/FREE-BUSINESS-ICONS
Progressive Deployment

56/111Credit : https://thenewstack.io/the-rise-of-progressive-delivery-for-systems-resilience/
Image Credit : https://www.trisignup.com/Race/24568/Sponsor/94608
Progressive deployment is continuous deploy
with fine-grained control over the blast radius.
漸進式部署是對爆破半徑進行細部控制的連續部署。

57/111Credit : https://thenewstack.io/the-rise-of-progressive-delivery-for-systems-resilience/
Image Credit : https://support.google.com/optimize/answer/7012154?hl=en
Deploying a service is not that same as activating it for all users.
部署一個服務，未必需要讓所有使用者皆同時生效。

58/111
DevOps / SRE

59/111
DevOps / SRE

60/111
Code Test Acceptance Deploy Release
Continuous Deployment + Feature Flag / 持續部署 + 功能標誌
Operate
DevOps / SRE

61/111
Code Test Acceptance Deploy Release
Continuous Deployment + Feature Flag / 持續部署 + 功能標誌
Operate
DevOps / SRE
Code Push ➡
Deployment
Developer
Production
Server User
Feature Flaged ➡
Release

62/111
Feature Flag
01 所有程式皆為部署，並可控制開關，減少整合問題
All code is deployed, but control exposure, reduces integration issue.
02 控制細緻度可至特定層級或是開發分支
Flags provide runtime control down to individual user.
03 開發者可以增加或刪除功能，無需再次部署
Users can be added or removed with no redeployment
04 暗啟動
Enables dark launch
Credit : Progressive Deployment, Experimentation, Multitenancy, No Downtime, Cloud Security (2018-02-23).pdf
Image Credit : https://opensource.com/article/18/2/feature-flags-ring-deployment-model

Cost in Time
時間成本
hours
mins ~ hour
secs
~
mins
Applicability
適用性
2%
8%
90%
Toggle
Feature Flag
ReDeploy
Previous Version
Rollback
Revert Commit
Incident Recovery
事件恢復

64/111
Why
⬇
How

65/111
01 Feature Flag with Story Mapping
功能標誌及使用者故事對照
02 Progressive Deployment
漸進式部署
03 Data(base) Synchronization Strategy
資料 ( 庫 ) 同步策略
04 Pull Requests to Pair/Mob Programming
從 Pull Requests 轉為 Pair/Mob Programming
05 Service Mesh
服務網格
Testing in Production, Deploy on Fridays

66/111
漸進式部署
05 Service Mesh
服務網格

67/111
Group = Beta
Group = Normal
If ( group = “beta” ) {
...
}
else {
...
}
Beta
Normal
By User Group
TimeURIIP AddressUser Group

68/111
IP = o.o.o.o
IP = x.x.x.x
If ( IP = “o.o.o.o” ) {
...
}
else {
...
}
Beta
By IP Address
Normal

69/111
URI ~ ?flag=1
If ( URI ~ “flag=1” ) {
...
}
else {
...
}
Beta
Normal
By URI

70/111
URI ~ 20201005
If ( URI ~“20201005”) {
...
}
else {
...
}
Beta
Normal
By Time

71/111
Story Mapping
Code Test Acceptance Deploy Release Done
#19
Fix
Login Bug
#20
Fix
Email Bug
#21
Perf.
Tuning
#17
Sort By
Date
#18
Filter By
Group
#22
German
i18n

72/111
Story Mapping
#19
Fix
Login Bug
#20
Fix
Email Bug
#21
Perf.
Tuning
#17
Sort By
Date
Flag#3
#18
Filter By
Group
Flag#4
#22
German
i18n
Flag#5

73/111
Story Mapping
#19
Fix
Login Bug
#20
Fix
Email Bug
#21
Perf.
Tuning
#17
Sort By
Date
Flag#3
#18
Filter By
Group
Flag#4
#22
German
i18n
Flag#5
#17
Sort By
Date
Flag#3

74/111
#22
German
i18n
Flag#5
Story Mapping
#19
Fix
Login Bug
#20
Fix
Email Bug
#21
Perf.
Tuning
#17
Sort By
Date
Flag#3
#18
Filter By
Group
Flag#4

75/111
01 提交程式至主幹可能會影響其他功能
Commit code on trunk may affects other features.
02 未完成的功能可能也會部署，如果配置有誤可能將未完成的功能開啟
Unfinished features may be deployed, and turned on if the configuration is incorrect.
Image Credit : https://opensource.com/article/18/2/feature-flags-ring-deployment-model
Feature Flag

76/111
#22
German
i18n
Flag#5
Story Mapping
#19
Fix
Login Bug
#20
Fix
Email Bug
#21
Perf.
Tuning
#17
Sort By
Date
Flag#3
#18
Filter By
Group
Flag#4

77/111
漸進式部署
05 Service Mesh
服務網格

78/111
Facebook
Image Credit : https://engineering.fb.com/web/rapid-release-at-massive-scale (2017-08-31)

79/111
Microsoft/Azure
Image Credit : Progressive Deployment, Experimentation, Multitenancy, No Downtime, Cloud Security (2018-02-23).pdf

80/111
漸進式部署
05 Service Mesh
服務網格

81/111
Image Credit : https://blog.codonomics.com/2020/08/multi-tenant-architectures.html
Multi-tenant Architectures ( 多租戶架構 )
Shared Database Separate Database

82/111

83/111

84/111
Disable DROP/TRUNCATE/DELETE
Soft Delete

85/111

86/111
Code Push ➡
5 mins
⬅ Data Sync
? mins
Code Push ➡
15 mins
Code Push ➡
15 mins
⬅ Data Sync
? mins
⬅ Data Sync
? mins
Easy
Hard

87/111
Image Credit : https://memezila.com/Pizza-rolls-in-the-oven-vs-in-the-microwave-meme-3854

88/111
Image Credit : https://twitter.com/GitPushAll/status/1309533288567058434

89/111
漸進式部署
05 Service Mesh
服務網格

90/111
Image Credit : https://www.atlassian.com/blog/bitbucket/5-pull-request-must-haves
Pull Requests

91/111
Pull Requests
Code Reviews are great
But slow pipeline

92/111
Pull Requests
Code Reviews are great
But slow pipeline
Code Reviews Latency

93/111
Image Credit : https://blog.submain.com/code-review-vs-pair-programming-2/
Pair Programming

94/111
Image Credit : https://blog.submain.com/code-review-vs-pair-programming-2/
Pair Programming
Code Reviews Inline

95/111
Image Credit : https://team-coder.com/mob-programming/
Mob Programming

96/111
Image Credit : https://team-coder.com/mob-programming/
Mob Programming
Continuous Code Reviews Inline

97/111
漸進式部署
05 Service Mesh
服務網格

98/111
Image Credit : https://docs.microsoft.com/zh-tw/dotnet/architecture/cloud-native/service-mesh-communication-infrastructure
Service Mesh

99/111
漸進式部署
05 Service Mesh
服務網格

100/111

101/111

102/111
Reality ⬆
Cost ⬇
Maintenance ⬇

103/111
Image Credit : https://twitter.com/reillyusa/status/1260272851778768903

104/111

105/111
SRE≠ 呂布 (Reboot)

Confidence & Trust ⬆

109/111
Image Credit : https://memes.tw/collection/388

110/111
yftzeng@gmail.com
https://www.facebook.com/yftzeng.tw
https://twitter.com/yftzeng
曾義峰 (Ant)

曾義峰 (Ant)
LeadBest Consulting Group 首席執行顧問
yftzeng@gmail.com

Testing in Production, Deploy on Fridays

More Related Content

What's hot (20)

Similar to Testing in Production, Deploy on Fridays (20)

More from Yi-Feng Tzeng (20)

Testing in Production, Deploy on Fridays