SlideShare a Scribd company logo
© 2019 Trend Micro Inc.1
Container and Test
Automation
Management Practices in
Trend Micro
Larry Huang
© 2019 Trend Micro Inc.2
Abstract
• Introduction
• CD Practice
• TestRail and Auto Test Case Management
– Test case auto sync
– Auto test run creation
• Container + k8s Practice
– Benefits
– CI/CD
– Logging
– Monitoring
– Testing
© 2019 Trend Micro Inc.3
Self Introduction
• Larry Huang
– Experience
• Dev
• QA
– Trend Micro PLS
• Lepus team
© 2019 Trend Micro Inc.4
Lepus Team
© 2019 Trend Micro Inc.5
Service by Our Team
• Web Services
– API
– Frontend
– Backend
© 2019 Trend Micro Inc.6
QA Routine
• Testing
• Case Support
• Deployment
– QA (inner integration)
– Beta (external integration)
– Stage (external integration)
– Production
© 2019 Trend Micro Inc.7
Deployment Tool - Octopus
 Process control pipeline
 Permission control
 Project dashboard
 Variable management
 Plugins
 Cross Platform
 GUI
© 2019 Trend Micro Inc.8
Deployment Tool - Octopus
• Octopus Process
– target selection
– env selection
• Common Steps
– API or DB connection
– Deploy
– DVT Test
– Teardown
© 2019 Trend Micro Inc.9
Configuration Deployment
• Frequent configuration update request
• Independent deploy process
• Light weight
– Less effort for Dev and QA
© 2019 Trend Micro Inc.10
TestRail and Automation Test Integration
© 2019 Trend Micro Inc.11
TestRail
• Case management
• Test report
© 2019 Trend Micro Inc.12
Automation Testing Framework
• Robot Framework
– Python Based
– Keyword
– Tags for filtering
• Case category
• TestRail Case id
© 2019 Trend Micro Inc.13
© 2019 Trend Micro Inc.14
Case Synchronization
On git change
© 2019 Trend Micro Inc.15
Case Synchronization
TestRail Section Path
Case ID
© 2019 Trend Micro Inc.16
Case Synchronization
• Sync target
– Case Title
– Case ID
– Customized Attributes
• Sync options
– direction (From robot or TestRail)
– mode (Create or delete cases if not existed)
© 2019 Trend Micro Inc.17
Case Skeleton Creation
• Generate a robot file with the skeletons of the test cases
– Title
– ID, customized attributes
– Steps as comments
– Force Fail
© 2019 Trend Micro Inc.18
Auto Test Plan (Run)
• Test report before going production
© 2019 Trend Micro Inc.19
Auto Test Run
Create Test Plan
Create Test Run
Update Test Results
© 2019 Trend Micro Inc.20
Container + k8s Practice
Before Containerization
Load Balancer
Machine 1 Machine 2 Machine 4Machine 3
Load Balancer
Machine 1 Machine 2 Machine 4Machine 3
Tech Operator
© 2019 Trend Micro Inc.22
Before Containerization
• Server Provision
1 or 2 weeks later …
Request for new server
Tech Operator
CentOS 7
Python 3.6
Python 2.7 ?
© 2019 Trend Micro Inc.23
Containerization
• Infra as code
Service A
Service B
Service C
YAML Files
© 2019 Trend Micro Inc.24
Deployment Requirements
© 2019 Trend Micro Inc.25
Before Containerization
• Rolling Update
– Old and new version might be published in the same time
– Bind and Unbind
– Time consuming
Tech Operator
QA
Please unbind machine group 1
Group 1 has been unbound
3 min later …
QA Deploy and test group 1 …
Please bind machine group 1
Group 1 has been bound
3 min later …
…
© 2019 Trend Micro Inc.26
Containerization
• Blue Green Deployment
– Time Saving
• Switching pod selector
– Easy to recover
• Scale up/down pod replica k8s service
pod selector
Replica = 2
Replica = 0
© 2019 Trend Micro Inc.27
Before Containerization
• Single Point Test
– Every machine needs to be checked individually
© 2019 Trend Micro Inc.28
Containerization
• No need for Single Point Test
– All pods are replicas
• k8s Configmap
– Light weight deploy
– Central Management
• Only one replica
Configmap A
© 2019 Trend Micro Inc.29
Containerization
• Auto Scaling
• Health Assurance
k8s deployment
Replica = 4
Running RunningRunning Exit Container Creating
© 2019 Trend Micro Inc.30
Containerization
• Service / Machine
– Microservice Friendly
– Light way deployment
– Cost down (10 -> 2 nodes)
Service A
Service B
Service C
Standby
© 2019 Trend Micro Inc.31
• Each environment needs individual Cname
Before Containerization
service-stage.trendmicro.comservice-beta.trendmicro.com
Service A Service B Service C Service A Service B Service C
© 2019 Trend Micro Inc.32
• Ingress
– No need to build individual k8s for every environment
– Services are mapped by routes
• Dev/QA
• Beta/Stage
• Production
Containerization
service-test.iam.trend.com
/beta/login/ /stage/login/
© 2019 Trend Micro Inc.33
k8s Secrets
• sensitive data
© 2019 Trend Micro Inc.34
CI Flow
• CI
– Docker Image
• Product Image = Base Image + App Image
• Base Image
– OS
– System Library
• App Image
– Source Code
– App Library
– Version control
Base
Product
© 2019 Trend Micro Inc.35
Deployment Process
Environment Check
Create pods to test connection to external services (DB, API, …)
Blue (Green) Deploy
DVT Test
Pod Switch
© 2019 Trend Micro Inc.36
CD Environment
• Dev/QA
– k8s
• Beta/Stage
– AWS EKS
• Production
– AWS EKS
• Any k8s compatible environment can be deployed
© 2019 Trend Micro Inc.37
Lesson Learn
• Node Selection
– Connection super slow
VPN
© 2019 Trend Micro Inc.38
• Replica or Deployment
– RestartPolicy: Always
• Job
– RestartPolicy: OnFailure or Never
Lesson Learn
© 2019 Trend Micro Inc.39
• (App configuration + Deployment yaml) * service * env
• Solution
– Config change review
Lesson Learn
YAML Files
© 2019 Trend Micro Inc.40
Logging
• Log cannot not be preserved in container
• EKS (Beta to Production): Gray Log
• K8s (Dev/QA):
– Mount log folder on k8s Volumes
– Cronjob sync to NAS
K8s volume
App
cronjob
© 2019 Trend Micro Inc.41
Monitoring
• Monitis
– E2E case
• User scenario
• Response time
© 2019 Trend Micro Inc.42
Monitoring
• Prometheus + Grafana
– Resource
• CPU
• Memory
• Pod Restart Count
– Slack Channel Integration
• Warning
• Recovering
© 2019 Trend Micro Inc.43
Monitoring
Monitoring
© 2019 Trend Micro Inc.45
• Three environments to maintain
• Shared by multiple teams
– Computing Resources
– Libraries
– Browser
Test using Traditional Way
Test Client 1 Test Client 2 Test Client 3
© 2019 Trend Micro Inc.46
• Dockerfile tells everything
– Easy to replicate
• No worry about resource dependency
• Python migration to 3
• Selenium Grid Enabled
Test in Container
© 2019 Trend Micro Inc.47
Selenium Grid
74 73
• Parallel Test
© 2019 Trend Micro Inc.48
Summary
• YAML: code as infrastructure
• Replica: HA, Auto Scaling
• Ingress: Routing mapping service
• Pod Selection: Blue Green Deploy
• Volume: Log preservation
• Prometheus + Grafana: Monitoring
• Test in Container
© 2019 Trend Micro Inc.49
Thank You
Ad

Recommended

AllTheTalks 2020: "The Past, Present, and Future of Cloud Native API Gateways"
AllTheTalks 2020: "The Past, Present, and Future of Cloud Native API Gateways"
Daniel Bryant
 
DevOpsCon 2020: The Past, Present, and Future of Cloud Native API Gateways
DevOpsCon 2020: The Past, Present, and Future of Cloud Native API Gateways
Daniel Bryant
 
SoftwareCircus 2020 "The Past, Present, and Future of Cloud Native API Gateways"
SoftwareCircus 2020 "The Past, Present, and Future of Cloud Native API Gateways"
Daniel Bryant
 
20151019 v mworld2015-recap-02
20151019 v mworld2015-recap-02
Kevin Groat
 
Docker Orchestrators
Docker Orchestrators
Andrew Sullivan
 
[QCon London 2020] The Future of Cloud Native API Gateways - Richard Li
[QCon London 2020] The Future of Cloud Native API Gateways - Richard Li
Ambassador Labs
 
UrbanCode Deploy course and product overview slides
UrbanCode Deploy course and product overview slides
IBM Rational software
 
SaaS company in north america
SaaS company in north america
Gabe Akisanmi
 
Orchestrating Complex Multi Cloud Enterprise Applications
Orchestrating Complex Multi Cloud Enterprise Applications
Cloudify Community
 
Orchestrating Cloud-Native and Traditional Application Architectures
Orchestrating Cloud-Native and Traditional Application Architectures
Apprenda
 
Serverless design with Fn project
Serverless design with Fn project
Siva Rama Krishna Chunduru
 
Urban code deploy helps with traditional websphere app server migration
Urban code deploy helps with traditional websphere app server migration
Laurel Dickson-Bull
 
In-Stream Processing Service Blueprint, Reference architecture for real-time ...
In-Stream Processing Service Blueprint, Reference architecture for real-time ...
Grid Dynamics
 
Cloud Economics - Crayon Optimization Services
Cloud Economics - Crayon Optimization Services
Anfernee Bonds
 
Driving success in the cloud with NGINX
Driving success in the cloud with NGINX
NGINX, Inc.
 
Continuous Performance Testing
Continuous Performance Testing
Grid Dynamics
 
Docker12 factor
Docker12 factor
John Zaccone
 
Cloud-Native Data: What data questions to ask when building cloud-native apps
Cloud-Native Data: What data questions to ask when building cloud-native apps
VMware Tanzu
 
Using Git with Rational Team Concert and Rational ClearCase in enterprise env...
Using Git with Rational Team Concert and Rational ClearCase in enterprise env...
Bartosz Chrabski
 
Migrate Heroku & OpenShift Applications to IBM BlueMix
Migrate Heroku & OpenShift Applications to IBM BlueMix
Rohit Kelapure
 
Building a PaaS Platform like Bluemix on OpenStack
Building a PaaS Platform like Bluemix on OpenStack
Animesh Singh
 
CollabNet TeamForge 8.1 Webinar
CollabNet TeamForge 8.1 Webinar
CollabNet
 
Run Stateful Apps on Kubernetes with VMware PKS - Highlight WebLogic Server
Run Stateful Apps on Kubernetes with VMware PKS - Highlight WebLogic Server
Simone Morellato
 
Kovair Omnibus Integration with Multi Vendor Tools
Kovair Omnibus Integration with Multi Vendor Tools
Kovair
 
NI Package Manager
NI Package Manager
DMC, Inc.
 
Kovair ALM application on model based process improvement
Kovair ALM application on model based process improvement
Kovair
 
VMware - Application Portability
VMware - Application Portability
VMUG IT
 
VMware Developer-Ready Transformation
VMware Developer-Ready Transformation
VMware Tanzu
 
Ship code like a keptn
Ship code like a keptn
Rob Jahn
 
Building Autonomous Operations for Kubernetes with keptn
Building Autonomous Operations for Kubernetes with keptn
Johannes Bräuer
 

More Related Content

What's hot (20)

Orchestrating Complex Multi Cloud Enterprise Applications
Orchestrating Complex Multi Cloud Enterprise Applications
Cloudify Community
 
Orchestrating Cloud-Native and Traditional Application Architectures
Orchestrating Cloud-Native and Traditional Application Architectures
Apprenda
 
Serverless design with Fn project
Serverless design with Fn project
Siva Rama Krishna Chunduru
 
Urban code deploy helps with traditional websphere app server migration
Urban code deploy helps with traditional websphere app server migration
Laurel Dickson-Bull
 
In-Stream Processing Service Blueprint, Reference architecture for real-time ...
In-Stream Processing Service Blueprint, Reference architecture for real-time ...
Grid Dynamics
 
Cloud Economics - Crayon Optimization Services
Cloud Economics - Crayon Optimization Services
Anfernee Bonds
 
Driving success in the cloud with NGINX
Driving success in the cloud with NGINX
NGINX, Inc.
 
Continuous Performance Testing
Continuous Performance Testing
Grid Dynamics
 
Docker12 factor
Docker12 factor
John Zaccone
 
Cloud-Native Data: What data questions to ask when building cloud-native apps
Cloud-Native Data: What data questions to ask when building cloud-native apps
VMware Tanzu
 
Using Git with Rational Team Concert and Rational ClearCase in enterprise env...
Using Git with Rational Team Concert and Rational ClearCase in enterprise env...
Bartosz Chrabski
 
Migrate Heroku & OpenShift Applications to IBM BlueMix
Migrate Heroku & OpenShift Applications to IBM BlueMix
Rohit Kelapure
 
Building a PaaS Platform like Bluemix on OpenStack
Building a PaaS Platform like Bluemix on OpenStack
Animesh Singh
 
CollabNet TeamForge 8.1 Webinar
CollabNet TeamForge 8.1 Webinar
CollabNet
 
Run Stateful Apps on Kubernetes with VMware PKS - Highlight WebLogic Server
Run Stateful Apps on Kubernetes with VMware PKS - Highlight WebLogic Server
Simone Morellato
 
Kovair Omnibus Integration with Multi Vendor Tools
Kovair Omnibus Integration with Multi Vendor Tools
Kovair
 
NI Package Manager
NI Package Manager
DMC, Inc.
 
Kovair ALM application on model based process improvement
Kovair ALM application on model based process improvement
Kovair
 
VMware - Application Portability
VMware - Application Portability
VMUG IT
 
VMware Developer-Ready Transformation
VMware Developer-Ready Transformation
VMware Tanzu
 
Orchestrating Complex Multi Cloud Enterprise Applications
Orchestrating Complex Multi Cloud Enterprise Applications
Cloudify Community
 
Orchestrating Cloud-Native and Traditional Application Architectures
Orchestrating Cloud-Native and Traditional Application Architectures
Apprenda
 
Urban code deploy helps with traditional websphere app server migration
Urban code deploy helps with traditional websphere app server migration
Laurel Dickson-Bull
 
In-Stream Processing Service Blueprint, Reference architecture for real-time ...
In-Stream Processing Service Blueprint, Reference architecture for real-time ...
Grid Dynamics
 
Cloud Economics - Crayon Optimization Services
Cloud Economics - Crayon Optimization Services
Anfernee Bonds
 
Driving success in the cloud with NGINX
Driving success in the cloud with NGINX
NGINX, Inc.
 
Continuous Performance Testing
Continuous Performance Testing
Grid Dynamics
 
Cloud-Native Data: What data questions to ask when building cloud-native apps
Cloud-Native Data: What data questions to ask when building cloud-native apps
VMware Tanzu
 
Using Git with Rational Team Concert and Rational ClearCase in enterprise env...
Using Git with Rational Team Concert and Rational ClearCase in enterprise env...
Bartosz Chrabski
 
Migrate Heroku & OpenShift Applications to IBM BlueMix
Migrate Heroku & OpenShift Applications to IBM BlueMix
Rohit Kelapure
 
Building a PaaS Platform like Bluemix on OpenStack
Building a PaaS Platform like Bluemix on OpenStack
Animesh Singh
 
CollabNet TeamForge 8.1 Webinar
CollabNet TeamForge 8.1 Webinar
CollabNet
 
Run Stateful Apps on Kubernetes with VMware PKS - Highlight WebLogic Server
Run Stateful Apps on Kubernetes with VMware PKS - Highlight WebLogic Server
Simone Morellato
 
Kovair Omnibus Integration with Multi Vendor Tools
Kovair Omnibus Integration with Multi Vendor Tools
Kovair
 
NI Package Manager
NI Package Manager
DMC, Inc.
 
Kovair ALM application on model based process improvement
Kovair ALM application on model based process improvement
Kovair
 
VMware - Application Portability
VMware - Application Portability
VMUG IT
 
VMware Developer-Ready Transformation
VMware Developer-Ready Transformation
VMware Tanzu
 

Similar to Container and Test Automation Management Practices in TrendMicro (20)

Ship code like a keptn
Ship code like a keptn
Rob Jahn
 
Building Autonomous Operations for Kubernetes with keptn
Building Autonomous Operations for Kubernetes with keptn
Johannes Bräuer
 
Containerize your Blackbox tests
Containerize your Blackbox tests
Kevin Beeman
 
Chicago DevOps Meetup Nov2019
Chicago DevOps Meetup Nov2019
Mike Villiger
 
Automating Image build to ensure compliance, metrics, and auditing, in Multi-...
Automating Image build to ensure compliance, metrics, and auditing, in Multi-...
Evaldo Felipe
 
Automated Deployment & Benchmarking with Chef, Cobbler and Rally for OpenStack
Automated Deployment & Benchmarking with Chef, Cobbler and Rally for OpenStack
NTT Communications Technology Development
 
Deploying Kubernetes in the Enterprise (IBM #Think2019 #7678 Tech Talk)
Deploying Kubernetes in the Enterprise (IBM #Think2019 #7678 Tech Talk)
Michael Elder
 
Continuous Delivery With Containers
Continuous Delivery With Containers
All Things Open
 
From Zero to DevOps Superhero: The Container Edition (JenkinsWorld SF)
From Zero to DevOps Superhero: The Container Edition (JenkinsWorld SF)
Jessica Deen
 
Go, Swarm and DevOps vs The Mighty Monolith
Go, Swarm and DevOps vs The Mighty Monolith
Igor Karpovich
 
State of Union - Containerz
State of Union - Containerz
Shiva Narayanaswamy
 
Cloud Native Summit 2019 Summary
Cloud Native Summit 2019 Summary
Everett Toews
 
Evolving Infrastructure
Evolving Infrastructure
louisadunne
 
OpenStack Operations Guide 1st Edition Tom Fifield
OpenStack Operations Guide 1st Edition Tom Fifield
tuekamrasyid
 
Session 3 - CloudStack Test Automation and CI
Session 3 - CloudStack Test Automation and CI
tcloudcomputing-tw
 
Continuous Integration and Delivery using TeamCity and Jenkins
Continuous Integration and Delivery using TeamCity and Jenkins
Mahmoud Ali
 
Openstack Operations Guide 1st Edition Tom Fifield Diane Fleming
Openstack Operations Guide 1st Edition Tom Fifield Diane Fleming
molaxmeizu
 
Xpdays: Kubernetes CI-CD Frameworks Case Study
Xpdays: Kubernetes CI-CD Frameworks Case Study
Denys Vasyliev
 
Introduction to DevOps and the Practical Use Cases at Credit OK
Introduction to DevOps and the Practical Use Cases at Credit OK
Kriangkrai Chaonithi
 
Completing the Microservices Puzzle: Kubernetes, Prometheus and FreshTracks.io
Completing the Microservices Puzzle: Kubernetes, Prometheus and FreshTracks.io
CA Technologies
 
Ship code like a keptn
Ship code like a keptn
Rob Jahn
 
Building Autonomous Operations for Kubernetes with keptn
Building Autonomous Operations for Kubernetes with keptn
Johannes Bräuer
 
Containerize your Blackbox tests
Containerize your Blackbox tests
Kevin Beeman
 
Chicago DevOps Meetup Nov2019
Chicago DevOps Meetup Nov2019
Mike Villiger
 
Automating Image build to ensure compliance, metrics, and auditing, in Multi-...
Automating Image build to ensure compliance, metrics, and auditing, in Multi-...
Evaldo Felipe
 
Automated Deployment & Benchmarking with Chef, Cobbler and Rally for OpenStack
Automated Deployment & Benchmarking with Chef, Cobbler and Rally for OpenStack
NTT Communications Technology Development
 
Deploying Kubernetes in the Enterprise (IBM #Think2019 #7678 Tech Talk)
Deploying Kubernetes in the Enterprise (IBM #Think2019 #7678 Tech Talk)
Michael Elder
 
Continuous Delivery With Containers
Continuous Delivery With Containers
All Things Open
 
From Zero to DevOps Superhero: The Container Edition (JenkinsWorld SF)
From Zero to DevOps Superhero: The Container Edition (JenkinsWorld SF)
Jessica Deen
 
Go, Swarm and DevOps vs The Mighty Monolith
Go, Swarm and DevOps vs The Mighty Monolith
Igor Karpovich
 
Cloud Native Summit 2019 Summary
Cloud Native Summit 2019 Summary
Everett Toews
 
Evolving Infrastructure
Evolving Infrastructure
louisadunne
 
OpenStack Operations Guide 1st Edition Tom Fifield
OpenStack Operations Guide 1st Edition Tom Fifield
tuekamrasyid
 
Session 3 - CloudStack Test Automation and CI
Session 3 - CloudStack Test Automation and CI
tcloudcomputing-tw
 
Continuous Integration and Delivery using TeamCity and Jenkins
Continuous Integration and Delivery using TeamCity and Jenkins
Mahmoud Ali
 
Openstack Operations Guide 1st Edition Tom Fifield Diane Fleming
Openstack Operations Guide 1st Edition Tom Fifield Diane Fleming
molaxmeizu
 
Xpdays: Kubernetes CI-CD Frameworks Case Study
Xpdays: Kubernetes CI-CD Frameworks Case Study
Denys Vasyliev
 
Introduction to DevOps and the Practical Use Cases at Credit OK
Introduction to DevOps and the Practical Use Cases at Credit OK
Kriangkrai Chaonithi
 
Completing the Microservices Puzzle: Kubernetes, Prometheus and FreshTracks.io
Completing the Microservices Puzzle: Kubernetes, Prometheus and FreshTracks.io
CA Technologies
 
Ad

More from Jen-Chieh Ko (20)

Agile.Taipei 2025.1月聚會 - AI加持的人本軟體開發未來, Terry
Agile.Taipei 2025.1月聚會 - AI加持的人本軟體開發未來, Terry
Jen-Chieh Ko
 
DevOpsDays Taipei 2024 以系統思考來討論 LeSS 如何處理多頭馬車
DevOpsDays Taipei 2024 以系統思考來討論 LeSS 如何處理多頭馬車
Jen-Chieh Ko
 
RSG Taipei 2023 LeSS Design Principles
RSG Taipei 2023 LeSS Design Principles
Jen-Chieh Ko
 
Practical Testing Strategy for Agile Team
Practical Testing Strategy for Agile Team
Jen-Chieh Ko
 
O.R.I.D 初探 - 新竹敏捷分享.pdf
O.R.I.D 初探 - 新竹敏捷分享.pdf
Jen-Chieh Ko
 
2021 台灣軟體測試現狀調查
2021 台灣軟體測試現狀調查
Jen-Chieh Ko
 
Agile summit2021 - Talk About Exploratory Testing
Agile summit2021 - Talk About Exploratory Testing
Jen-Chieh Ko
 
Stop Retrospective, Start Continuous Improving
Stop Retrospective, Start Continuous Improving
Jen-Chieh Ko
 
Mopcon 2021 Scrum 是新的死亡行軍嗎?
Mopcon 2021 Scrum 是新的死亡行軍嗎?
Jen-Chieh Ko
 
啟動敏捷轉型的工具箱
啟動敏捷轉型的工具箱
Jen-Chieh Ko
 
Exploratory testing survey in 2020
Exploratory testing survey in 2020
Jen-Chieh Ko
 
如何把看板和 Scrum 發揮到極致
如何把看板和 Scrum 發揮到極致
Jen-Chieh Ko
 
Scrum和XP 的實戰經驗 (第二版)
Scrum和XP 的實戰經驗 (第二版)
Jen-Chieh Ko
 
Agile Hsinchu 七月線上聚會: 我的教練旅程
Agile Hsinchu 七月線上聚會: 我的教練旅程
Jen-Chieh Ko
 
The right It : How to make your assumption - Agile HsinChu 2020 Mar Gathering
The right It : How to make your assumption - Agile HsinChu 2020 Mar Gathering
Jen-Chieh Ko
 
Agile tourhsinchushare踩過的scrum event坑
Agile tourhsinchushare踩過的scrum event坑
Jen-Chieh Ko
 
Design sprint experience at Trend Micro
Design sprint experience at Trend Micro
Jen-Chieh Ko
 
Design sprint sharing of DS team
Design sprint sharing of DS team
Jen-Chieh Ko
 
Beer game-public
Beer game-public
Jen-Chieh Ko
 
Agile Summit Taipei 2019 - Agile Testing Strategy
Agile Summit Taipei 2019 - Agile Testing Strategy
Jen-Chieh Ko
 
Agile.Taipei 2025.1月聚會 - AI加持的人本軟體開發未來, Terry
Agile.Taipei 2025.1月聚會 - AI加持的人本軟體開發未來, Terry
Jen-Chieh Ko
 
DevOpsDays Taipei 2024 以系統思考來討論 LeSS 如何處理多頭馬車
DevOpsDays Taipei 2024 以系統思考來討論 LeSS 如何處理多頭馬車
Jen-Chieh Ko
 
RSG Taipei 2023 LeSS Design Principles
RSG Taipei 2023 LeSS Design Principles
Jen-Chieh Ko
 
Practical Testing Strategy for Agile Team
Practical Testing Strategy for Agile Team
Jen-Chieh Ko
 
O.R.I.D 初探 - 新竹敏捷分享.pdf
O.R.I.D 初探 - 新竹敏捷分享.pdf
Jen-Chieh Ko
 
2021 台灣軟體測試現狀調查
2021 台灣軟體測試現狀調查
Jen-Chieh Ko
 
Agile summit2021 - Talk About Exploratory Testing
Agile summit2021 - Talk About Exploratory Testing
Jen-Chieh Ko
 
Stop Retrospective, Start Continuous Improving
Stop Retrospective, Start Continuous Improving
Jen-Chieh Ko
 
Mopcon 2021 Scrum 是新的死亡行軍嗎?
Mopcon 2021 Scrum 是新的死亡行軍嗎?
Jen-Chieh Ko
 
啟動敏捷轉型的工具箱
啟動敏捷轉型的工具箱
Jen-Chieh Ko
 
Exploratory testing survey in 2020
Exploratory testing survey in 2020
Jen-Chieh Ko
 
如何把看板和 Scrum 發揮到極致
如何把看板和 Scrum 發揮到極致
Jen-Chieh Ko
 
Scrum和XP 的實戰經驗 (第二版)
Scrum和XP 的實戰經驗 (第二版)
Jen-Chieh Ko
 
Agile Hsinchu 七月線上聚會: 我的教練旅程
Agile Hsinchu 七月線上聚會: 我的教練旅程
Jen-Chieh Ko
 
The right It : How to make your assumption - Agile HsinChu 2020 Mar Gathering
The right It : How to make your assumption - Agile HsinChu 2020 Mar Gathering
Jen-Chieh Ko
 
Agile tourhsinchushare踩過的scrum event坑
Agile tourhsinchushare踩過的scrum event坑
Jen-Chieh Ko
 
Design sprint experience at Trend Micro
Design sprint experience at Trend Micro
Jen-Chieh Ko
 
Design sprint sharing of DS team
Design sprint sharing of DS team
Jen-Chieh Ko
 
Agile Summit Taipei 2019 - Agile Testing Strategy
Agile Summit Taipei 2019 - Agile Testing Strategy
Jen-Chieh Ko
 
Ad

Recently uploaded (20)

A Visual Introduction to the Prophet Jeremiah
A Visual Introduction to the Prophet Jeremiah
Steve Thomason
 
ENGLISH-5 Q1 Lesson 1.pptx - Story Elements
ENGLISH-5 Q1 Lesson 1.pptx - Story Elements
Mayvel Nadal
 
This is why students from these 44 institutions have not received National Se...
This is why students from these 44 institutions have not received National Se...
Kweku Zurek
 
June 2025 Progress Update With Board Call_In process.pptx
June 2025 Progress Update With Board Call_In process.pptx
International Society of Service Innovation Professionals
 
Romanticism in Love and Sacrifice An Analysis of Oscar Wilde’s The Nightingal...
Romanticism in Love and Sacrifice An Analysis of Oscar Wilde’s The Nightingal...
KaryanaTantri21
 
Q1_TLE 8_Week 1- Day 1 tools and equipment
Q1_TLE 8_Week 1- Day 1 tools and equipment
clairenotado3
 
English 3 Quarter 1_LEwithLAS_Week 1.pdf
English 3 Quarter 1_LEwithLAS_Week 1.pdf
DeAsisAlyanajaneH
 
VCE Literature Section A Exam Response Guide
VCE Literature Section A Exam Response Guide
jpinnuck
 
YSPH VMOC Special Report - Measles Outbreak Southwest US 6-14-2025.pptx
YSPH VMOC Special Report - Measles Outbreak Southwest US 6-14-2025.pptx
Yale School of Public Health - The Virtual Medical Operations Center (VMOC)
 
LDMMIA Yoga S10 Free Workshop Grad Level
LDMMIA Yoga S10 Free Workshop Grad Level
LDM & Mia eStudios
 
How to Manage Different Customer Addresses in Odoo 18 Accounting
How to Manage Different Customer Addresses in Odoo 18 Accounting
Celine George
 
Aprendendo Arquitetura Framework Salesforce - Dia 02
Aprendendo Arquitetura Framework Salesforce - Dia 02
Mauricio Alexandre Silva
 
2025 June Year 9 Presentation: Subject selection.pptx
2025 June Year 9 Presentation: Subject selection.pptx
mansk2
 
IIT KGP Quiz Week 2024 Sports Quiz (Prelims + Finals)
IIT KGP Quiz Week 2024 Sports Quiz (Prelims + Finals)
IIT Kharagpur Quiz Club
 
Public Health For The 21st Century 1st Edition Judy Orme Jane Powell
Public Health For The 21st Century 1st Edition Judy Orme Jane Powell
trjnesjnqg7801
 
M&A5 Q1 1 differentiate evolving early Philippine conventional and contempora...
M&A5 Q1 1 differentiate evolving early Philippine conventional and contempora...
ErlizaRosete
 
LAZY SUNDAY QUIZ "A GENERAL QUIZ" JUNE 2025 SMC QUIZ CLUB, SILCHAR MEDICAL CO...
LAZY SUNDAY QUIZ "A GENERAL QUIZ" JUNE 2025 SMC QUIZ CLUB, SILCHAR MEDICAL CO...
Ultimatewinner0342
 
THE PSYCHOANALYTIC OF THE BLACK CAT BY EDGAR ALLAN POE (1).pdf
THE PSYCHOANALYTIC OF THE BLACK CAT BY EDGAR ALLAN POE (1).pdf
nabilahk908
 
Great Governors' Send-Off Quiz 2025 Prelims IIT KGP
Great Governors' Send-Off Quiz 2025 Prelims IIT KGP
IIT Kharagpur Quiz Club
 
Q1_ENGLISH_PPT_WEEK 1 power point grade 3 Quarter 1 week 1
Q1_ENGLISH_PPT_WEEK 1 power point grade 3 Quarter 1 week 1
jutaydeonne
 
A Visual Introduction to the Prophet Jeremiah
A Visual Introduction to the Prophet Jeremiah
Steve Thomason
 
ENGLISH-5 Q1 Lesson 1.pptx - Story Elements
ENGLISH-5 Q1 Lesson 1.pptx - Story Elements
Mayvel Nadal
 
This is why students from these 44 institutions have not received National Se...
This is why students from these 44 institutions have not received National Se...
Kweku Zurek
 
Romanticism in Love and Sacrifice An Analysis of Oscar Wilde’s The Nightingal...
Romanticism in Love and Sacrifice An Analysis of Oscar Wilde’s The Nightingal...
KaryanaTantri21
 
Q1_TLE 8_Week 1- Day 1 tools and equipment
Q1_TLE 8_Week 1- Day 1 tools and equipment
clairenotado3
 
English 3 Quarter 1_LEwithLAS_Week 1.pdf
English 3 Quarter 1_LEwithLAS_Week 1.pdf
DeAsisAlyanajaneH
 
VCE Literature Section A Exam Response Guide
VCE Literature Section A Exam Response Guide
jpinnuck
 
LDMMIA Yoga S10 Free Workshop Grad Level
LDMMIA Yoga S10 Free Workshop Grad Level
LDM & Mia eStudios
 
How to Manage Different Customer Addresses in Odoo 18 Accounting
How to Manage Different Customer Addresses in Odoo 18 Accounting
Celine George
 
Aprendendo Arquitetura Framework Salesforce - Dia 02
Aprendendo Arquitetura Framework Salesforce - Dia 02
Mauricio Alexandre Silva
 
2025 June Year 9 Presentation: Subject selection.pptx
2025 June Year 9 Presentation: Subject selection.pptx
mansk2
 
IIT KGP Quiz Week 2024 Sports Quiz (Prelims + Finals)
IIT KGP Quiz Week 2024 Sports Quiz (Prelims + Finals)
IIT Kharagpur Quiz Club
 
Public Health For The 21st Century 1st Edition Judy Orme Jane Powell
Public Health For The 21st Century 1st Edition Judy Orme Jane Powell
trjnesjnqg7801
 
M&A5 Q1 1 differentiate evolving early Philippine conventional and contempora...
M&A5 Q1 1 differentiate evolving early Philippine conventional and contempora...
ErlizaRosete
 
LAZY SUNDAY QUIZ "A GENERAL QUIZ" JUNE 2025 SMC QUIZ CLUB, SILCHAR MEDICAL CO...
LAZY SUNDAY QUIZ "A GENERAL QUIZ" JUNE 2025 SMC QUIZ CLUB, SILCHAR MEDICAL CO...
Ultimatewinner0342
 
THE PSYCHOANALYTIC OF THE BLACK CAT BY EDGAR ALLAN POE (1).pdf
THE PSYCHOANALYTIC OF THE BLACK CAT BY EDGAR ALLAN POE (1).pdf
nabilahk908
 
Great Governors' Send-Off Quiz 2025 Prelims IIT KGP
Great Governors' Send-Off Quiz 2025 Prelims IIT KGP
IIT Kharagpur Quiz Club
 
Q1_ENGLISH_PPT_WEEK 1 power point grade 3 Quarter 1 week 1
Q1_ENGLISH_PPT_WEEK 1 power point grade 3 Quarter 1 week 1
jutaydeonne
 

Container and Test Automation Management Practices in TrendMicro

  • 1. © 2019 Trend Micro Inc.1 Container and Test Automation Management Practices in Trend Micro Larry Huang
  • 2. © 2019 Trend Micro Inc.2 Abstract • Introduction • CD Practice • TestRail and Auto Test Case Management – Test case auto sync – Auto test run creation • Container + k8s Practice – Benefits – CI/CD – Logging – Monitoring – Testing
  • 3. © 2019 Trend Micro Inc.3 Self Introduction • Larry Huang – Experience • Dev • QA – Trend Micro PLS • Lepus team
  • 4. © 2019 Trend Micro Inc.4 Lepus Team
  • 5. © 2019 Trend Micro Inc.5 Service by Our Team • Web Services – API – Frontend – Backend
  • 6. © 2019 Trend Micro Inc.6 QA Routine • Testing • Case Support • Deployment – QA (inner integration) – Beta (external integration) – Stage (external integration) – Production
  • 7. © 2019 Trend Micro Inc.7 Deployment Tool - Octopus  Process control pipeline  Permission control  Project dashboard  Variable management  Plugins  Cross Platform  GUI
  • 8. © 2019 Trend Micro Inc.8 Deployment Tool - Octopus • Octopus Process – target selection – env selection • Common Steps – API or DB connection – Deploy – DVT Test – Teardown
  • 9. © 2019 Trend Micro Inc.9 Configuration Deployment • Frequent configuration update request • Independent deploy process • Light weight – Less effort for Dev and QA
  • 10. © 2019 Trend Micro Inc.10 TestRail and Automation Test Integration
  • 11. © 2019 Trend Micro Inc.11 TestRail • Case management • Test report
  • 12. © 2019 Trend Micro Inc.12 Automation Testing Framework • Robot Framework – Python Based – Keyword – Tags for filtering • Case category • TestRail Case id
  • 13. © 2019 Trend Micro Inc.13
  • 14. © 2019 Trend Micro Inc.14 Case Synchronization On git change
  • 15. © 2019 Trend Micro Inc.15 Case Synchronization TestRail Section Path Case ID
  • 16. © 2019 Trend Micro Inc.16 Case Synchronization • Sync target – Case Title – Case ID – Customized Attributes • Sync options – direction (From robot or TestRail) – mode (Create or delete cases if not existed)
  • 17. © 2019 Trend Micro Inc.17 Case Skeleton Creation • Generate a robot file with the skeletons of the test cases – Title – ID, customized attributes – Steps as comments – Force Fail
  • 18. © 2019 Trend Micro Inc.18 Auto Test Plan (Run) • Test report before going production
  • 19. © 2019 Trend Micro Inc.19 Auto Test Run Create Test Plan Create Test Run Update Test Results
  • 20. © 2019 Trend Micro Inc.20 Container + k8s Practice
  • 21. Before Containerization Load Balancer Machine 1 Machine 2 Machine 4Machine 3 Load Balancer Machine 1 Machine 2 Machine 4Machine 3 Tech Operator
  • 22. © 2019 Trend Micro Inc.22 Before Containerization • Server Provision 1 or 2 weeks later … Request for new server Tech Operator CentOS 7 Python 3.6 Python 2.7 ?
  • 23. © 2019 Trend Micro Inc.23 Containerization • Infra as code Service A Service B Service C YAML Files
  • 24. © 2019 Trend Micro Inc.24 Deployment Requirements
  • 25. © 2019 Trend Micro Inc.25 Before Containerization • Rolling Update – Old and new version might be published in the same time – Bind and Unbind – Time consuming Tech Operator QA Please unbind machine group 1 Group 1 has been unbound 3 min later … QA Deploy and test group 1 … Please bind machine group 1 Group 1 has been bound 3 min later … …
  • 26. © 2019 Trend Micro Inc.26 Containerization • Blue Green Deployment – Time Saving • Switching pod selector – Easy to recover • Scale up/down pod replica k8s service pod selector Replica = 2 Replica = 0
  • 27. © 2019 Trend Micro Inc.27 Before Containerization • Single Point Test – Every machine needs to be checked individually
  • 28. © 2019 Trend Micro Inc.28 Containerization • No need for Single Point Test – All pods are replicas • k8s Configmap – Light weight deploy – Central Management • Only one replica Configmap A
  • 29. © 2019 Trend Micro Inc.29 Containerization • Auto Scaling • Health Assurance k8s deployment Replica = 4 Running RunningRunning Exit Container Creating
  • 30. © 2019 Trend Micro Inc.30 Containerization • Service / Machine – Microservice Friendly – Light way deployment – Cost down (10 -> 2 nodes) Service A Service B Service C Standby
  • 31. © 2019 Trend Micro Inc.31 • Each environment needs individual Cname Before Containerization service-stage.trendmicro.comservice-beta.trendmicro.com Service A Service B Service C Service A Service B Service C
  • 32. © 2019 Trend Micro Inc.32 • Ingress – No need to build individual k8s for every environment – Services are mapped by routes • Dev/QA • Beta/Stage • Production Containerization service-test.iam.trend.com /beta/login/ /stage/login/
  • 33. © 2019 Trend Micro Inc.33 k8s Secrets • sensitive data
  • 34. © 2019 Trend Micro Inc.34 CI Flow • CI – Docker Image • Product Image = Base Image + App Image • Base Image – OS – System Library • App Image – Source Code – App Library – Version control Base Product
  • 35. © 2019 Trend Micro Inc.35 Deployment Process Environment Check Create pods to test connection to external services (DB, API, …) Blue (Green) Deploy DVT Test Pod Switch
  • 36. © 2019 Trend Micro Inc.36 CD Environment • Dev/QA – k8s • Beta/Stage – AWS EKS • Production – AWS EKS • Any k8s compatible environment can be deployed
  • 37. © 2019 Trend Micro Inc.37 Lesson Learn • Node Selection – Connection super slow VPN
  • 38. © 2019 Trend Micro Inc.38 • Replica or Deployment – RestartPolicy: Always • Job – RestartPolicy: OnFailure or Never Lesson Learn
  • 39. © 2019 Trend Micro Inc.39 • (App configuration + Deployment yaml) * service * env • Solution – Config change review Lesson Learn YAML Files
  • 40. © 2019 Trend Micro Inc.40 Logging • Log cannot not be preserved in container • EKS (Beta to Production): Gray Log • K8s (Dev/QA): – Mount log folder on k8s Volumes – Cronjob sync to NAS K8s volume App cronjob
  • 41. © 2019 Trend Micro Inc.41 Monitoring • Monitis – E2E case • User scenario • Response time
  • 42. © 2019 Trend Micro Inc.42 Monitoring • Prometheus + Grafana – Resource • CPU • Memory • Pod Restart Count – Slack Channel Integration • Warning • Recovering
  • 43. © 2019 Trend Micro Inc.43 Monitoring
  • 45. © 2019 Trend Micro Inc.45 • Three environments to maintain • Shared by multiple teams – Computing Resources – Libraries – Browser Test using Traditional Way Test Client 1 Test Client 2 Test Client 3
  • 46. © 2019 Trend Micro Inc.46 • Dockerfile tells everything – Easy to replicate • No worry about resource dependency • Python migration to 3 • Selenium Grid Enabled Test in Container
  • 47. © 2019 Trend Micro Inc.47 Selenium Grid 74 73 • Parallel Test
  • 48. © 2019 Trend Micro Inc.48 Summary • YAML: code as infrastructure • Replica: HA, Auto Scaling • Ingress: Routing mapping service • Pod Selection: Blue Green Deploy • Volume: Log preservation • Prometheus + Grafana: Monitoring • Test in Container
  • 49. © 2019 Trend Micro Inc.49 Thank You

Editor's Notes

  • #5: 傳統文化Dev QA壁壘分明 Dev QA會一起cowork 想test case 或者QA有能力也可以協助開發
  • #8: QA 和 DEV 壁壘分明 DEV 不准碰 qa以上的環境 缺點?
  • #10: 某個project外部整合 頻繁且小的更動 Dev不需開branch QA可以縮小測試範圍
  • #14: 工程師懶惰或忘記只改一邊 新人看不懂
  • #19: 人工去create test plan or test run
  • #28: 獨立機器 單點測試 手動hotfix 一台一台去fix
  • #29: 測FQDN下課
  • #32: 按照之前的邏輯 Dev QA Beta Stage要額外一個k8s
  • #33: Routing 對應 k8s service http service可以掛在同一個domain下面
  • #35: Base image可以重複利用for不同的專案
  • #39: 要注意你container的restart policy Pending的原因
  • #42: 比起health check更能知道目前系統的瓶頸 設定SLI 我們可以了解user在任何時刻的使用上的感受度