SlideShare a Scribd company logo
The Cloud Specialists
CloudStack Upgrades Best Practices
Dag Sonstebo
Cloud Architect
dag.sonstebo@shapeblue.com
Twitter: @dagsonstebo
C l i c k t o e d i t
The Cloud Specialists
ShapeBlue.com @ShapeBlue
A b o u t M e
• Cloud Architect @ ShapeBlue
• Background:
• Cloud and virtualization architect with 19
years experience from the service provider,
financial and manufacturing industries.
• Specialize in:
• Cloud infrastructure architecture and
engineering.
• Virtualization - VMware vSphere, Citrix
XenServer, KVM.
• Automation and configuration management.
• Involved with CloudStack since version 2.1.
C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
“ShapeBlue are expert builders of public & private
clouds. They are the leading global CloudStack
services company.”
A b o u t S h a p e B l u e
C l i c k t o e d i t
The Cloud Specialists
ShapeBlue.com @ShapeBlue
S h a p e B l u e c u s t o m e r s
C l i c k t o e d i t
The Cloud Specialists
ShapeBlue.com @ShapeBlue
S h a p e B l u e c u s t o m e r s
C l i c k t o e d i t
The Cloud Specialists
ShapeBlue.com @ShapeBlue
S h a p e B l u e c u s t o m e r s
C l i c k t o e d i t
The Cloud Specialists
ShapeBlue.com @ShapeBlue
Upgrading CloudStack
C l i c k t o e d i t
The Cloud Specialists
ShapeBlue.com @ShapeBlue
Upgrades - general advice
• 5p’s – proper planning prevents poor performance.
• Divide and conquer – don’t plan too many changes at the same time.
• Do as much work as possible in advance:
• Prepare all builds.
• Upgrade hypervisors if required.
• Ensure system VM templates are uploaded.
• RTFM!
• Ensure infrastructure is healthy.
• Keep your users informed – and disable their access until upgraded systems healthy.
C l i c k t o e d i t
The Cloud Specialists
ShapeBlue.com @ShapeBlue
Planning - customer impact
• CloudStack GUI and API will be offline during the
upgrade.
• All VMs continue to run.
• Virtual Routers
• Will continue to function post upgrade, but can not be changed…
• To upgrade these they need to be restarted
• System VMs: are offline with the GUI, also need restarted.
C l i c k t o e d i t
The Cloud Specialists
ShapeBlue.com @ShapeBlue
Lab Upgrade Testing
• Production equivalent lab upgrade test – try to match:
• Hardware
• OS and S/W versions
• Storage – at least protocol
• Prepare:
• VMs with production type workloads
• Guest VMs with similar networking models utilised and disks
attached.
• Post upgrade tests:
• Functional, regression and user acceptance tests.
• User actions
• VM, network and storage lifecycle actions
• Don’t forget your integrated systems!
• Rinse and repeat….
C l i c k t o e d i t
The Cloud Specialists
ShapeBlue.com @ShapeBlue
Production DB upgrade test
• Warning: carry out in a network isolated location!
• Single upgraded CS server with MySQL installed.
• Import production DB, then:
• Cloudstack-setup-databases
• Cloudstack-setup-management
• Troubleshoot and fix MySQL upgrade problems,
document for production run.
C l i c k t o e d i t
The Cloud Specialists
ShapeBlue.com @ShapeBlue
Strategy – parallel builds
• Standard in-place upgrade works well, but:
• Doesn’t allow for technology refresh.
• Rollbacks are trickier and involves snapshots, backup
restores as well as loss of logs.
• Overall higher risk.
• Building a parallel upgrade infrastructure has
some benefits:
• Old CS management servers can simply be disabled.
• Upgrade done on brand new VMs / servers built in
advance.
• Rollbacks are much simpler – switch off new servers,
switch old ones back on.
• The cost of this is minimal, but it does mean a few extra
steps.
C l i c k t o e d i t
The Cloud Specialists
ShapeBlue.com @ShapeBlue
Strategy – parallel build overview
• New management server builds
• Just RPM install
• Do not seed system VM template
• Do not carry out cloudstack-setup-databases
• Do not run cloudstack-setup-management
• Don’t start services
• New MySQL servers
• Ensure all permissions used in your existing MySQL
servers is replicated on the new ones.
• Create new empty databases
• Configure master-slave and ensure the empty
databases are replicated.
C l i c k t o e d i t
The Cloud Specialists
ShapeBlue.com @ShapeBlue
Agent upgrades
• VMware:
• Minor – but watch out for password updates.
• XenServer:
• Minor - watch out for changes to vhd-utils and scripts.
• KVM:
• Only hypervisor which requires agent update.
• If large number of nodes – automate!
• Test automation well in advance.
• Remember to automate backup, upgrade and rollback!
• Use e.g. Ansible, Puppet or Chef.
C l i c k t o e d i t
The Cloud Specialists
ShapeBlue.com @ShapeBlue
Sacrificial infrastructure
• Optional - plan and prepare sacrificial infrastructure:
• For any hypervisor where agent upgrade is required , i.e. KVM. It is
easier to roll back a subset of KVM hosts rather than your whole
infrastructure.
• Create test cases which won’t impact end users:
• Accounts
• VMs
• Networks – with every service offering you may use.
• These can be used for the initial testing before decision is made to
especially do VR restarts for all end users.
C l i c k t o e d i t
The Cloud Specialists
ShapeBlue.com @ShapeBlue
U p g r a d e P r o c e s s u s i n g p a r a l l e l b u i l d sStep0
RTFM!
Build parallel
upgrade
infrastructure
Upload System
VM templates
Step1
Confirm system
VM templates
in place
Step2
Stop and
disable existing
CloudStack
management
services
Step3
Back up all
your existing
databases.
Disable MySQL
on the original
DB servers
Step4
Copy and
import the
databases to
the new
MySQL master
server.
Step5
Update
the
"host"
global
setting
Step6
Upgrade
hypervisor
s
Step7
Configure
and start
first
managem
ent server
Step8
Monitor
startup
Ensure all
HV check
in – esp.
KVM!
Step9
Restart
CPVM
then
SSVM.
Restart
VRs
Step10
Configure
and start
additional
mgmt
servers
C l i c k t o e d i t
The Cloud Specialists
ShapeBlue.com @ShapeBlue
Rollback
• Only roll back when all other options exhausted!
• All changes since upgrade will be lost – take timing
and customer access into account.
Step1
Disable new CS
mgmt servers
Disable new DB
servers
Step2On hypervisor
infrastructure
remove all VMs
created since
upgrade
Step3
Enable and start
the original
MySQL servers
Enable and start
the original
CloudStack mgmt.
servers
Step4
Restart system
VMs and VRs if
they don't auto
start
C l i c k t o e d i t
The Cloud Specialists
ShapeBlue.com @ShapeBlue
Questions?
C l i c k t o e d i t
The Cloud Specialists
ShapeBlue.com @ShapeBlue
Some references
• This talk is based on the blog post in http://www.shapeblue.com/cloudstack-
upgrades-best-practices/ - this has more technical detail.
• CloudStack upgrades – reference the release notes – e.g.
http://docs.cloudstack.apache.org/projects/cloudstack-release-
notes/en/4.9.2.0/upgrade/upgrade-4.7.html
C l i c k t o e d i t
The Cloud Specialists
ShapeBlue.com @ShapeBlue
M o r e i n f o r m a t i o n
• Slide deck: http://www.slideshare.net/shapeblue
• Blog: http://shapeblue.com/blog
http://dsonstebo.wordpress.com
• Email: dag.sonstebo@shapeblue.com
• Twitter: @dagsonstebo
• Web: http://shapeblue.com
The Cloud Specialists
CloudStack Upgrades Best
Practices
Dag Sonstebo
Cloud Architect
dag.sonstebo@shapeblue.com
Twitter: @dagsonstebo
Ad

Recommended

CCCNA17 Reliable Host Fencing
CCCNA17 Reliable Host Fencing
ShapeBlue
 
CCCNA17 Introduction
CCCNA17 Introduction
ShapeBlue
 
CCCNA17 CloudStack Container Service
CCCNA17 CloudStack Container Service
ShapeBlue
 
CCNA17 KVM and CloudStack
CCNA17 KVM and CloudStack
ShapeBlue
 
CCCNA17 Distributed CI and Testing for Cloudstack in a Hybrid Community
CCCNA17 Distributed CI and Testing for Cloudstack in a Hybrid Community
ShapeBlue
 
CCCNA17 Dynamic Roles in CloudStack
CCCNA17 Dynamic Roles in CloudStack
ShapeBlue
 
CCNA17 CloudStack and NFV
CCNA17 CloudStack and NFV
ShapeBlue
 
Dynamic roles in cloudstack
Dynamic roles in cloudstack
ShapeBlue
 
CloudStack and NFV
CloudStack and NFV
ShapeBlue
 
Paul Angus - CloudStack Backup and Recovery Framework
Paul Angus - CloudStack Backup and Recovery Framework
ShapeBlue
 
CloudStack UI
CloudStack UI
ShapeBlue
 
New stuff in CloudStack!
New stuff in CloudStack!
ShapeBlue
 
Giles Sirett: Introduction and CloudStack news
Giles Sirett: Introduction and CloudStack news
ShapeBlue
 
Paul Angus – Backup & Recovery in CloudStack
Paul Angus – Backup & Recovery in CloudStack
ShapeBlue
 
CloudStack EU user group - Trillian
CloudStack EU user group - Trillian
ShapeBlue
 
Boris Stoyanov - some new features in Apache cloudStack
Boris Stoyanov - some new features in Apache cloudStack
ShapeBlue
 
What’s New in CloudStack 4.15 - CloudStack European User Group Virtual, May 2021
What’s New in CloudStack 4.15 - CloudStack European User Group Virtual, May 2021
ShapeBlue
 
CloudStack usage service
CloudStack usage service
ShapeBlue
 
ApacheCon Miami / CCCNA17 CloudStack upgrade best practices
ApacheCon Miami / CCCNA17 CloudStack upgrade best practices
Dag Sonstebo
 
TechUG Glasgow talk 22/Feb/17 Configuration Management Best Practices
TechUG Glasgow talk 22/Feb/17 Configuration Management Best Practices
Dag Sonstebo
 
ApacheCon Miami / CCCNA17 Using KVM in CloudStack
ApacheCon Miami / CCCNA17 Using KVM in CloudStack
Dag Sonstebo
 
Designing Lean CloudStack Environments for the Edge - IndiQus - CloudStack E...
Designing Lean CloudStack Environments for the Edge - IndiQus - CloudStack E...
ShapeBlue
 
Using the KVMhypervisor in CloudStack
Using the KVMhypervisor in CloudStack
ShapeBlue
 
KVM High Availability Regardless of Storage - Gabriel Brascher, VP of Apache ...
KVM High Availability Regardless of Storage - Gabriel Brascher, VP of Apache ...
ShapeBlue
 
CSEUG introduction
CSEUG introduction
ShapeBlue
 
vBACD - Deploying Infrastructure-as-a-Service with CloudStack - 2/28
vBACD - Deploying Infrastructure-as-a-Service with CloudStack - 2/28
CloudStack - Open Source Cloud Computing Project
 
Introduction and CloudStack news
Introduction and CloudStack news
ShapeBlue
 
Network Functions Virtualization and CloudStack
Network Functions Virtualization and CloudStack
Chiradeep Vittal
 
CloudStack upgrade best practices - Dag Sonstebo
CloudStack upgrade best practices - Dag Sonstebo
Ingo Jochim
 
Improving CloudStack for operators
Improving CloudStack for operators
ShapeBlue
 

More Related Content

What's hot (20)

CloudStack and NFV
CloudStack and NFV
ShapeBlue
 
Paul Angus - CloudStack Backup and Recovery Framework
Paul Angus - CloudStack Backup and Recovery Framework
ShapeBlue
 
CloudStack UI
CloudStack UI
ShapeBlue
 
New stuff in CloudStack!
New stuff in CloudStack!
ShapeBlue
 
Giles Sirett: Introduction and CloudStack news
Giles Sirett: Introduction and CloudStack news
ShapeBlue
 
Paul Angus – Backup & Recovery in CloudStack
Paul Angus – Backup & Recovery in CloudStack
ShapeBlue
 
CloudStack EU user group - Trillian
CloudStack EU user group - Trillian
ShapeBlue
 
Boris Stoyanov - some new features in Apache cloudStack
Boris Stoyanov - some new features in Apache cloudStack
ShapeBlue
 
What’s New in CloudStack 4.15 - CloudStack European User Group Virtual, May 2021
What’s New in CloudStack 4.15 - CloudStack European User Group Virtual, May 2021
ShapeBlue
 
CloudStack usage service
CloudStack usage service
ShapeBlue
 
ApacheCon Miami / CCCNA17 CloudStack upgrade best practices
ApacheCon Miami / CCCNA17 CloudStack upgrade best practices
Dag Sonstebo
 
TechUG Glasgow talk 22/Feb/17 Configuration Management Best Practices
TechUG Glasgow talk 22/Feb/17 Configuration Management Best Practices
Dag Sonstebo
 
ApacheCon Miami / CCCNA17 Using KVM in CloudStack
ApacheCon Miami / CCCNA17 Using KVM in CloudStack
Dag Sonstebo
 
Designing Lean CloudStack Environments for the Edge - IndiQus - CloudStack E...
Designing Lean CloudStack Environments for the Edge - IndiQus - CloudStack E...
ShapeBlue
 
Using the KVMhypervisor in CloudStack
Using the KVMhypervisor in CloudStack
ShapeBlue
 
KVM High Availability Regardless of Storage - Gabriel Brascher, VP of Apache ...
KVM High Availability Regardless of Storage - Gabriel Brascher, VP of Apache ...
ShapeBlue
 
CSEUG introduction
CSEUG introduction
ShapeBlue
 
vBACD - Deploying Infrastructure-as-a-Service with CloudStack - 2/28
vBACD - Deploying Infrastructure-as-a-Service with CloudStack - 2/28
CloudStack - Open Source Cloud Computing Project
 
Introduction and CloudStack news
Introduction and CloudStack news
ShapeBlue
 
Network Functions Virtualization and CloudStack
Network Functions Virtualization and CloudStack
Chiradeep Vittal
 
CloudStack and NFV
CloudStack and NFV
ShapeBlue
 
Paul Angus - CloudStack Backup and Recovery Framework
Paul Angus - CloudStack Backup and Recovery Framework
ShapeBlue
 
CloudStack UI
CloudStack UI
ShapeBlue
 
New stuff in CloudStack!
New stuff in CloudStack!
ShapeBlue
 
Giles Sirett: Introduction and CloudStack news
Giles Sirett: Introduction and CloudStack news
ShapeBlue
 
Paul Angus – Backup & Recovery in CloudStack
Paul Angus – Backup & Recovery in CloudStack
ShapeBlue
 
CloudStack EU user group - Trillian
CloudStack EU user group - Trillian
ShapeBlue
 
Boris Stoyanov - some new features in Apache cloudStack
Boris Stoyanov - some new features in Apache cloudStack
ShapeBlue
 
What’s New in CloudStack 4.15 - CloudStack European User Group Virtual, May 2021
What’s New in CloudStack 4.15 - CloudStack European User Group Virtual, May 2021
ShapeBlue
 
CloudStack usage service
CloudStack usage service
ShapeBlue
 
ApacheCon Miami / CCCNA17 CloudStack upgrade best practices
ApacheCon Miami / CCCNA17 CloudStack upgrade best practices
Dag Sonstebo
 
TechUG Glasgow talk 22/Feb/17 Configuration Management Best Practices
TechUG Glasgow talk 22/Feb/17 Configuration Management Best Practices
Dag Sonstebo
 
ApacheCon Miami / CCCNA17 Using KVM in CloudStack
ApacheCon Miami / CCCNA17 Using KVM in CloudStack
Dag Sonstebo
 
Designing Lean CloudStack Environments for the Edge - IndiQus - CloudStack E...
Designing Lean CloudStack Environments for the Edge - IndiQus - CloudStack E...
ShapeBlue
 
Using the KVMhypervisor in CloudStack
Using the KVMhypervisor in CloudStack
ShapeBlue
 
KVM High Availability Regardless of Storage - Gabriel Brascher, VP of Apache ...
KVM High Availability Regardless of Storage - Gabriel Brascher, VP of Apache ...
ShapeBlue
 
CSEUG introduction
CSEUG introduction
ShapeBlue
 
Introduction and CloudStack news
Introduction and CloudStack news
ShapeBlue
 
Network Functions Virtualization and CloudStack
Network Functions Virtualization and CloudStack
Chiradeep Vittal
 

Similar to CCCNA17 CloudStack upgrade best practices (20)

CloudStack upgrade best practices - Dag Sonstebo
CloudStack upgrade best practices - Dag Sonstebo
Ingo Jochim
 
Improving CloudStack for operators
Improving CloudStack for operators
ShapeBlue
 
Designing CloudStack Clouds
Designing CloudStack Clouds
ShapeBlue
 
Silicon Valley CloudStack User Group - Designing CloudStack Clouds
Silicon Valley CloudStack User Group - Designing CloudStack Clouds
ShapeBlue
 
CloudStack Container Service
CloudStack Container Service
ShapeBlue
 
Paul Angus - CloudStack Container Service
Paul Angus - CloudStack Container Service
ShapeBlue
 
Automating CloudStack and hypervisor installation and configuration
Automating CloudStack and hypervisor installation and configuration
Dag Sonstebo
 
Cloudstack collaboration - customer focus
Cloudstack collaboration - customer focus
ShapeBlue
 
Paul Angus - what's new in ACS 4.11
Paul Angus - what's new in ACS 4.11
ShapeBlue
 
Whats new in Cloudstack 4.11 - behind the headlines
Whats new in Cloudstack 4.11 - behind the headlines
ShapeBlue
 
Secret Techniques to Manage Apache Cloudstack with ActOnCloud
Secret Techniques to Manage Apache Cloudstack with ActOnCloud
Madan Ganesh Velayudham
 
Designing cloud stack clouds geoff higginbottom/shapeblue
Designing cloud stack clouds geoff higginbottom/shapeblue
ShapeBlue
 
Operating OpenStack on a Budget
Operating OpenStack on a Budget
Susan Wu
 
Operating OpenStack on a Budget
Operating OpenStack on a Budget
Samir Ibradzic
 
CloudStack - Top 5 Technical Issues and Troubleshooting
CloudStack - Top 5 Technical Issues and Troubleshooting
ShapeBlue
 
Whats the Use!? (Real Customer Use-Cases)
Whats the Use!? (Real Customer Use-Cases)
ShapeBlue
 
An introduction to Citrix CloudPlatform (powered by Apache CloudStack), Citri...
An introduction to Citrix CloudPlatform (powered by Apache CloudStack), Citri...
ShapeBlue
 
Sameer Mitter - Management Responsibilities by Cloud service model types
Sameer Mitter - Management Responsibilities by Cloud service model types
Sameer Mitter
 
CloudStack EU user group - CloudStack news
CloudStack EU user group - CloudStack news
ShapeBlue
 
OpenStack + VMware: Deploy, Upgrade, & Operate a Powerful Production OpenStac...
OpenStack + VMware: Deploy, Upgrade, & Operate a Powerful Production OpenStac...
Mark Voelker
 
CloudStack upgrade best practices - Dag Sonstebo
CloudStack upgrade best practices - Dag Sonstebo
Ingo Jochim
 
Improving CloudStack for operators
Improving CloudStack for operators
ShapeBlue
 
Designing CloudStack Clouds
Designing CloudStack Clouds
ShapeBlue
 
Silicon Valley CloudStack User Group - Designing CloudStack Clouds
Silicon Valley CloudStack User Group - Designing CloudStack Clouds
ShapeBlue
 
CloudStack Container Service
CloudStack Container Service
ShapeBlue
 
Paul Angus - CloudStack Container Service
Paul Angus - CloudStack Container Service
ShapeBlue
 
Automating CloudStack and hypervisor installation and configuration
Automating CloudStack and hypervisor installation and configuration
Dag Sonstebo
 
Cloudstack collaboration - customer focus
Cloudstack collaboration - customer focus
ShapeBlue
 
Paul Angus - what's new in ACS 4.11
Paul Angus - what's new in ACS 4.11
ShapeBlue
 
Whats new in Cloudstack 4.11 - behind the headlines
Whats new in Cloudstack 4.11 - behind the headlines
ShapeBlue
 
Secret Techniques to Manage Apache Cloudstack with ActOnCloud
Secret Techniques to Manage Apache Cloudstack with ActOnCloud
Madan Ganesh Velayudham
 
Designing cloud stack clouds geoff higginbottom/shapeblue
Designing cloud stack clouds geoff higginbottom/shapeblue
ShapeBlue
 
Operating OpenStack on a Budget
Operating OpenStack on a Budget
Susan Wu
 
Operating OpenStack on a Budget
Operating OpenStack on a Budget
Samir Ibradzic
 
CloudStack - Top 5 Technical Issues and Troubleshooting
CloudStack - Top 5 Technical Issues and Troubleshooting
ShapeBlue
 
Whats the Use!? (Real Customer Use-Cases)
Whats the Use!? (Real Customer Use-Cases)
ShapeBlue
 
An introduction to Citrix CloudPlatform (powered by Apache CloudStack), Citri...
An introduction to Citrix CloudPlatform (powered by Apache CloudStack), Citri...
ShapeBlue
 
Sameer Mitter - Management Responsibilities by Cloud service model types
Sameer Mitter - Management Responsibilities by Cloud service model types
Sameer Mitter
 
CloudStack EU user group - CloudStack news
CloudStack EU user group - CloudStack news
ShapeBlue
 
OpenStack + VMware: Deploy, Upgrade, & Operate a Powerful Production OpenStac...
OpenStack + VMware: Deploy, Upgrade, & Operate a Powerful Production OpenStac...
Mark Voelker
 
Ad

More from ShapeBlue (20)

Fully Open-Source Private Clouds: Freedom, Security, and Control
Fully Open-Source Private Clouds: Freedom, Security, and Control
ShapeBlue
 
Pushing the Limits: CloudStack at 25K Hosts
Pushing the Limits: CloudStack at 25K Hosts
ShapeBlue
 
Stretching CloudStack over multiple datacenters
Stretching CloudStack over multiple datacenters
ShapeBlue
 
Proposed Feature: Monitoring and Managing Cloud Usage Costs in Apache CloudStack
Proposed Feature: Monitoring and Managing Cloud Usage Costs in Apache CloudStack
ShapeBlue
 
CloudStack + KVM: Your Local Cloud Lab
CloudStack + KVM: Your Local Cloud Lab
ShapeBlue
 
I’d like to resell your CloudStack services, but...
I’d like to resell your CloudStack services, but...
ShapeBlue
 
Storage Setup for LINSTOR/DRBD/CloudStack
Storage Setup for LINSTOR/DRBD/CloudStack
ShapeBlue
 
Apache CloudStack 101 - Introduction, What’s New and What’s Coming
Apache CloudStack 101 - Introduction, What’s New and What’s Coming
ShapeBlue
 
Development of an Оbject Storage Plugin for CloudStack, Christian Reichert, s...
Development of an Оbject Storage Plugin for CloudStack, Christian Reichert, s...
ShapeBlue
 
VM-HA with CloudStack and Linstor, Rene Peinthor
VM-HA with CloudStack and Linstor, Rene Peinthor
ShapeBlue
 
How We Use CloudStack to Provide Managed Hosting, Swen Brüseke, proIO
How We Use CloudStack to Provide Managed Hosting, Swen Brüseke, proIO
ShapeBlue
 
Internet Facing VMs and the DDoS Problem, Wido den Hollander, Your.Online
Internet Facing VMs and the DDoS Problem, Wido den Hollander, Your.Online
ShapeBlue
 
Transitioning from VMware to Apache CloudStack: A Path to Profitability and C...
Transitioning from VMware to Apache CloudStack: A Path to Profitability and C...
ShapeBlue
 
What’s New and What’s Upcoming in Apache CloudStack, Giles Sirett, ShapeBlue
What’s New and What’s Upcoming in Apache CloudStack, Giles Sirett, ShapeBlue
ShapeBlue
 
CloudStack Authentication Methods – Harikrishna Patnala, ShapeBlue
CloudStack Authentication Methods – Harikrishna Patnala, ShapeBlue
ShapeBlue
 
CloudStack Tooling Ecosystem – Kiran Chavala, ShapeBlue
CloudStack Tooling Ecosystem – Kiran Chavala, ShapeBlue
ShapeBlue
 
Elevating Cloud Infrastructure with Object Storage, DRS, VM Scheduling, and D...
Elevating Cloud Infrastructure with Object Storage, DRS, VM Scheduling, and D...
ShapeBlue
 
VM Migration from VMware to CloudStack and KVM – Suresh Anaparti, ShapeBlue
VM Migration from VMware to CloudStack and KVM – Suresh Anaparti, ShapeBlue
ShapeBlue
 
How We Grew Up with CloudStack and its Journey – Dilip Singh, DataHub
How We Grew Up with CloudStack and its Journey – Dilip Singh, DataHub
ShapeBlue
 
What’s New in CloudStack 4.19, Abhishek Kumar, Release Manager Apache CloudSt...
What’s New in CloudStack 4.19, Abhishek Kumar, Release Manager Apache CloudSt...
ShapeBlue
 
Fully Open-Source Private Clouds: Freedom, Security, and Control
Fully Open-Source Private Clouds: Freedom, Security, and Control
ShapeBlue
 
Pushing the Limits: CloudStack at 25K Hosts
Pushing the Limits: CloudStack at 25K Hosts
ShapeBlue
 
Stretching CloudStack over multiple datacenters
Stretching CloudStack over multiple datacenters
ShapeBlue
 
Proposed Feature: Monitoring and Managing Cloud Usage Costs in Apache CloudStack
Proposed Feature: Monitoring and Managing Cloud Usage Costs in Apache CloudStack
ShapeBlue
 
CloudStack + KVM: Your Local Cloud Lab
CloudStack + KVM: Your Local Cloud Lab
ShapeBlue
 
I’d like to resell your CloudStack services, but...
I’d like to resell your CloudStack services, but...
ShapeBlue
 
Storage Setup for LINSTOR/DRBD/CloudStack
Storage Setup for LINSTOR/DRBD/CloudStack
ShapeBlue
 
Apache CloudStack 101 - Introduction, What’s New and What’s Coming
Apache CloudStack 101 - Introduction, What’s New and What’s Coming
ShapeBlue
 
Development of an Оbject Storage Plugin for CloudStack, Christian Reichert, s...
Development of an Оbject Storage Plugin for CloudStack, Christian Reichert, s...
ShapeBlue
 
VM-HA with CloudStack and Linstor, Rene Peinthor
VM-HA with CloudStack and Linstor, Rene Peinthor
ShapeBlue
 
How We Use CloudStack to Provide Managed Hosting, Swen Brüseke, proIO
How We Use CloudStack to Provide Managed Hosting, Swen Brüseke, proIO
ShapeBlue
 
Internet Facing VMs and the DDoS Problem, Wido den Hollander, Your.Online
Internet Facing VMs and the DDoS Problem, Wido den Hollander, Your.Online
ShapeBlue
 
Transitioning from VMware to Apache CloudStack: A Path to Profitability and C...
Transitioning from VMware to Apache CloudStack: A Path to Profitability and C...
ShapeBlue
 
What’s New and What’s Upcoming in Apache CloudStack, Giles Sirett, ShapeBlue
What’s New and What’s Upcoming in Apache CloudStack, Giles Sirett, ShapeBlue
ShapeBlue
 
CloudStack Authentication Methods – Harikrishna Patnala, ShapeBlue
CloudStack Authentication Methods – Harikrishna Patnala, ShapeBlue
ShapeBlue
 
CloudStack Tooling Ecosystem – Kiran Chavala, ShapeBlue
CloudStack Tooling Ecosystem – Kiran Chavala, ShapeBlue
ShapeBlue
 
Elevating Cloud Infrastructure with Object Storage, DRS, VM Scheduling, and D...
Elevating Cloud Infrastructure with Object Storage, DRS, VM Scheduling, and D...
ShapeBlue
 
VM Migration from VMware to CloudStack and KVM – Suresh Anaparti, ShapeBlue
VM Migration from VMware to CloudStack and KVM – Suresh Anaparti, ShapeBlue
ShapeBlue
 
How We Grew Up with CloudStack and its Journey – Dilip Singh, DataHub
How We Grew Up with CloudStack and its Journey – Dilip Singh, DataHub
ShapeBlue
 
What’s New in CloudStack 4.19, Abhishek Kumar, Release Manager Apache CloudSt...
What’s New in CloudStack 4.19, Abhishek Kumar, Release Manager Apache CloudSt...
ShapeBlue
 
Ad

Recently uploaded (20)

Curietech AI in action - Accelerate MuleSoft development
Curietech AI in action - Accelerate MuleSoft development
shyamraj55
 
Coordinated Disclosure for ML - What's Different and What's the Same.pdf
Coordinated Disclosure for ML - What's Different and What's the Same.pdf
Priyanka Aash
 
“MPU+: A Transformative Solution for Next-Gen AI at the Edge,” a Presentation...
“MPU+: A Transformative Solution for Next-Gen AI at the Edge,” a Presentation...
Edge AI and Vision Alliance
 
CapCut Pro Crack For PC Latest Version {Fully Unlocked} 2025
CapCut Pro Crack For PC Latest Version {Fully Unlocked} 2025
pcprocore
 
AI vs Human Writing: Can You Tell the Difference?
AI vs Human Writing: Can You Tell the Difference?
Shashi Sathyanarayana, Ph.D
 
9-1-1 Addressing: End-to-End Automation Using FME
9-1-1 Addressing: End-to-End Automation Using FME
Safe Software
 
From Manual to Auto Searching- FME in the Driver's Seat
From Manual to Auto Searching- FME in the Driver's Seat
Safe Software
 
Oh, the Possibilities - Balancing Innovation and Risk with Generative AI.pdf
Oh, the Possibilities - Balancing Innovation and Risk with Generative AI.pdf
Priyanka Aash
 
Salesforce Summer '25 Release Frenchgathering.pptx.pdf
Salesforce Summer '25 Release Frenchgathering.pptx.pdf
yosra Saidani
 
"How to survive Black Friday: preparing e-commerce for a peak season", Yurii ...
"How to survive Black Friday: preparing e-commerce for a peak season", Yurii ...
Fwdays
 
Tech-ASan: Two-stage check for Address Sanitizer - Yixuan Cao.pdf
Tech-ASan: Two-stage check for Address Sanitizer - Yixuan Cao.pdf
caoyixuan2019
 
OpenPOWER Foundation & Open-Source Core Innovations
OpenPOWER Foundation & Open-Source Core Innovations
IBM
 
Enhance GitHub Copilot using MCP - Enterprise version.pdf
Enhance GitHub Copilot using MCP - Enterprise version.pdf
Nilesh Gule
 
"Database isolation: how we deal with hundreds of direct connections to the d...
"Database isolation: how we deal with hundreds of direct connections to the d...
Fwdays
 
Cyber Defense Matrix Workshop - RSA Conference
Cyber Defense Matrix Workshop - RSA Conference
Priyanka Aash
 
Python Conference Singapore - 19 Jun 2025
Python Conference Singapore - 19 Jun 2025
ninefyi
 
Connecting Data and Intelligence: The Role of FME in Machine Learning
Connecting Data and Intelligence: The Role of FME in Machine Learning
Safe Software
 
OWASP Barcelona 2025 Threat Model Library
OWASP Barcelona 2025 Threat Model Library
PetraVukmirovic
 
Securing Account Lifecycles in the Age of Deepfakes.pptx
Securing Account Lifecycles in the Age of Deepfakes.pptx
FIDO Alliance
 
" How to survive with 1 billion vectors and not sell a kidney: our low-cost c...
" How to survive with 1 billion vectors and not sell a kidney: our low-cost c...
Fwdays
 
Curietech AI in action - Accelerate MuleSoft development
Curietech AI in action - Accelerate MuleSoft development
shyamraj55
 
Coordinated Disclosure for ML - What's Different and What's the Same.pdf
Coordinated Disclosure for ML - What's Different and What's the Same.pdf
Priyanka Aash
 
“MPU+: A Transformative Solution for Next-Gen AI at the Edge,” a Presentation...
“MPU+: A Transformative Solution for Next-Gen AI at the Edge,” a Presentation...
Edge AI and Vision Alliance
 
CapCut Pro Crack For PC Latest Version {Fully Unlocked} 2025
CapCut Pro Crack For PC Latest Version {Fully Unlocked} 2025
pcprocore
 
AI vs Human Writing: Can You Tell the Difference?
AI vs Human Writing: Can You Tell the Difference?
Shashi Sathyanarayana, Ph.D
 
9-1-1 Addressing: End-to-End Automation Using FME
9-1-1 Addressing: End-to-End Automation Using FME
Safe Software
 
From Manual to Auto Searching- FME in the Driver's Seat
From Manual to Auto Searching- FME in the Driver's Seat
Safe Software
 
Oh, the Possibilities - Balancing Innovation and Risk with Generative AI.pdf
Oh, the Possibilities - Balancing Innovation and Risk with Generative AI.pdf
Priyanka Aash
 
Salesforce Summer '25 Release Frenchgathering.pptx.pdf
Salesforce Summer '25 Release Frenchgathering.pptx.pdf
yosra Saidani
 
"How to survive Black Friday: preparing e-commerce for a peak season", Yurii ...
"How to survive Black Friday: preparing e-commerce for a peak season", Yurii ...
Fwdays
 
Tech-ASan: Two-stage check for Address Sanitizer - Yixuan Cao.pdf
Tech-ASan: Two-stage check for Address Sanitizer - Yixuan Cao.pdf
caoyixuan2019
 
OpenPOWER Foundation & Open-Source Core Innovations
OpenPOWER Foundation & Open-Source Core Innovations
IBM
 
Enhance GitHub Copilot using MCP - Enterprise version.pdf
Enhance GitHub Copilot using MCP - Enterprise version.pdf
Nilesh Gule
 
"Database isolation: how we deal with hundreds of direct connections to the d...
"Database isolation: how we deal with hundreds of direct connections to the d...
Fwdays
 
Cyber Defense Matrix Workshop - RSA Conference
Cyber Defense Matrix Workshop - RSA Conference
Priyanka Aash
 
Python Conference Singapore - 19 Jun 2025
Python Conference Singapore - 19 Jun 2025
ninefyi
 
Connecting Data and Intelligence: The Role of FME in Machine Learning
Connecting Data and Intelligence: The Role of FME in Machine Learning
Safe Software
 
OWASP Barcelona 2025 Threat Model Library
OWASP Barcelona 2025 Threat Model Library
PetraVukmirovic
 
Securing Account Lifecycles in the Age of Deepfakes.pptx
Securing Account Lifecycles in the Age of Deepfakes.pptx
FIDO Alliance
 
" How to survive with 1 billion vectors and not sell a kidney: our low-cost c...
" How to survive with 1 billion vectors and not sell a kidney: our low-cost c...
Fwdays
 

CCCNA17 CloudStack upgrade best practices

  • 1. The Cloud Specialists CloudStack Upgrades Best Practices Dag Sonstebo Cloud Architect [email protected] Twitter: @dagsonstebo
  • 2. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue A b o u t M e • Cloud Architect @ ShapeBlue • Background: • Cloud and virtualization architect with 19 years experience from the service provider, financial and manufacturing industries. • Specialize in: • Cloud infrastructure architecture and engineering. • Virtualization - VMware vSphere, Citrix XenServer, KVM. • Automation and configuration management. • Involved with CloudStack since version 2.1.
  • 3. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue “ShapeBlue are expert builders of public & private clouds. They are the leading global CloudStack services company.” A b o u t S h a p e B l u e
  • 4. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue S h a p e B l u e c u s t o m e r s
  • 5. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue S h a p e B l u e c u s t o m e r s
  • 6. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue S h a p e B l u e c u s t o m e r s
  • 7. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue Upgrading CloudStack
  • 8. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue Upgrades - general advice • 5p’s – proper planning prevents poor performance. • Divide and conquer – don’t plan too many changes at the same time. • Do as much work as possible in advance: • Prepare all builds. • Upgrade hypervisors if required. • Ensure system VM templates are uploaded. • RTFM! • Ensure infrastructure is healthy. • Keep your users informed – and disable their access until upgraded systems healthy.
  • 9. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue Planning - customer impact • CloudStack GUI and API will be offline during the upgrade. • All VMs continue to run. • Virtual Routers • Will continue to function post upgrade, but can not be changed… • To upgrade these they need to be restarted • System VMs: are offline with the GUI, also need restarted.
  • 10. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue Lab Upgrade Testing • Production equivalent lab upgrade test – try to match: • Hardware • OS and S/W versions • Storage – at least protocol • Prepare: • VMs with production type workloads • Guest VMs with similar networking models utilised and disks attached. • Post upgrade tests: • Functional, regression and user acceptance tests. • User actions • VM, network and storage lifecycle actions • Don’t forget your integrated systems! • Rinse and repeat….
  • 11. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue Production DB upgrade test • Warning: carry out in a network isolated location! • Single upgraded CS server with MySQL installed. • Import production DB, then: • Cloudstack-setup-databases • Cloudstack-setup-management • Troubleshoot and fix MySQL upgrade problems, document for production run.
  • 12. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue Strategy – parallel builds • Standard in-place upgrade works well, but: • Doesn’t allow for technology refresh. • Rollbacks are trickier and involves snapshots, backup restores as well as loss of logs. • Overall higher risk. • Building a parallel upgrade infrastructure has some benefits: • Old CS management servers can simply be disabled. • Upgrade done on brand new VMs / servers built in advance. • Rollbacks are much simpler – switch off new servers, switch old ones back on. • The cost of this is minimal, but it does mean a few extra steps.
  • 13. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue Strategy – parallel build overview • New management server builds • Just RPM install • Do not seed system VM template • Do not carry out cloudstack-setup-databases • Do not run cloudstack-setup-management • Don’t start services • New MySQL servers • Ensure all permissions used in your existing MySQL servers is replicated on the new ones. • Create new empty databases • Configure master-slave and ensure the empty databases are replicated.
  • 14. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue Agent upgrades • VMware: • Minor – but watch out for password updates. • XenServer: • Minor - watch out for changes to vhd-utils and scripts. • KVM: • Only hypervisor which requires agent update. • If large number of nodes – automate! • Test automation well in advance. • Remember to automate backup, upgrade and rollback! • Use e.g. Ansible, Puppet or Chef.
  • 15. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue Sacrificial infrastructure • Optional - plan and prepare sacrificial infrastructure: • For any hypervisor where agent upgrade is required , i.e. KVM. It is easier to roll back a subset of KVM hosts rather than your whole infrastructure. • Create test cases which won’t impact end users: • Accounts • VMs • Networks – with every service offering you may use. • These can be used for the initial testing before decision is made to especially do VR restarts for all end users.
  • 16. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue U p g r a d e P r o c e s s u s i n g p a r a l l e l b u i l d sStep0 RTFM! Build parallel upgrade infrastructure Upload System VM templates Step1 Confirm system VM templates in place Step2 Stop and disable existing CloudStack management services Step3 Back up all your existing databases. Disable MySQL on the original DB servers Step4 Copy and import the databases to the new MySQL master server. Step5 Update the "host" global setting Step6 Upgrade hypervisor s Step7 Configure and start first managem ent server Step8 Monitor startup Ensure all HV check in – esp. KVM! Step9 Restart CPVM then SSVM. Restart VRs Step10 Configure and start additional mgmt servers
  • 17. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue Rollback • Only roll back when all other options exhausted! • All changes since upgrade will be lost – take timing and customer access into account. Step1 Disable new CS mgmt servers Disable new DB servers Step2On hypervisor infrastructure remove all VMs created since upgrade Step3 Enable and start the original MySQL servers Enable and start the original CloudStack mgmt. servers Step4 Restart system VMs and VRs if they don't auto start
  • 18. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue Questions?
  • 19. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue Some references • This talk is based on the blog post in http://www.shapeblue.com/cloudstack- upgrades-best-practices/ - this has more technical detail. • CloudStack upgrades – reference the release notes – e.g. http://docs.cloudstack.apache.org/projects/cloudstack-release- notes/en/4.9.2.0/upgrade/upgrade-4.7.html
  • 20. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue M o r e i n f o r m a t i o n • Slide deck: http://www.slideshare.net/shapeblue • Blog: http://shapeblue.com/blog http://dsonstebo.wordpress.com • Email: [email protected] • Twitter: @dagsonstebo • Web: http://shapeblue.com
  • 21. The Cloud Specialists CloudStack Upgrades Best Practices Dag Sonstebo Cloud Architect [email protected] Twitter: @dagsonstebo