ApacheCon Miami / CCCNA17 Using KVM in CloudStack
0 likes758 views
The document discusses using KVM as the hypervisor with CloudStack. It provides background on KVM, how it works with CloudStack, and guidelines for installing and configuring it on the host including: - KVM is included in the Linux kernel and provides hardware virtualization. It works with CloudStack since version 2.0. - When using KVM with CloudStack, Linux bridges or Open vSwitch can be used for network bridging between VMs. Configuration examples are provided for both. - Storage configuration using a dedicated VLAN is also demonstrated. - The document provides tips for ensuring the KVM and libvirt services are configured correctly on the host for CloudStack management.
More Related Content
What's hot (20)
Similar to ApacheCon Miami / CCCNA17 Using KVM in CloudStack (20)
Recently uploaded (20)
ApacheCon Miami / CCCNA17 Using KVM in CloudStack
- 2. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue A b o u t M e • Cloud Architect @ ShapeBlue • Background: • Cloud and virtualization architect with 19 years experience from the service provider, financial and manufacturing industries. • Specialize in: • Cloud infrastructure architecture and engineering. • Virtualization - VMware vSphere, Citrix XenServer, KVM. • Automation • Involved with CloudStack since version 2.1.
- 3. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue “ShapeBlue are expert builders of public & private clouds. They are the leading global CloudStack services company.” A b o u t S h a p e B l u e
- 4. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue S h a p e B l u e c u s t o m e r s
- 5. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue S h a p e B l u e c u s t o m e r s
- 6. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue S h a p e B l u e c u s t o m e r s
- 7. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue CloudStack and hypervisor choice
- 8. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue C l o u d S t a c k h y p e r v i s o r c h o i c e Hypervisor Pros Cons VMware • Feature rich • Mature • Native HA / DRS • Bare metal • Cost – capex. • IP consumption • Virtual Centre XenServer • Mature • HA • Disk chains • Poolmaster • No workload balancing KVM • Fast • Maturing. • Cost • Non-proprietary • Lacking some networking, storage and snapshotting features. • No workload balancing
- 9. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue KVM background
- 10. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue K V M b a c k g r o u n d • KVM = Kernel-based Virtual Machine. • In CloudStack KVM is used as a type 2 hypervisor – provides virtualization support on top of a running Linux OS instance. • Included in the Linux kernel since version 2.6.20 (2007). • KVM is Free Software released under the GPL. • Supported in CloudStack since early cloud.com days (version 2.0).
- 11. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue K V M b a c k g r o u n d • Hardware: • 32 / 64 bit with CPU virtualization support (Intel-VTX / AMD-V). • Kernel modules • kvm • kvm_intel • kvm_amd • Para-virtualization through the Virtio API. • KVM and Qemu: • KVM abstracts access to the CPU and memory. • QEMU emulates the hardware resources (disks, graphics, USB, etc).
- 12. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue K V M a n d C l o u d S t a c k • Pros: • Flexible - not proprietary. • Fast. • Lightweight. • No SPOF – no poolmasters or VirtualCentre like components. • Cons: • No workload balancing (yet…). • No network throttling. • No VM snapshot support. • No native clustered file system. • Root volume migration requires VM to be shut down.
- 13. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue Installation and configuration
- 14. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue B a s e i n s t a l l a t i o n • Host OS: • CentOS / RHEL 6.x / 7.x • Ubuntu 12.04 / 14.04 / 16.04 • 4GB memory / 64-bit X86. • Components (CloudStack 4.9): • NTP + DNS • Libvirt 1.2.0 or higher • Qemu / KVM: 2.0 or higher • Installing the CloudStack agent installs KVM and libvirt: • yum install cloudstack-agent • apt-get install cloudstack-agent
- 15. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue C o n f i g u r a t i o n • /etc/libvirt/libvirtd.conf: listen_tls = 0 listen_tcp = 1 tcp_port = "16509” auth_tcp = "none” mdns_adv = 0 • /etc/sysconfig/libvirtd: LIBVIRTD_ARGS="--listen” libvirtd_opts="-d -l” • Security policies: • CentOS: Selinux: permissive • Ubuntu: Apparmor • Firewall ports: • TCP/22 (SSH) • TCP/1798 (KVM) • TCP/16509 (Libvirt) • TCP/5900-6100 (VNC) • TCP/49152-49216 (Libvirt migration)
- 16. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue Networking and storage
- 17. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue K V M n e t w o r k i n g • KVM utilizes network bridges – similar to Vmware vSwitches and XenServer networks. • Bridging back ends: • Linux bridge • Open Vswitch (OVS)
- 18. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue B r i d g e b a c k e n d s – L i n u x b r i d g e • Linux bridge • In Linux kernel since version 2.2. • Fast, simple, reliable and mature. • Provides L2 requirements for CloudStack. • Lacks automation options and scalable tunneling.
- 19. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue B r i d g e b a c k e n d s – O p e n V s w i t c h • Open vSwitch • Written for multi-host virtualization environments with dynamic end points, higher level of abstraction and potential for hardware offloading. • Network state kept in a network state database (OVSDB). • Allows for better automation. • SDN options (tunneling). • External controllers: OpenDaylight, Nicira, VMware NSX. • Default bridge backend in XenServer 6.0 and later versions (does not work with basic zones). • Some issues with VLAN configuration and stability (depending on host OS and version).
- 20. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue N e t w o r k i n g e x a m p l e • Cloudbr0: • Bond0: Eth0 + Eth1 • Carries management, guest and storage traffic. • Cloudbr1: • Bond1: Eth2 + Eth3 • Carries public traffic. • Cloud0: • Internal bridge, carries system VM management traffic. • Managed by CloudStack – does not need configuration.
- 21. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue L i n u x b r i d g e c o n f i g u r a t i o n ( C e n t O S ) ifcfg-eth0: DEVICE=eth0 TYPE=Ethernet USERCTL=no MASTER=bond0 SLAVE=yes BOOTPROTO=none NM_CONTROLLED=no ONBOOT=yes ifcfg-bond0: DEVICE=bond0 ONBOOT=yes BONDING_OPTS='mode=1 miimon=100' BRIDGE=cloudbr0 NM_CONTROLLED=no Ifcfg-cloudbr0: DEVICE=cloudbr0 ONBOOT=yes TYPE=Bridge IPADDR=192.168.100.20 NETMASK=255.255.255.0 GATEWAY=192.168.100.1 NM_CONTROLLED=no DELAY=0
- 22. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue L i n u x b r i d g e c o n f i g u r a t i o n ( C e n t O S ) ifcfg-cloudbr1: DEVICE=cloudbr1 ONBOOT=yes TYPE=Bridge NM_CONTROLLED=no DELAY=0
- 23. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue L i n u x b r i d g e c o n f i g u r a t i o n ( C e n t O S ) Storage VLAN ifcfg-bond.100: DEVICE=bond0.100 VLAN=yes BOOTPROTO=static ONBOOT=yes TYPE=Unknown BRIDGE=cloudbr100 Storage VLAN ifcfg-cloudbr100: DEVICE=cloudbr100 ONBOOT=yes TYPE=Bridge VLAN=yes IPADDR=10.0.100.20 NETMASK=255.255.255.0 NM_CONTROLLED=no DELAY=0
- 24. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue L i n u x b r i d g e c o n f i g u r a t i o n ( C e n t O S ) • Check bridges: # brctl show bridge name bridge id STP enabled interfaces brbond0-113 8000.000c29b43c4d no bond0.113 cloudbr0 8000.000c29b55932 no bond0 cloudbr1 8000.000c29b45956 no bond1 cloudbr100 8000.000c29b43c4d no bond0.100 • Check bonds: # cat /proc/net/bonding/bond0 Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011) Bonding Mode: fault-tolerance (active-backup) Primary Slave: None Currently Active Slave: eth0 MII Status: up MII Polling Interval (ms): 100 Up Delay (ms): 0 Down Delay (ms): 0 Slave Interface: eth0 MII Status: up
- 25. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue O V S c o n f i g u r a t i o n ( C e n t O S ) • Requires OVS installation: # apt-get install openvswitch-switch # yum install openvswitch-<version>.rpm # yum install openvswitch-kmod-<version>.rpm • Add bridges and bonds with OVS command line tools: # ovs-vsctl add-br cloudbr0 # ovs-vsctl add-br cloudbr1 # ovs-vsctl add-bond cloudbr0 bond0 eth0 eth1 # ovs-vsctl add-bond cloudbr1 bond1 eth2 eth3
- 26. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue O V S c o n f i g u r a t i o n ( C e n t O S ) ifcfg-eth0: DEVICE=eth0 ONBOOT=yes NM_CONTROLLED=no BOOTPROTO=none HOTPLUG=no ifcfg-bond0: DEVICE=bond0 ONBOOT=yes DEVICETYPE=ovs TYPE=OVSBond OVS_BRIDGE=cloudbr0 BOOTPROTO=none BOND_IFACES="eth0 eth1" OVS_OPTIONS="bond_mode= active-backup lacp=off other_config:bond- detect-mode=miimon other_config:bond- miimon-interval=100" HOTPLUG=no Ifcfg-cloudbr0: DEVICE=cloudbr0 ONBOOT=yes DEVICETYPE=ovs TYPE=OVSBridge BOOTPROTO=static IPADDR=192.168.100.20 NETMASK=255.255.255.0 HOTPLUG=no
- 27. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue O V S c o n f i g u r a t i o n ( C e n t O S ) ifcfg-cloud0: DEVICE=cloud0 ONBOOT=yes DEVICETYPE=ovs TYPE=OVSBridge BOOTPROTO=static HOTPLUG=no IPADDR=169.254.0.1 NETMASK=255.255.0.0 VLAN configuration: # ovs-vsctl add-br cloudbr100 cloudbr0 100 ifcfg-cloudbr100 DEVICE=cloudbr100 ONBOOT=yes DEVICETYPE=ovs TYPE=OVSBridge BOOTPROTO=static IPADDR=10.0.100.20 NETMASK=255.255.255.0 HOTPLUG=no
- 28. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue O V S c o n f i g u r a t i o n ( C e n t O S ) • Some issues with network drivers cause VLANs to not propagate: # ovs-vsctl set interface eth0 other-config:enable-vlan- splinters=true • List bridges: # ovs-vsctl show 27daed4e-52f3-4177-9827-550f0e7df452 Bridge "cloudbr1" Port "vnet2" Interface "vnet2" Port "bond1" Interface "eth3" Interface "eth2" Port "cloudbr1" Interface "cloudbr1" type: internal Bridge "cloud0" … • List bonds: ovs-appctl bond/show bond0 ---- bond0 ---- bond_mode: active-backup bond may use recirculation: no, Recirc-ID : -1 bond-hash-basis: 0 updelay: 0 ms downdelay: 0 ms lacp_status: off active slave mac: 00:0c:xx:xx:xx:xx(eth0) slave eth0: enabled active slave may_enable: true slave eth1: enabled may_enable: true
- 29. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue O V S c o n f i g u r a t i o n ( C e n t O S ) • Agent properties file - /etc/cloudstack/agent/agent.properties: guest.network.device=cloudbr0 workers=5 private.network.device=cloudbr0 network.bridge.type=openvswitch port=8250 resource=com.cloud.hypervisor.kvm.resource.LibvirtComputingResource pod=3 zone=2 hypervisor.type=kvm guid=c3c6a2fc-13d3-3af1-ae2e-e48a21d9b883 public.network.device=cloudbr1 cluster=5 local.storage.uuid=2f2220ef-2624-4e69-b442-0a4ae5c5add6 domr.scripts.dir=scripts/network/domr/kvm LibvirtComputingResource.id=28 host=192.168.100.20
- 30. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue S t o r a g e • Disks, templates and snapshots use QCOW2 format. • Primary storage options: • NFS – recommended if performance is sufficient, easy option, supports CloudStack managed thin provisioning. • Local storage (does not support storage migration). • SharedMountpoint option used for iSCSI / FC. • CEPH. • SharedMountpoint: • Used to access pre-defined block storage. • Must be running a clustered file system. • Preconfigured on each host. • Consistent – same on each host – e.g. /mnt/primary
- 31. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue S t o r a g e – s h a r e d m o u n t p o i n t • KVM lacks a native clustered file system like VMFS or SR (CLVM). • Clustered file system options: • OCFS2 (Oracle) • GFS2 (RedHat) • CLVM (not officially supported in CloudStack) • CEPH
- 32. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue Management and troubleshooting
- 33. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue H o w d o I m a n a g e K V M • Countless tools (40+). • Most common: • Virsh command line tool: • Virt-manager: linux native but works well with SSH X session forwarding. • Configuration management: • Anything that utilises libvirt. • Ansible: virt modules • Puppet modules and Chef Cookbooks. # ssh –X kvmhost1 root@kvmhost1:~# virsh list Id Name State ----------------------------------- 2 r-540-VM running 3 s-548-VM running 4 v-509-VM running root@kvmhost1:~# virt-manager &
- 34. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue Tr o u b l e s h o o t i n g • Check KVM is running: # lsmod | grep kvm kvm_intel 151552 9 kvm 479232 1 kvm_intel • Log file: /var/log/cloudstack/agent/agent.log • Increase logging verbosity: # sed -i 's/INFO/DEBUG/g' /etc/cloudstack/agent/log4j-cloud.xml • KVM uses link local networking, hence connect to system VMs using: # ssh -i /root/.ssh/id_rsa.cloud -p 3922 root@<linklocalIP>
- 35. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue W h a t ’ s n e x t • HA (and IPMI control) is being developed for KVM. https://cwiki.apache.org/confluence/display/CLOUDSTACK/KV M+HA+with+IPMI+Fencing • DRS functionality has been discussed but is not yet in development.
- 36. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue Questions?
- 37. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue M o r e i n f o r m a t i o n • Background: • http://www.linux-kvm.org • http://wiki.qemu.org/KVM • https://libvirt.org/ • Installation: • http://docs.cloudstack.apache.org/projects/cloudstack- installation/en/4.9/hypervisor/kvm.html • Management tools: • http://www.linux-kvm.org/page/Management_Tools
- 38. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue M o r e i n f o r m a t i o n • Networking: • http://openvswitch.org • http://openvswitch.org/support/dist-docs/WHY-OVS.md.txt • Storage: • CEPH: http://docs.ceph.com/docs/hammer/rbd/rbd-cloudstack/ • OCFS2: https://oss.oracle.com/projects/ocfs2/ • GFS2: https://access.redhat.com/documentation/en- US/Red_Hat_Enterprise_Linux/6/html- single/Global_File_System_2/index.html
- 39. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue M o r e i n f o r m a t i o n • Slide deck: https://www.slideshare.net/ShapeBlue/ccna17-kvm- and-cloudstack • Blog: http://shapeblue.com/blog http://dsonstebo.wordpress.com • Email: [email protected] • Twitter: @dagsonstebo • Web: http://shapeblue.com
- 40. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue S h a m e l e s s s e l f p r o m o t i o n … … • My next talk: • What: Apache CloudStack Upgrade Best Practices • When: Tomorrow – Thursday @ 2:40PM • Where: Rafael (here….)