OpenShift Virtualization- Technical Overview.pdf

Technical presentation
OpenShift Virtualization
1

What is OpenShift
Virtualization?
3

Containers are not virtual machines
4
Infrastructure
Operating System
App 1 App 3
App 2
Hypervisor
Guest
OS
Guest
OS
Guest
OS
Infrastructure
Virtualization Containerization
App 1 App 3
App 2
● Containers are process isolation
● Kernel namespaces provide isolation and
cgroups provide resource controls
● No hypervisor needed for containers
● Contain only binaries, libraries, and tools
which are needed by the application
● Ephemeral

Virtual machines can be put into containers
5
● A KVM virtual machine is a process
● Containers encapsulate processes
● Both have the same underlying
resource needs:
○ Compute
○ Network
○ (sometimes) Storage

6
● Virtual machines
○ Running in containers, managed as Pods
○ Using the KVM hypervisor
● Scheduled, deployed, and managed by Kubernetes
● Integrated with container orchestrator resources and
services
○ Traditional Pod-like SDN connectivity and/or
connectivity to external VLAN and other networks
via multus
○ Persistent storage paradigm (PVC, PV,
StorageClass)

VM containers use KVM
7
● OpenShift Virtualization uses KVM, the Linux kernel
hypervisor
● KVM is a core component of the Red Hat Enterprise
Linux kernel
○ KVM has 10+ years of production use: Red Hat
Virtualization, Red Hat OpenStack Platform, and
RHEL all leverage KVM, QEMU, and libvirt
● QEMU uses KVM to execute virtual machines
● libvirt provides a management abstraction layer
HARDWARE
RHCOS
KVM
CPU/RAM STORAGE NETWORK
DRIVER DRIVER DRIVER
OTHER APPS
QEMU
libvirt

Virtual machines in a container world
9
● Provides a way to transition application components
which can’t be directly containerized into a Kubernetes
system
○ Integrates directly into existing k8s clusters
○ Follows Kubernetes paradigms:
■ Container Networking Interface (CNI)
■ Container Storage Interface (CSI)
■ Custom Resource Definitions (CRD, CR)
● Schedule, connect, and consume VM resources as
container-native
RHEL CoreOS
OpenShift
Physical Machine
VM pod App pod

Virtualization native to Kubernetes
10
● Operators are a Kubernetes-native way to introduce
new capabilities
● New CustomResourceDefinitions (CRDs) for native
VM integration, for example:
○ VirtualMachine
○ VirtualMachineInstance
○ VirtualMachineInstanceMigration
○ VirtualMachineSnapshot
○ DataVolume

Containerized virtual machines
11
Kubernetes resources
● Every VM runs in a launcher pod. The launcher process will
supervise, using libvirt, and provide pod integration.
Red Hat Enterprise Linux
● libvirt and qemu from RHEL are mature, have high
performance, provide stable abstractions, and have a
minimal overhead.
Security - Defense in depth
● Immutable RHCOS by default, SELinux MCS, plus KVM
isolation - inherited from the Red Hat Portfolio stack
Storage
Network
CPU
Memory
Device

Using VMs and containers together
12
● Virtual machines connected to pod networks
are accessible using standard Kubernetes
methods:
○ Service
○ Route
○ Ingress
● Network policies apply to VM pods the same
as application pods
● VM-to-pod, and vice-versa, communication
happens over SDN or ingress depending on
network connectivity

Virtual Machine Management
14
● Create, modify, and destroy virtual
machines, and their resources, using
the OpenShift web interface or CLI
● Use the virtctl command to
simplify virtual machine interaction
from the CLI

Virtual Machine creation
16
● Streamlined and simplified creation via the GUI or
create VMs programmatically using YAML
● Full configuration options for compute, network, and
storage resources
○ Clone VMs from templates or import disks using
DataVolumes
○ Pre-defined and customizable presets for
CPU/RAM allocations
○ Workload profile to tune KVM for expected
behavior
● Import VMs from VMware vSphere or Red Hat
Virtualization

Using templates for virtual machines
17
● Simplified and streamlined virtual machine
creation experience for VM consumers
● Administrators configure templates with an OS
disk, consumers select from the options

Creating a virtual machine
18
● Flavor represents the preconfigured CPU and
RAM assignments
● Storage, including PVC size and storage class, are
determined by the template administrator
● A default network configuration is defined in the
template
● Workload profile defines the category of workload
expected and is used to set KVM performance
flags
● The VM can be further customized by selecting
the option, or immediately created and deployed
○ Additional customization includes
CPU/memory, storage, network, cloud-init,
and more

Templates
20
● Templates are a core concept for
virtual machine creation with
● Red Hat provides default templates,
administrators can create and
customize additional as needed
● Boot sources provide disk images for
the template
● Creating VMs can be done from the
template page in just a few clicks

Create a template - General
21
● In addition to unique names, each template is
associated with a provider
○ Providers represent who created the
template, with optional support
information
● The guest operating system and source boot
disk are provided. A boot disk can be imported
during the process, or an ISO can be used to
boot and install the OS
● A default flavor, representing CPU and memory
allotments, is assigned
● Workload type determines optimizations to
balance between performance and efficiency

Create a template - Networks
22
● Add or edit network adapters
● One or more network connections
○ Pod network for the default SDN
○ Additional multus-based interfaces
for specific connectivity
● Multiple NIC models for guest OS
compatibility or paravirtualized
performance with VirtIO
● Masquerade, bridge, or SR-IOV
connection types
● MAC address customization if desired

Create a template - Storage
23
● Add or edit persistent storage
● Disks can be sourced from
○ Imported QCOW2 or raw images
○ New or existing PVCs
○ Clone existing PVCs
● Use SATA/SCSI interface for compatibility
or VirtIO for paravirtual performance
● For new or cloned disks, select from
available storage classes
○ Customize volume and access mode as
needed
○ RWX PVCs are required for live
migration

Create a template - Advanced
24
● Customize the operating system
deployment using cloud-init
scripts
○ Guest OS must have
cloud-init installed
○ RHEL, Fedora, etc. cloud images
○ Default templates will
auto-generate a simple
cloud-init to set the password

Virtual Machine Import
26
● Wizard supports importing from VMware or
Red Hat Virtualization
○ Single-VM workflow
● VMware import uses VDDK to expedite the
disk import process
○ User is responsible for downloading the
VDDK from VMware and adding it to a
container image
● Credentials stored as Secrets
● ResourceMapping CRD configures default
source -> destination storage and network
associations

Virtual Machine - Overview
28
● General overview about the virtual machine
● Information populated from guest when
integrations are available
○ IP address, etc.
● Inventory quickly shows configured hardware
with access to view/manage
● Utilization reporting for CPU, RAM, disk, and
network
● Events related to the Pod, scheduling, and
resources are displayed

Virtual Machine - Actions
29
● Actions menu allows quick access to
common VM tasks
○ Start/stop/restart
○ Live migration
○ Clone
○ Edit application group, labels, and
annotations
○ Delete
● Accessible from all tabs of VM details
screen and the VM list

Virtual Machine - Details
30
● Details about the virtual machine
○ Labels, annotations
○ Configured OS
○ Template used, if any
○ Configured boot order
○ Associated workload profile
○ Flavor
● Additional details about scheduling
○ Node selector, tolerations, (anti)affinity
rules
● Services configured for the VM

Virtual Machine - Console
31
● Browser-based access to the serial and
graphical console of the virtual machine
● Access the console using native OS tools,
e.g. virt-viewer, using the virtctl CLI
command
○ virtctl console vmname
○ virtctl vnc vmname

Virtual Machine - Disks and NICs
32
● Add, edit, and remove NICs
and disks for non-running
virtual machines

Destroying a Virtual Machine
34
● Deleting a VM removes the VM definition
○ Optionally delete PVC-backed disks
associated with the VM
● Running VMs are terminated first
● Other associated resources, e.g. Services, are
not affected

Overview Virtual Machine metrics
36
● Summary metrics for 1, 6, and 24 hour periods are
quickly viewable from the VM overview page
● Clicking a graph will display it enlarged in the
metrics UI

Detailed Virtual Machine metrics
37
● Virtual machine, and VM pod, metrics are collected
by the OpenShift metrics service
○ Available under the kubevirt namespace in
Prometheus
● Available per-VM metrics include
○ Active memory
○ Active CPU time
○ Network in/out errors, packets, and bytes
○ Storage R/W IOPS, latency, and throughput
● VM metrics are for VMs, not for VM pods
○ Management overhead not included in output
○ Look at virt-launcher pod metrics for
● No preexisting Grafana dashboards

Containerizing KVM
39
Trusted, mature KVM wrapped in modern management and automation
RHEL CoreOS Host
KubeVirt Container
RHV Host
RHV-M Console / CLI
vdsm
libvirt
QEMU / KVM
VM
Red Hat Virtualization
OpenShift Console / CLI
kubelet
libvirt
QEMU / KVM
VM
OSP Compute
OpenStack Horizon / CLI
nova-compute
libvirt
QEMU / KVM
VM
Red Hat OpenStack Platform

Architectural Overview
40
kubelet
(DaemonSet) Pod
virt-handler
Cluster Services Nodes
VM Pod
virt-launcher
Other Pod(s)
container 1
libvirtd container 2
VM container n
API Server
virt-controller

Containerized virtual machines
43
● Inherit many features and functions from Kubernetes
○ Scheduling, high availability, attach/detach resources
● Containerized virtual machines have the same characteristics as
non-containerized
○ CPU, RAM, etc. limitations dictated by libvirt and QEMU
○ Linux and Windows guest operating systems
● Storage
○ Use Persistent Volumes Claims (PVCs) for VM disks
○ Containerized Data Importer (CDI) import VM images
● Network
○ Inherit pod network by default
○ Multus enables direct connection to external network

Virtual Machine Networking
46
● Virtual machines optionally connect to the
standard pod network
○ OpenShift SDN, OVNKubernetes
○ Partners, such as Calico, are also supported
● Additional network interfaces accessible via
Multus:
○ Bridge, SR-IOV
○ VLAN and other networks can be created at
the host level using nmstate
● When using at least one interface on the default
SDN, Service, Route, and Ingress configuration
applies to VM pods the same as others

47
● Pod, service, and machine network are configured
by OpenShift automatically
○ Use kernel parameters (dracut) for
configuration at install - bond0 in the example
to the right
● Use kubernetes-nmstate, via the NMstate
Operator, to configure additional host network
interfaces
○ bond1 and br1 in the example to the right
● VM pods connect to one or more networks
simultaneously
The following slides show an example of how this
setup is configured Node
NIC NIC
br0 br1
Service
Net
NIC
bond0
Pod
Net
Example host network configuration
NIC
bond1
SDN
Multus
Machine Net

48
Host bond configuration
● NodeNetworkConfiguration-
Policy (NNCP)
○ Nmstate operator CRD
○ Configure host network
using declarative
language
● Applies to all nodes specified
in the nodeSelector,
including newly added nodes
automatically
● Update or add new NNCPs
for additional host configs
Node
NIC NIC
br0 br1
Service
Net
NIC
bond0
Pod
Net
NIC
bond1
SDN
Multus
Machine Net

49
Host bridge configuration
Node
NIC NIC
br0 br1
Service
Net
NIC
bond0
Pod
Net
NIC
bond1
SDN
Multus
Machine Net

50
Host network status
● Use the
NodeNetworkConfigurationEnactment
(NNCE) object to view status of NNCP
application
● Further details of the node network state can
be seen using the NodeNetworkState CRD
○ oc get nns/node-name -o yaml

51
Connecting Pods to networks
● Multus uses CNI network definitions in the
NetworkAttachmentDefinition to allow access
○ net-attach-def are namespaced
○ Pods cannot connect to a net-attach-def
in a different namespace
● cnv-bridge and cnv-tuning types are used to
enable VM specific functions
○ MAC address customization
○ MTU and promiscuous mode
○ sysctls, if needed
● Pod connections are defined using an annotation
○ Pods can have many connections to many
networks

52
Connecting VMs to networks
● Virtual machine interfaces describe NICs
attached to the VM
○ spec.domain.devices.interfaces
○ Model: virtio, e1000, pcnet, rtl8139, etc.
○ Type: masquerade, bridge
○ MAC address: customize the MAC
● The networks definition describes the connection
type
○ spec.networks
○ Pod = default SDN
○ Multus = secondary network using Multus
● Using the GUI makes this easier and removes
the need to edit / manage connections in YAML

Virtual Machine Storage
54
● OpenShift Virtualization uses the Kubernetes
PersistentVolume (PV) paradigm
● PVs can be backed by
○ In-tree iSCSI, NFS, etc.
○ CSI drivers
○ Local storage using host path provisioner
○ OpenShift Container Storage
● Use dynamically or statically provisioned PVs
● RWX is required for live migration
● Disks are attached using VirtIO or SCSI controllers
○ Connection order specified in the VM definition
● Boot order customized via VM definition

VM disks in PVCs
55
● VM disks on FileSystem mode PVCs are created as thin
provisioned raw images
○ As of OCP 4.7 / OpenShift Virtualization 2.6,
DataVolumes can create thin or thick volumes
● Block mode PVCs are attached directly to the VM
● CSI operations, e.g. snapshot and clone, should be used
with care
○ Use DataVolumes to clone VM disks
○ Use VM details interface for (powered off) VM snaps
● PVC resize does not modify the size of the VM disk
○ Not currently supported
● Hot add is not supported (for any virtual hardware)
OpenShift Node(s)
Driver
registrar
Side-
cars
Storage Array
Vendor CSI Deployment
Node
Driver
CSI
Controller
kubelet

DataVolumes
56
● VM disks can be imported from multiple sources using
DataVolumes, e.g. an HTTP(S) or S3 URL for a QCOW2 or
raw disk image, optionally compressed
● VM disks can be cloned / copied from existing PVCs
● DataVolumes are created as distinct objects or as a part of
the VM definition as a dataVolumeTemplate
● DataVolumes use the ContainerizedDataImporter to
connect, download, and prepare the disk image
● DataVolumes create PVCs based on defaults defined in
the kubevirt-storage-class-defaults ConfigMap or
according to the profile (as of version 4.8)

Storage Profiles
57
● Introduced with OpenShift Virtualization 4.8
● Provide default settings and properties for
StorageClasses used by DataVolumes
● Created automatically for every StorageClass
● Preconfigured values for some storage providers,
administrator can modify and customize
● DataVolume definitions only need to specify
StorageClass, without knowledge of underlying
details
○ spec.storage doesn’t require fields other than
the size and StorageClass

Data source
58
CDI
Controller
PVC
PV
VM 1. The user creates a virtual
machine with a DataVolume
2. The StorageClass is used to
satisfy the PVC request
3. The CDI controller creates an
importer pod, which mounts
the PVC and retrieves the
disk image. The image could
be sourced from S3, HTTP, or
other accessible locations
4. After completing the import,
the import pod is destroyed
and the PVC is available for
the VM
Requests
Import Pod
Writes
Creates
1
2
3
4
Containerized Data Importer

Ephemeral Virtual Machine Disks
59
● VMs booted via PXE or using a container image can be
“diskless”
○ PVCs may be attached and mounted as secondary
devices for application data persistence
● VMs based on container images use the standard
copy-on-write graph storage for OS disk R/W
○ Consider and account for capacity and IOPS
during RHCOS disk sizing if using this type
● An emptyDisk may be used to add additional
ephemeral capacity for the VM

Helper disks
60
● OpenShift Virtualization attaches disks to VMs for
injecting data
○ Cloud-Init
○ ConfigMap
○ Secrets
○ ServiceAccount
● These disks are read-only and can be mounted by the OS
to access the data within

Comparing with
traditional
virtualization
platforms
61

Live Migration
62
● Live migration moves a virtual machine from one node to another in the OpenShift cluster
● Can be triggered via GUI, CLI, API, or automatically
● RWX storage is required, cannot use bridge connection to pod network
● Live migration is cancellable by deleting the API object
● Default maximum of five (5) simultaneous live migrations
○ Maximum of two (2) outbound migrations per node, 64MiB/s throughput each
Migration Reason vSphere RHV OpenShift Virtualization
Resource contention DRS Cluster policy Pod eviction policy, pod
descheduler
Node maintenance Maintenance mode Maintenance mode Maintenance mode, node
drain

Automated live migration
63
● OpenShift / Kubernetes triggers Pod rebalance actions based on multiple factors
○ Soft / hard eviction policies
○ Pod descheduler
○ Pod disruption policy
○ Node resource contention resulting in evictions
■ Pods are Burstable QoS class by default
■ All memory is requested in Pod definition, only CPU overhead is requested
● Pod rebalance applies to VM pods equally
● VMs will behave according to the eviction strategy
○ LiveMigrate - use live migration to move the VM to a different node
○ No definition - terminate the VM if the node is drained or Pod evicted

VM scheduling
64
● VM scheduling follows pod scheduling rules
○ Node selectors
○ Taints / tolerations
○ Pod and node affinity / anti-affinity
● Kubernetes scheduler takes into account many additional factors
○ Resource load balancing - requests and reservations
○ Large / Huge page support for VM memory
○ Use scheduler profiles to provide additional hints (for all Pods)
● Resources are managed by Kubernetes
○ CPU and RAM requests less than limit - Burstable QoS by default
○ K8s QoS policy determines scheduling priority: BestEffort class is evicted before
Burstable class, which is evicted before Guaranteed class

Node Resource Management
65
● VM density is determined by multiple factors controlled at the cluster, OpenShift Virtualization,
Pod, and VM levels
● Pod QoS policy
○ Burstable (limit > request) allows more overcommit, but may lead to more frequent migrations
○ Guaranteed (limit = request) allows less overcommitment, but may have less physical resource
utilization on the hosts
● Cluster Resource Override Operator provides global overcommit policy, can be customized per
project for additional control
● Pods request full amount of VM memory and approx. 10% of VM CPU
○ VM pods request a small amount of additional memory, used for libvirt/QEMU overhead
■ Administrator can set this to be overcommitted

High availability
66
● Node failure is detected by Kubernetes and results in the Pods from the lost node being
rescheduled to the surviving nodes
● VMs are not scheduled to nodes which have not had a heartbeat from virt-handler, regardless of
Kubernetes node state
● Additional monitoring may trigger automated action to force stop the VM pods, resulting in
rescheduling
○ May take up to 5 minutes for virt-handler and/or Kubernetes to detect failure
○ Liveness and Readiness probes may be configured for VM-hosted applications
○ Machine health checks can decrease failure detection time

Terminology comparison
67
Feature RHV OpenShift Virtualization vSphere
Where VM disks are stored Storage Domain PVC datastore
Policy based storage None StorageClass SPBM
Non-disruptive VM
migration
Live migration Live migration vMotion
Non-disruptive VM storage
migration
Storage live migration N/A Storage vMotion
Active resource balancing Cluster scheduling policy Pod eviction policy, descheduler Dynamic Resource
Scheduling (DRS)
Physical network
configuration
Host network config (via
nmstate w/4.4)
nmstate Operator, Multus vSwitch / DvSwitch
Overlay network
configuration
OVN OCP SDN (OpenShiftSDN,
OVNKubernetes, and partners),
Multus
NSX-T
Host / VM metrics Data warehouse +
Grafana (RHV 4.4)
OpenShift Metrics, health checks vCenter, vROps

Deploy and configure
69
● OpenShift Virtualization is deployed as an
Operator utilizing multiple CRDs, ConfigMaps, etc.
for primary configuration
● Many aspects are controlled by native Kubernetes
functionality
○ Scheduling
○ Overcommitment
○ High availability
● Utilize standard Kubernetes / OpenShift practices
for applying and managing configuration

Compute configuration
70
● VM nodes should be physical with CPU virtualization technology enabled in the BIOS
○ Nested virtualization works, but is not supported
○ Emulation works, but is not supported (and is extremely slow)
● Node labeler detects CPU type and labels nodes for compatibility and scheduling
● Configure overcommitment using native OpenShift functionality - Cluster Resource Override
Operator
○ Optionally, customize the project template so that non-VM pods are not overcommitted
○ Customize projects hosting VMs for overcommit policy
● Apply Quota and LimitRange controls to projects with VMs to manage resource consumption
● VM definitions default to all memory “reserved” via a request, but only a small amount of CPU
○ CPU and memory request/limit values are modified in the VM definition

Network configuration
71
● Apply traditional network architecture decision framework to OpenShift Virtualization
○ Resiliency, isolation, throughput, etc. determined by combination of application, management,
storage, migration, and console traffic
○ Most clusters are not VM only, include non-VM traffic when planning
● Node interface on the MachineNetwork.cidr is used for “primary” communication, including SDN
○ This interface should be both resilient and high throughput
○ Used for migration and console traffic
○ Configure this interface at install time using kernel parameters, reinstall node if configuration
changes
● Additional interfaces, whether single or bonded, may be used for traffic isolation, e.g. storage and VM
traffic
○ Configure using nmstate Operator, apply configuration to nodes using selectors on NNCP

Storage configuration
72
● Shared storage is not required, but very highly encouraged
○ Live migration depends on RWX PVCs
● Create shared storage from local resources using OpenShift Container Storage
○ RWX file and block devices for live migration
● No preference for storage protocol, use what works best for the application(s)
● Storage backing PVs should provide adequate performance for VM workload
○ Monitor latency from within VM, monitor throughput from OpenShift
● For IP storage (NFS, iSCSI), consider using dedicated network interfaces
○ Will be used for all PVs, not just VM PVs
● Certified CSI drivers are recommended
○ Many non-certified CSI provisioners work, but do not have same level of OpenShift testing
● Local storage may be utilized via the Host Path Provisioner

Deploying a VM operating system
73
Creating virtual machines can be accomplished in multiple ways, each offering different options and capabilities
● Start by answering the question “Do I want to manage my VM like a container or a traditional VM?”
● Deploying the OS persistently, i.e. “I want to manage like a traditional VM”
○ Methods:
■ Import a disk with the OS already installed (e.g. cloud image) from a URL or S3 endpoint using a
DataVolume, or via CLI using virtctl
■ Clone from an existing PVC or VM template
■ Install to a PVC using an ISO
○ VM state will remain through reboots and, when using RWX PVCs, can be live migrated
● Deploying the OS non-persistently, i.e. “I want to manage like a container”
○ Methods:
■ Diskless, via PXE
■ Container image, from a registry
○ VM has no state, power off will result in disk reset. No live migration.
● Import disks deployed from a container image using CDI to make them persistent

Deploying an application
74
Once the operating system is installed, the application can be deployed and configured several ways
● The application is pre-installed with the OS
○ This is helpful when deploying from container image or PXE as all components can be managed and
treated like other container images
● The application is installed to a container image
○ Allows the application to be mounted to the VM using a secondary disk. Decouples OS and app lifecycle.
When used with a VM that has a persistently deployed OS this breaks live migration
● The application is installed after OS is installed to a persistent disk
○ cloud-init - perform configuration operations on first boot, including OS customization and app
deployment
○ SSH/Console - connect and administer the OS just like any other VM
○ Ansible or other automation - An extension of the SSH/console method, just automated

More information
76
● Documentation:
○ OpenShift Virtualization: https://docs.openshift.com
○ KubeVirt: https://kubevirt.io
● Demos and video resources: http://demo.openshift.com
● Labs and workshops: coming soon to RHPDS

linkedin.com/company/red-hat
youtube.com/user/RedHatVideos
facebook.com/redhatinc
twitter.com/RedHat
Red Hat is the world’s leading provider of enterprise
open source software solutions. Award-winning
support, training, and consulting services make
Red Hat a trusted adviser to the Fortune 500.
Thank you
77

OpenShift Virtualization- Technical Overview.pdf

Recommended

More Related Content

What's hot (20)

Similar to OpenShift Virtualization- Technical Overview.pdf (20)

Recently uploaded (20)

OpenShift Virtualization- Technical Overview.pdf