System Virtual Machines

Outline
• Need and genesis of system Virtual Machines
• Basic concepts
– User Interface and Appearance
– State Management
– Resource Control
– Bare Metal and Hosted Virtual Machines
– Co-designed Virtual Machines
• Case-Studies
– Native Virtual Machines: Xen, Vmware-Esxi
– Hosted Virtual Machines: Vmware-Workstation,
Palazzo, Linux-KVM
Feb 26, 2019 System Virtual Machines 2
 Need for System VMs
• Multiple single-application Virtual Machines
• Multiple secure environments
• Managed application environments
• Mixed-OS environments
• Legacy applications
• Multi-platform application development and
testing environment
• System Encapsulation

Feb 26, 2019 System Virtual Machines 3

Conceptual view of System VMs

Feb 26, 2019 System Virtual Machines 4

 Key concepts of System VMs
• Outward appearance
• State Management
• Resource Control
• VM realization:
– Native VMs
– Hosted VMs
Assumptions: Uni-processor machine supporting
same ISA VMs
Feb 26, 2019 System Virtual Machines 5
 Outward Appearance
• System VMs give the illusion of hosting multiple
systems on a single machine
• Using software virtualizers:
– Time-multiplexing of resources from one VM to another
• Using Hardware replication:
– Independent sets of replicated hardware resources
dedicated to specific VMs
– Common resources are time-shared
• Hosted VMs:
– One of the VM is more privileged than others
– Other VMs are manifested in special environments of the
hostOS.

Feb 26, 2019 System Virtual Machines 6

 State Management
• The VM state is composed of
collective information from
processor registers, memory
pages, files etc.
• Single machine hardware may
not be sufficient to hold
multiple VMs state
information.
• Store each VM state in fixed
locations in the VMM’s
memory and identify using a
pointer to the present active
VM.
• VM State contents can be
accessed through redirection
or copy mode.
Using Indirection Using Copy
Feb 26, 2019 System Virtual Machines 7
 Resource Control
• The VMM maintains overall control of all hardware
resources.
• Each VM is allocated virtual resources according to
configuration specifications.
• VMM allows for direct execution of non-privileged
instructions from within VM’s environment.
• All privileged instructions trap to the VMM for processing.
• VMM handles the timer interrupts
• All shared resources are time-shared across contesting VMs
using fair-shared or credit share scheduling.

8
Feb 26, 2019 System Virtual Machines
 VM Realization:
Native & Hosted VMs

Feb 26, 2019 System Virtual Machines 9

 Resource Virtualization - Processors
• Processors:
– Same ISA VMs
• Direct execution through VM
• Emulation or para-virtualization of privileged instructions
– Different ISA VMs
• Emulation or binary translation

Prerequisite come prepared for the rest of classes by

reading Appendix A – Real Machines from Smith &
Nair’s Virtual Machine text book.

Feb 26, 2019 System Virtual Machines 10

 ISA Virtualizability
• User ISA: Instructions that enable use of hardware
(innocuous instructions )
• System ISA: Instructions that enable management of
hardware and access to nonconcurrent devices
(sensitive and privileged instructions)
• ISA must allow multiple modes of operation
– User mode (for innocuous instrs)
– Privileged mode (for privileged instrs)
• All privileged instructions when executed in user mode
must trap to OS.
– Control-sensitive instructions are those that change the
configuration of the system resources
– Behavior-sensitive instructions are those whose behavior
or results change with the configuration of resources
Feb 26, 2019 System Virtual Machines 11
 Necessary condition for ISA
Virtualizability

User ISA User ISA

System ISA
System ISA

Feb 26, 2019 System Virtual Machines 12

Components of a VMM

Feb 26, 2019 System Virtual Machines 14

Handling of Privileged Instruction in
GuestOS

Feb 26, 2019 System Virtual Machines 15

 Difficulty in i86 Virtualization
• Some of the sensitive instructions in Intel IA-
32 are not privileged (POPF instruction)
• Handling problem instructions
– Interpret the guest software
– Scan and patch before execution (standard
techniques of binary translation are applied)
– Every sensitive instruction leads to a change in
privilege level since GuestOS is executing in non-
privileged mode.
Feb 26, 2019 System Virtual Machines 16
Execution of Privileged instructions in
VM/370
• System VM/370 is Application
ISA virtualizable Privileged Instrs
trap to VMM &
reflected to OS
– All sensitive and
OS
privileged Application Privileged Instrs
instructions trap to Privileged Instrs trap to VMM and
emulated in
OS when executed in trap to OS
VMM
user mode. OS VMM
– Supports two CPU Privileged Instrs
directly executed Privileged Instrs
modes – user and directly executed

system Hardware Hardware

Feb 26, 2019 System Virtual Machines 17

 Concept of Virtual CPUs
• Implementation specifics of virtualizing a CPU
• Hypervisor schedulers
– What are they?
• Multiple VMs share the same CPU
• Variants of time-sharing CPU schedulers are mostly
used
– How do they impact the application performance
when executed inside the VM?

Feb 26, 2019 System Virtual Machines 18

 Resource Virtualization - Memory
• Most virtualization technologies extend the prevailing
Virtual Memory concept to support memory virtualization
for VMs.
• Each VM is given a logical view of its real memory using the
virtual memory in VMM.
• Each VM can support virtual memory in its address space
on its real memory.
• The real memory of any VM is further mapped to the
physical memory by the VMM.
• Every virtual address in application process undergoes two
translations:
– Application virtual address to VM’s real memory
– VM’s real memory to Host’s physical memory

Feb 26, 2019 System Virtual Machines 19

 Support for Virtual Memory
• Virtual Memory basics:
– Application is provided with a logical view of memory in the form of
virtual address space.
– Underlying OS manages the real memory (hardware memory)
– Per process Page-Tables used to map logical (virtual address) to real
(physical address) memory.
– TLBs cache the most commonly used mappings
– ISA defines the support for page table walks and page sizes.
• Segment registers or Relocation registers to support virtual
addresses
• Address translation hardware
• Page-tables and TLBs to map physical to virtual addresses
• Page table cache structures to fasten the address translation
lookups.

Feb 26, 2019 System Virtual Machines 20

Virtual Memory Abstraction

Feb 26, 2019 System Virtual Machines 21

Virtual to Physical Address Translation

Feb 26, 2019 System Virtual Machines 22

Page Table walk in x86-64 with 4KB Pages

Feb 26, 2019 System Virtual Machines 23

63 47 38 29 20 0
L3 Index L2 Index L1 Index Page Offset

V-CR3

63 47 38 29 20 0

L3 Index L2 Index L1 Index Page Offset

CR3

Feb 26, 2019 System Virtual Machines 24

 Memory Virtualization Issues with
page based structures
• Hypervisors perform two-level address
translation to support memory virtualization:
– Guest Virtual Address to Guest Physical Address
– Guest Physical Address to Host Physical Address
• Every TLB/Cache miss introduces this penalty.
• Virtualization using this technique makes it
unattractive for workloads that have
significant memory reads and writes!

Feb 26, 2019 System Virtual Machines 25

 Software methods for memory
virtualization – Shadow page tables
• GuestOS of each VM
maintains its own page
tables to map GVA 
GPA
• VMM maintains
shadow page tables to
map GPA  HPA
• Shadow page tables
are used by the
hardware to translate
virtual addresses to
keep TLB updated
• The Page Table Pointer
is virtualized and VMM
manages it.
• This technique is used
on hardware where
the ISA dictates page
Feb 26, 2019 System Virtual Machines
table architecture.
26
 Software methods for memory
virtualization – Virtualized TLBs
OS based TLB Management Virtualizing architected TLBs
• Some ISAs allow OS to decide • The VMM manages each VM’s
on the page table structure TLB by maintaining a copy and
and hence page walks in such also the physical TLB.
case are software based. • Any instruction that modify
• TLBs are architected and the TLB are sensitive and trap
special instructions are to VMM.
available to update them. • When a VM is activated the
• A TLB miss results in an OS VMM copies the virtual TLB’s
trap and the page table entries into the physical TLB
information is used by the OS after appropriately translating
to update the TLB. the VM’s PA to Host’s PA.
• Recent RISC ISAs use
architected TLBs

Feb 26, 2019 System Virtual Machines 27

Virtualizing an Architected TLB
• VMM needs to virtualize the TLB
when the ISA provides a software
managed TLB (SPARC ISA).
• VMM maintains a copy of the VM’s
TLB contents and manages it.
– VMM copies the Guest OS specific
TLB contents whenever the VM is
activated (has an issue?).
– Each entry in TLB is associated
with an ASID (address space
identifier) and the TLB can host
multiple VM address mappings.
• VMM copies the VM’s TLB contents
to the real TLB after appropriately
translating the GPA->HPA.

Feb 26, 2019 System Virtual Machines 28

 Hardware support for memory
virtualization – Nested Page Tables
• Nested page tables need
hardware support and
enables memory
virtualization
• Processor has two page
table pointers to complete a
page table translation
– One points to the guest page
table gptr
– Other points to the host
page table hptr
• Guest page table holds the
translation for GVAGPA
• Host page table holds
GPAHPA translation

Feb 26, 2019 System Virtual Machines 29

 I/O Device Virtualization
• The complicated part of system virtualization is
the Input/Output device virtualization.
• No standard interface; each device has it’s own
intricacies with regard to control and access.
• A general purpose OS has support for a large and
a variety of I/O devices; number of different I/O
devices also is growing.
• Unlike the processor and memory, I/O devices are
oblivious of sharing and concurrency; OS
supports these features by way of OS level
resource abstractions.
Feb 26, 2019 System Virtual Machines 31
 Virtualizing I/O Activity
I/O Instruction Level Device Driver Level
• I/O devices communicate with the • Device drivers abstract out the
processors using special PIO device specific I\O instructions
instructions or using memory and provide the interface to
mapping feature. support OS level resource
• All PIO instructions are privileged abstractions.
so execute in privileged mode and • Virtual device abstractions are
hence trap to VMM when executed created in the GuestOS which
in GuestOS. then connect to the physical
• Execution of PIO instruction by the device drivers for execution of
GuestOS needs support at the PIO instructions.
VMM end in terms of address • Commonly used techniques are
translation. emulation and para-virtualization
• Issues also with bulk execution of with front-end and backend pairs.
I/O requests and from various VMs. • Front-end drivers are resident in
• Normally adopted in device the GuestOS, back-end drivers
emulation modes of virtualization reside in the hypervisor or the
• Has high performance overheads. hostOS.

Feb 26, 2019 System Virtual Machines 32

I/O Devices - Recap
• In most general purpose
OS’s I/O devices are
accessed using the system
call interface.
• The OS has different
abstractions for different
devices, viz files for disk
storage, sockets for network
interfaces, terminals for
display devices, etc.
• Most I/O devices today
using memory mapping to
interface with the system.
Feb 26, 2019 System Virtual Machines 33
Virtualizing Devices

Feb 26, 2019 System Virtual Machines 34

 Performance side-effects of VMs
• VM setup: Extra time is involved in setting up the
appropriate registers, program counters and timing
facilities before a VM can be activated.
• Emulation/para-virtualization: Privileged instructions need
to be emulated or para-virtualized by the VMM. This leads
to higher time spent on executing such instructions.
• Interrupt handling: All interrupts are intercepted by the
VMM before being passed on to the GuestOS of a VM.
• State saving: For every VMM entry, a VM’s state needs to
be saved to enable control transfer.
• Bookkeeping: VMM has to perform special operations to
reflect equivalent behavior of that of a real machine.
• Time elongation: Some instructions, particularly those
involved with memory management, require more
processing time because of the requirement to
walkthrough multiple structures for the same operation.
Feb 26, 2019 System Virtual Machines 35
 Virtual Machine Assists
• Hardware extensions or support to improve
performance of applications when executing
inside a VM are called VM assists.
• Instruction emulation assists:
– VMM emulates an instruction using a routine whose
operation depends on whether the VM is executing in
a system mode or user mode.
– IBM system/370 used a hardware assist that would
detect the VM’s execution mode while performing
emulation.
– Such assists use the knowledge that the hardware is
virtualized!

Feb 26, 2019 System Virtual Machines 36

 VMM assists
• VMM improvement can potentially benefit all hosted
VMs.
– Hardware assisted context switch between VM and VMM –
storing and restoring of machine state registers using
hardware
– Privileged Instructions Decode – Hardware assisted
decoding might help improve performance along with
software techniques to optimize.
– Virtual Timers – True implementation of timers in VMs
need hardware assistance. Ex. System/370 ISA requires
that the virtual timers in a VM be located in a specific
place inside the VM’s memory and the VMM decrements
this timer counter every time a true timer interrupt occurs.
– Enhanced ISA for VMM support – I/O device assignment,
DMA remapping, Interrupt remapping, interrupt posting
are some of the evolving list of assists.

Feb 26, 2019 System Virtual Machines 37

OS use of DMA Remapping

Feb 26, 2019 System Virtual Machines 38

VMM usage of DMA Remapping Direct IO

Feb 26, 2019 System Virtual Machines 39

 IO-Virtualization Assists
• Single Root IO Virtualization (SR-IOV)
– Effort by the PCI-SIG to enable I/O device
virtualization ensuring clean isolation interfaces
– Isolation of
• Device memory
• I/O streams
• I/O interrupts
• Control and I/O operations
• Errors

Feb 26, 2019 System Virtual Machines 40

SR-IOV NIC Example

Feb 26, 2019 System Virtual Machines 41

 Multi-Processor Virtualization
• Multi-core and many-core architectures have
higher number of processors that share
memory and I/O devices.
• Techniques for system isolation
– Dynamic partitioning
• Time-sharing
– Static partitioning
• Space-sharing
• When does static partitioning help?

Feb 26, 2019 System Virtual Machines 42

 Prerequisite
• https://classes.soe.ucsc.edu/cmps111/Fall02/
Chapter08.pdf
• Appendix A section 7 of Virtual Machines,
Smith & Nair

Feb 26, 2019 System Virtual Machines 43

 Partitioning Techniques
• Physical Partitioning Partitioning
Techniques

– Failure Isolation
– Security Isolation
– Oriented towards
Hardware Software
Supported Supported

specific system level

objectives
• Logical Partitioning
Logical
System
Partitioning
VMs

– Flexible resource Physical

Partitioning
Process
VMs
sharing
– Improved utilization
– Fault Tolerant Micro-code Hypervisor Different
Based Based Same ISA ISA
Feb 26, 2019 System Virtual Machines 44
 Case Study
• Micro-code based Logical Partitioning:
– Extended ISA provides the necessary support in hardware
for partitioning
• IBM System/390 LPAR
• Hypervisor based Logical Partitioning
– Hardware support for extra supervisory level allows
hypervisors (system software layer) to execute under
privileged mode that is different from privileged mode of
the GuestOS. GuestOS still executes under privileged mode
and its applications execute in user mode.
• HP Superdome servers
• IBM Dynamic LPAR
• Cellular Disco

Feb 26, 2019 System Virtual Machines 45

 Virtualization:
Different Host and Guest ISAs
• What could be the reasons for supporting this
model of virtualization?
• Additional complexities to handle:
– Instructions of target ISA must be transparently
and dynamically translated to the host ISA.
– Memory model of the target ISA must be
observed in the VM so the VMM/hypervisor
needs to handle the memory ordering and
coherence rules of the target ISA

Feb 26, 2019 System Virtual Machines 46

Virtualizing Uniprocessor Cluster
Virtual Machine 1 Virtual Machine 2 Virtual Machine 3

P P P

M I/O M I/O M I/O

VMM VMM VMM

Real Processor Real Processor Real Processor

Real Real I/O Real Real I/O Real Real I/O

Memory Memory Memory

Feb 26, 2019 System Virtual Machines 47

 Virtualizing SMP Host
Virtual Machine 1 Virtual Machine 2

P P P

M I/O M I/O

VMM

Real Processor Real Processor Real Processor

Real Real I/O

Memory
Feb 26, 2019 System Virtual Machines 48
 Concerns with multi-core systems for
virtualizing
• Concurrency capability of the hypervisor or host-OS
can cause bottlenecks for consolidation
– Serial data structures of the kernel
– Increased resource usage (CPU cycles/Memory)
– Serial interrupt and device drivers
• Mitigation strategies
– Concurrent device design with device interrupt delivery
to any core
– Concurrent kernel design
– Assigning specific cores to kernel services
– Scheduling VMs on the same cores even without affinity
assignment

Feb 26, 2019 System Virtual Machines 49

 Co-designed Virtual Machines
• Co-design exploration space:
– To maintain code portability and backward compatibility, ISA
supported on hardware evolves in restricted sense when
compared to hardware that executes its and software that uses
it!
– Virtual Machines offer an opportunity to explore new ISA
features using a co-design approach.
– Host ISA is designed keeping in view the Virtual Machine that
runs on it.
– Target ISA uses a software/hardware approach to support the
host ISA dynamically keeping performance, efficiency and power
as the target goals.
– Mostly restricted to processor virtualization.
– Currently has been restricted to research interest and not
explored for workload consolidation.

Feb 26, 2019 System Virtual Machines 51

 Examples of Co-Designed VMs
• Transmeta Crusoe TM5000 series Processor: Implements
Intel IA-32 to proprietary VLIW ISA mapping using dynamic
binary translation and code cache.
• IBM AS/400 Systems:
– Full system developed with a co-design perspective
– Host ISA is built using a high-level instruction set called Machine
Interface (MI)
– Implementing the MI is a set of standard libraries called
Licensed Internal Code (LIC) that dealt with implementation
specific resource management.
– Initially AS/400 systems were built on a proprietary VLIW ISA
and later evolved over to PowerPC ISA.
• Exploring para-virtualization for co-design is a fertile ground
for innovation!
Feb 26, 2019 System Virtual Machines 52
 Summary
• System Virtual Machines:
– Requirements for virtualizability
• Basic concepts
– User Interface and Appearance
– State Management
– Resource Control
– Bare Metal and Hosted Virtual Machines
– Co-designed Virtual Machines
• Case-Studies (To be part of student seminars)
– Native Virtual Machines: Xen, Vmware-Esxi
– Hosted Virtual Machines: Vmware-Workstation, Palazzo,
Linux-KVM

Feb 26, 2019 System Virtual Machines 54