VMware Advance Troubleshooting Workshop - Day 5

Resource Management
Day 5
VMware vSphere:
Install, Configure, Manage

Content
• vMotion
• High Availability
• Fault Tolerance
• Distributed Resource Scheduler

Learner Objectives
By the end of this lesson, you should be able to meet the following
objectives:
• Verify VMware vSphere® vMotion® requirements, including CPU constraints
and guidelines
• Perform a vSphere vMotion migration
• Perform a vSphere Storage vMotion migration
• Perform a shared-nothing vSphere vMotion migration
• Describe the major enhancements to vSphere vMotion in vSphere 6

Migrating Virtual Machines
Migration means moving a virtual machine from one host, datastore, or
vCenter Server system to another host, datastore, or vCenter Server
system.
Types of migrations:
• Cold: Migrate a powered-off virtual machine to a new host or datastore.
• Suspended: Migrate a suspended virtual machine to a new host or datastore
• vSphere vMotion: Migrate a powered-on virtual machine to a new host.
• vSphere Storage vMotion: Migrate a powered-on virtual machine's files to a
new datastore.
• Shared-nothing vSphere vMotion: Migrate a powered-on virtual machine to a
new host and a new datastore.
Concurrent migrations are possible:
• A maximum of 128 concurrent vSphere vMotion accesses to a single VMware
vSphere® VMFS datastore.
• A maximum of 8 concurrent cloning, deployment, or vSphere Storage vMotion
accesses to a single VMFS datastore.

Comparison of Migration Types
Migration
Type
Virtual
Machine
Power State
Change
Host or
Datastore?
Across
vCenter
Servers?
Shared
Storage
Required?
CPU
Compatibility
Cold Off
Host or
datastore
or both
Yes No
Different CPU
families allowed
Suspended Suspended
Host or
datastore
or both
Yes No
Must meet CPU
compatibility
requirements
vSphere
vMotion
On Host Yes Yes
Must meet CPU
compatibility
requirements
vSphere
Storage
vMotion
On Datastore Yes No N/A
Shared-
nothing
vSphere
vMotion
On Both Yes No
Must meet CPU
compatibility
requirements

vSphere vMotion Migration
A vSphere vMotion migration moves a powered-on virtual machine from
one host to another.
vSphere vMotion provides these capabilities:
• Improves overall hardware use
• Continuous virtual machine operation while accommodating scheduled
hardware downtime
• VMware vSphere® Distributed Resource Scheduler™ balancing virtual
machines across hosts

The source host (ESXi01) and the destination host (ESXi02) have access
to the shared datastore that holds the virtual machine’s files.
vSphere vMotion Migration Workflow
Production
Network
vSphere
vMotion
Network
Memory
Bitmap
Memory
VM A
(Network =
Production)
VM A
(Network =
Production)
ESXi02ESXi01

vSphere vMotion Migration Requirements
A virtual machine must meet the following requirements:
• By default, migrating a virtual machine with vSphere vMotion produces an error
if the virtual machine has an active connection to an internal vswitch.
• The virtual machine must not have a connection to a virtual device, such as a
CD/DVD or floppy drive, with a local image mounted.
• The virtual machine must not have CPU affinity configured.
• If the virtual machine’s swap file is not accessible to the destination host,
vSphere vMotion must be able to create a swap file that is accessible to the
destination host before migration can begin.
• If a virtual machine uses an RDM, the RDM and the physical disk to which it
maps must be accessible by the destination host.

Host Requirements for vSphere vMotion Migration
Source and destination hosts must have these characteristics:
• Accessibility to all storage (Fibre Channel, iSCSI, or NAS) that is used by the
virtual machine:
– 128 concurrent vSphere vMotion migrations per VMFS datastore
• At least a 1 Gigabit Ethernet (1GigE) network:
– Four concurrent vSphere vMotion migrations on a 1 Gbps network
– Eight concurrent vSphere vMotion migrations on a 10 Gbps network
• Compatible CPUs:
– CPU feature sets of both the source and destination hosts must be compatible.
– Some features can be hidden by using Enhanced vMotion Compatibility (EVC) or
compatibility masks.

CPU Constraints on vSphere vMotion Migration
CPU Characteristics Exact Match Required Reason
Clock speeds, cache sizes,
hyperthreading, and number of
cores
N/A
Virtualized away by the
VMkernel.
Manufacturer (Intel or AMD)
family and generation (Opteron4,
Intel Westmere)
Applicable
Instruction sets contain many
small differences.
Presence or absence of SSE3,
SSSE3, or SSE4.1 instructions
Applicable
Multimedia instructions usable
directly by applications.
Virtualization hardware assist
For 32-bit VMs: N/A
Virtualized away by the
VMkernel.
For 64-bit VMs on Intel:
Applicable
Intel 64-bit with VMware
implementation uses Intel VT.
Execution-disable
(NX/XD bit)
Applicable but customizable
Guest operating system relies on
NX/XD bit if detected.

Other Cluster Settings: EVC for vSphere vMotion
Enhanced vMotion Compatibility (EVC) is a cluster feature that prevents
vSphere vMotion migrations from failing because of incompatible CPUs.

CPU Baselines for an EVC Cluster
EVC works at the cluster level, using CPU baselines to configure all
processors included in the cluster enabled for EVC.
Cluster Enabled for EVC
CPU Baseline Feature Set
A baseline is a set of CPU
features supported by
every host in the cluster.
CPUIDCPUIDCPUID CPUID
X… X… X… K…

EVC Cluster Requirements
All hosts in the cluster must meet the following requirements:
• Use CPUs from a single vendor, either Intel or AMD:
– Use Intel CPUs with Merom microarchitecture and later.
– Use AMD first-generation Opteron CPUs and later.
• Be enabled for hardware virtualization: AMD-V or Intel VT
• Be enabled for execution-disable technology: AMD No eXecute (NX) or Intel
eXecute Disable (XD)
• Be configured for vSphere vMotion migration
Applications in virtual machines must be CPU ID compatible.

Hiding or Exposing NX/XD
AMD No Execute (NX) and Intel Execute Disable (XD) technologies mark
memory pages as data-only to prevent malicious software exploits and
buffer overflow attacks.
Choose between NX/XD security features
and broadest vSphere vMotion compatibility.
For future CPU features,
edit mask at the bit level.

Identifying CPU Characteristics
To identify CPU characteristics, use the server and CPU specifications or
use the VMware CPU identification utility.

Checking vSphere vMotion Errors
When you select the host and cluster, a validation check is performed to
verify that most vSphere vMotion requirements were met.

vSphere Storage vMotion in Action
vSphere Storage vMotion uses
an I/O mirroring architecture to
copy disk blocks between source
and destination:
1. Initiate storage migration.
2. Use the VMkernel data mover or
VMware vSphere® Storage APIs
- Array Integration to copy data.
3. Start a new virtual machine
process.
4. Mirror I/O calls to file blocks that
are already copied to virtual disk
on the destination datastore.
5. Cut over to the destination virtual
machine process to begin
accessing the virtual disk copy.
Mirror Driver
Data Mover
VM
Process
VM
Process
Read/write
I/O to virtual
disk.
Destination DatastoreSource Datastore
VAAI
Storage Array
VMkernel

vSphere Storage vMotion Guidelines and Limitations
Guidelines:
• Plan the migration and coordinate with administrators.
• Perform migrations during off-peak hours.
• Ensure that the host has access to source datastores and target datastores.
Limitations:
• Virtual machine disks must be in persistent mode or be RDMs.

Shared-nothing vSphere vMotion migration enables a virtual machine to
change its host and datastore simultaneously, even if the two hosts do not
have a shared storage.
• This technique combines vSphere vMotion and vSphere Storage vMotion into a
single operation.
You can migrate virtual machines between hosts, within and across
clusters, data centers, and vCenter Server instances, beyond storage
accessibility boundaries.
Shared-Nothing vSphere vMotion Migration
ESXi ESXi

Shared-Nothing vSphere vMotion Migration Considerations
In vSphere 6, multiple changes can occur simultaneously with shared-nothing
vSphere vMotion migrations.

Migration Between vCenter Server Instances
In vSphere 6, vSphere vMotion can migrate virtual machines between
linked vCenter Server instances. This type of migration requires:
• ESXi hosts and vCenter Server systems must be upgraded to vSphere 6.
• vCenter Server instances must be in Enhanced Linked Mode.
• Hosts must be time-synchronized.
ESXi
vCenter
Server A
Network A
ESXi
vCenter
Server B
vSphere vMotion Network
Enhanced Linked
Mode
Network B

vSphere vMotion TCP/IP Stacks
In vSphere 6, each host has a second TCP/IP stack dedicated to
vSphere vMotion migration.
userworld
VMkernel
User
hostd PING DHCP
vSphere FT Virtual SAN NFS vSphere vMotion
Default TCP/IP
• Separate Memory Heap
• ARP Tables
• Routing Table
• Default Gateway
VMKTCP-API
vSphere vMotion TCP/IP
• Separate Memory Heap
• ARP Tables
• Routing Table
• Default Gateway

Long-Distance vSphere vMotion Migration
Long-distance vSphere vMotion migration is an extension of vSphere
vMotion migration across vCenter Server instances. This migration is
targeted at environments where vCenter Server systems are spread
across large geographic distances and where the latency across sites is
high.
Use cases for long-distance
vSphere vMotion migration:
• Permanent migrations
• Disaster avoidance
• VMware vCenter™ Site
Recovery Manager™ and
disaster avoidance testing
• Multisite load balancing
• Follow-the-Sun scenario support

Networking Requirements for Long-Distance vSphere vMotion
Migration
vSphere vMotion migrations between vCenter Server instances must
connect over layer 3 connections:
• Virtual machine network:
– L2 connection
– Same virtual machine IP address available at destination
• vSphere vMotion network:
– L3 connection
– Secure (dedicated or encrypted)
– 250 Mbps per vSphere vMotion operation

Network Checks for Migrations Between vCenter Server Instances
vCenter Server performs several network compatibility checks to prevent
the following configuration problems:
• MAC address compatibility on the destination host
• vSphere vMotion migration from a distributed switch to a standard switch
• vSphere vMotion migration between distributed switches of different versions
• vSphere vMotion migration to an internal network, for example, a network
without a physical NIC

Review of Learner Objectives
You should be able to meet the following objectives:
• Verify vSphere vMotion requirements, including CPU constraints and
guidelines
• Perform a vSphere vMotion migration
• Perform a vSphere Storage vMotion migration
• Perform a shared-nothing vSphere vMotion migration
• Describe the major enhancements to vSphere vMotion in vSphere 6

Learner Objectives
objectives:
• Describe the options that you can configure to make your VMware vSphere®
environment highly available
• Discuss the response of VMware vSphere® High Availability when a VMware
ESXi™ host, a virtual machine, or an application fails

Protection at Every Level
vSphere enables you to reduce planned downtime,
prevent unplanned downtime, and recover rapidly from
outages.
NIC Teaming,
Storage
Multipathing
vSphere vMotion,
vSphere DRS
vSphere Storage
vMotion, vSphere
Storage DRS
Site Recovery
Manager
Component Server Storage Data Site
vSphere Replication, Third-Party
Backup Solutions, vSphere Data Protection
vSphere HA and vSphere Fault
Tolerance

vCenter Server Availability: Recommendations
Make VMware vCenter Server™ and the components that it relies on
highly available.
vCenter Server relies on these major components:
• vCenter Server database:
– Create a cluster for the database.
• Authentication identity source:
– For example, VMware vCenter™ Single Sign-On™ (a service within the Platform
Services Controller) and Active Directory.
– Set up with multiple redundant servers.
Methods for making vCenter Server available:
• Use vSphere HA to protect the vCenter Server virtual machine.

About vSphere HA
vSphere HA uses multiple ESXi hosts that are configured as a cluster to
provide rapid recovery from outages and cost-effective high availability
for applications running in virtual machines.
Protects against
ESXi host failures
Protects against application
failures
Protects against datastore
accessibility failures
Protects virtual machines
against network isolation

vSphere HA Scenarios: ESXi Host Failure (1)
Virtual Machine A
Virtual Machine B
Virtual Machine C
Virtual Machine F
vCenter Server
ESXi Host ESXi HostESXi Host
Virtual Machine D
Virtual Machine E
Virtual Machine A Virtual Machine B
When a host fails,
vSphere HA restarts
the affected virtual
machines on other
hosts.
= vSphere HA Cluster

vSphere HA Scenarios: Guest Operating System Failure
vCenter Server
ESXi Host ESXi Host
ESXi Host
When a virtual
machine stops
sending heartbeats
or the virtual
machine process
crashes (vmx),
vSphere HA resets
the virtual machine.
Virtual Machine C
VMware Tools VMware Tools
Virtual Machine E
VMware Tools
Virtual Machine F
VMware Tools
Virtual Machine A
VMware Tools
Virtual Machine B
VMware Tools
Virtual Machine D

vSphere HA Scenarios: Application Failure
vCenter Server
ESXi Host ESXi HostESXi Host
Virtual Machine E
Application
When an application
fails, vSphere HA
restarts the affected
virtual machine on the
same host.
Requires installation of
VMware Tools™.
Virtual Machine C
Application
Virtual Machine F
Application
Virtual Machine D
Application
Virtual Machine A
Application
Virtual Machine B
Application

Importance of Redundant Heartbeat Networks
In a vSphere HA cluster, heartbeats have these characteristics:
• Heartbeats are sent between the master host and the slave hosts.
• They are used to determine whether a master host or slave host has failed.
• They are sent over a heartbeat network.
Redundant heartbeat networks ensure reliable failure detection, and
minimize the chance of host isolation scenarios.
Heartbeat network implementation:
• Implemented by using a VMkernel port that is marked for management.

Redundancy Using NIC Teaming
You can use NIC teaming to create a redundant heartbeat network on
ESXi hosts.
Ports or port groups used must be VMkernel ports.
NIC Teaming on an ESXi Host

Redundancy Using Additional Networks
You can also create redundancy by configuring more heartbeat
networks: On each ESXi host, create a second VMkernel port on a
separate virtual switch with its own physical adapter.

• Describe the options that you can configure to make your VMware vSphere®
environment highly available
• Discuss the response of VMware vSphere® High Availability when a VMware
ESXi™ host, a virtual machine, or an application fails

Learner Objectives
objectives:
• Describe the heartbeat mechanisms used by vSphere HA
• Identify and discuss other failure scenarios
• Recognize vSphere HA design considerations

vSphere HA Architecture: Agent Communication
To configure high availability,
ESXi hosts are grouped into
an object called a cluster.
vCenter Server
ESXi Host (Slave)
FDM
ESXi Host (Master)
FDM
ESXi Host (Slave)
FDM
vpxd
hostdhostdhostd
Datastore DatastoreDatastore
= Management Network
vpxa vpxa vpxa

vSphere HA Architecture: Network Heartbeats
The master host sends periodic
heartbeats to the slave hosts so
that the slave hosts know that
the master host is alive.
vCenter Server
Virtual Machine A
Virtual Machine B
Virtual Machine C
Virtual Machine D
Virtual Machine E
Virtual Machine F
Slave Host Slave Host Master Host
VMFS VMFS NFS
Management Network 1

vSphere HA Architecture: Datastore Heartbeats
vCenter Server
Virtual Machine A
Virtual Machine B
Virtual Machine C
Virtual Machine D
Virtual Machine E
Virtual Machine F
Slave Host Master Host Slave Host
VMFS VMFS NAS/NFS
Cluster Edit Settings Window
Datastores are used as a
backup communication
channel to detect virtual
machine and host heartbeats.

vSphere HA Failure Scenarios
• Slave host failure
• Master host failure
• Network failure (Host isolation)
• Datastore access failure:
– Virtual Machine Component Protection (VMCP)
• All Paths Down
• Permanent Device Loss

Failed Slave Host
When a slave host does not respond to the network heartbeat issued by
the master host, the master vSphere HA agent tries to identify the cause.
vCenter Server
Virtual Machine A
Virtual Machine B
Virtual Machine C
Virtual Machine D
Virtual Machine E
Virtual Machine F
Failed Slave
Host
Master Host Slave Host
NAS/NFS
(Lock File)
File Locks File Locks
Primary Heartbeat Network
Alternate Heartbeat Network
VMFS
(Heartbeat Region)

When the master host is placed in maintenance mode or crashes, the
slave hosts detect that the master host is no longer issuing heartbeats.
Failed Master Host
Virtual Machine A
Virtual Machine B
Virtual Machine C
Virtual Machine D
Virtual Machine E
Virtual Machine F
Slave Host
MOID: 98
File Locks
NAS/NFS
(Lock File)
File Locks
vCenter Server
MOID = Managed Object ID
Default Gateway
(Isolation Address)
Slave Host
MOID: 100
VMFS
(Heartbeat Region)
master host
MOID: 99
Failed Master Host
MOID: 99

Isolated Host
If the host does not receive any
heartbeats or observe any
election traffic on the management
and cannot ping its default
gateway, the host is isolated.
Virtual Machine A
Virtual Machine B
Virtual Machine C
Virtual Machine D
Virtual Machine E
Virtual Machine F
ESXi Host ESXi Host
Default Gateway
(Isolation Address)
ESXi Host

Design Considerations
Host isolation events can be minimized through good design:
• Implement redundant heartbeat networks.
• Implement redundant isolation addresses.
If host isolation events do occur, good design enables vSphere HA to
determine whether the isolated host is still alive.
Implement datastores so that they are separated from the management
network by using one or both of the following approaches:
• Fibre Channel over fiber optic
• Physically separating your IP storage network from the management network

Virtual Machine Storage Failures
With an increasing number of virtual
machines and datastores on each
host, storage connectivity issues
have high costs but are infrequent.
Connectivity problems due to:
• Network or switch failure
• Array misconfiguration
• Power outage
Virtual machine availability is
affected:
• Virtual machines on affected hosts are
difficult to manage.
• Applications with attached disks crash.
ESXi ESXi

Virtual Machine Component Protection
Virtual Machine Component Protection (VMCP) protects against storage
failures in a virtual machine.
Only vSphere HA clusters that contain ESXi 6 hosts can be used to
enable VMCP.
Runs on cluster
enabled for
vSphere HA.
VMCP detects
and responds to
failures.
Application
availability and
remediation.ESXi ESXi

• Describe the heartbeat mechanisms used by vSphere HA
• Identify and discuss other failure scenarios
• Recognize vSphere HA design considerations

Learner Objectives
objectives:
• Recognize the requirements for creating and using a vSphere HA cluster
• Configure a vSphere HA cluster

About Clusters
A cluster is used in vSphere to
share physical resources
between a group of ESXi hosts.
vCenter Server manages cluster
resources as a single pool of
resources.
Features such as vSphere HA,
vSphere DRS, and Virtual SAN
can be enabled in a cluster.
Cluster

vSphere HA Prerequisites
You must follow the guidelines to create a vSphere HA cluster:
• All hosts must be licensed for vSphere HA (Essentials Plus or higher).
• A cluster can contain between two and 64 ESXi hosts.
• All hosts must be configured with static IP addresses. If you are using DHCP,
you must ensure that the address for each host persists across reboots.
• All hosts must have at least one management network in common.
• All hosts must have access to the same virtual machine networks and
datastores.
• For Virtual Machine Monitoring to work, VMware Tools™ must be installed in
every virtual machine.
• Only vSphere HA clusters that contain ESXi 6 hosts can be used to enable
VMCP.

Configuring vSphere HA Settings
When you create a vSphere HA cluster or configure a cluster, you must
configure settings that determine how the feature works.

Permanent Device Loss and All Paths Down Overview
vSphere HA uses VMCP to move virtual machines in Permanent Device
Loss and All Paths Down situations to other fully connected hosts.
Permanent Device Loss:
• The datastore appears as unavailable in the Storage view.
• A storage adapter indicates the operational state as loss of communication.
• All paths to the device are marked as dead.
All Paths Down:
• The datastore appears as unavailable in the Storage view.
• A storage adapter indicates the operational state as dead or error.
• All paths to the device are marked as dead.
• The vSphere Client is unable to connect directly to the ESXi host.
• The ESXi host appears as disconnected in vCenter Server.

vSphere HA Settings: Virtual Machine Monitoring (1)
You use Virtual Machine Monitoring settings to control the monitoring of
virtual machines.

vSphere HA Settings: Virtual Machine Monitoring (2)

vSphere HA Settings: Datastore Heartbeating
A heartbeat file is created on the selected datastores and is used in the
event of a management network failure.

vSphere HA Settings: Admission Control
vCenter Server uses admission control to ensure that:
 Sufficient resources are available in a cluster to provide failover
protection
 Virtual machine resource reservations are respected

vSphere HA Settings: Advanced Options
To customize vSphere HA behavior, you set advanced vSphere HA options.
To force a cluster not to use the default isolation address (default gateway):
• das.usedefaultisolationaddress = false
To force a cluster to ping alternate isolation addresses:
• das.isolationaddressX = pintable address
To force a cluster to wait beyond the default 30-second isolation action
window:
• fdm.isolationpolicydelaysec = > 30 sec

Configuring Virtual Machine Overrides
You can override the vSphere HA cluster settings with individual virtual
machine settings.

Before changing the networking settings on an ESXi host (adding port
groups, removing virtual switches, and so on), you must suspend the
Host Monitoring feature and place the host in maintenance mode.
This practice prevents unwanted attempts to fail over virtual machines.
Network Configuration and Maintenance

Cluster Resource Reservation
The Resource Reservation tab reports total cluster CPU, memory,
memory overhead, storage capacity, the capacity reserved by virtual
machines, and how much capacity is still available.

Monitoring Cluster Status
You can monitor the status of a vSphere HA cluster on the Monitor tab.

• Recognize the requirements for creating and using a vSphere HA cluster
• Configure a vSphere HA cluster

Learner Objectives
objectives:
• Describe the benefits of using vSphere Fault Tolerance
• Explain how vSphere Fault Tolerance works
• Describe main vSphere Fault Tolerance features
• Describe how vSphere Fault Tolerance works with vSphere HA and vSphere
DRS
• Enable vSphere Fault Tolerance from vSphere Web Client

vSphere Fault Tolerance
vSphere Fault Tolerance provides instantaneous failover and continuous
availability:
• Zero downtime
• Zero data loss
• No loss of TCP connections
Clustered ESXi Hosts
Primary Virtual Machine Secondary Virtual Machine
Instantaneous
Failover
Fast Checkpointing

vSphere Fault Tolerance Features
vSphere Fault Tolerance protects mission-critical, high-performance
applications regardless of the operating system used.
vSphere Fault Tolerance:
• Supports virtual machines configured with:
– Up to four vCPUs
– Up to 64 GB of memory
• Supports VMware vSphere® vMotion® for primary and secondary virtual
machines
• Creates a secondary copy of all virtual machine files and disks
• Provides fast checkpoint copying to keep primary and secondary CPUs
synchronized
• Supports multiple virtual machine disk formats: Thin provision, thick provision
lazy-zeroed, and thick provision eager-zeroed.
• Can be used with vSphere DRS only when EVC is enabled.

How vSphere Fault Tolerance Works with vSphere HA and vSphere
DRS
vSphere Fault Tolerance works with vSphere HA and vSphere DRS.
vSphere HA:
• Is required for vSphere Fault Tolerance
• Restarts failed virtual machines
• Is vSphere Fault Tolerance aware
vSphere DRS:
• When a virtual machine is powered-on, selects which hosts run the primary
and secondary virtual machines.
• Does not automatically migrate fault-tolerant virtual machines.
ESXi ESXi ESXi
New Secondary
Machine
Primary
Machine
Secondary
Machine

Redundant VMDKs
vSphere Fault Tolerance creates two complete virtual machines.
Each virtual machine has its own .vmx configuration file and .vmdk
files. Each of these virtual machines can be on a different datastore.
Primary Secondary
.vmx file .vmx file
Datastore 1
vmdk file vmdk file vmdk file
Datastore 2
vmdk file vmdk file vmdk file

vSphere Fault Tolerance Checkpoint
vSphere Fault Tolerance supports multiple processors.
Changes on the primary machine are not processed on the secondary
machine. The memory is updated on the secondary.
ESXi
FT
Network
Result XResult X
Input
ESXi

vSphere vMotion: Precopy
During a vSphere vMotion migration, a second virtual machine is created
on the destination host. Then the memory of the source virtual machine
is copied to the destination.
VM A
vSphere vMotion
Network
Virtual Machine
Port Group
Memory
Bitmap
Memory Precopy
VM A
Virtual Machine
End User

vSphere vMotion: Memory Checkpoint
In vSphere vMotion migration, checkpoint data is the last changed pages
of memory.
VM A
Memory
Bitmap
Checkpoint Data
VM A
Virtual Machine
End User
VM A
vSphere vMotion
Network
Virtual Machine
Port Group

vSphere Fault Tolerance Fast Checkpointing
The SMP FT checkpoint interval is dynamic. It adapts to maximize the
workload performance and can range from as small as a few
milliseconds to as large as several hundred milliseconds.
Primary Host Secondary Host
Fault Tolerance
Network
vmx config
Devices
Disks
VM memory
checkpoint

Shared Files
vSphere Fault Tolerance has shared files:
• shared.vmft prevents UUID change.
• .ftgeneration is for the split-brain condition.
Primary Host Secondary Host
shared.vmft
.ftgeneration

shared.vmft File
The shared.vmft file, which is found on a shared datastore, is the
vSphere Fault Tolerance metadata file and contains the primary and
secondary instance UUIDs and the primary and secondary vmx paths.
UUID-1 UUID-2
Ref: UUID-1
UUID-1
VM Guest OS

Enabling vSphere Fault Tolerance on a Virtual Machine
You can turn on vSphere
Fault Tolerance for a
virtual machine through
the VMware vSphere®
Web Client.

• Describe the benefits of using vSphere Fault Tolerance
• Explain how vSphere Fault Tolerance works
• Describe main vSphere Fault Tolerance features
• Describe how vSphere Fault Tolerance works with vSphere HA and vSphere
DRS
• Enable vSphere Fault Tolerance from vSphere Web Client

Resource Management and
Monitoring

Virtual CPU and
Memory Concepts

Learner Objectives
objectives:
• Discuss CPU and memory concepts in a virtualized environment
• Describe what overcommitment of a resource means
• Identify additional technologies that improve memory utilization
• Describe how VMware vSphere® Virtual Symmetric Multiprocessing works and
how hyperthreading is used by the VMkernel

Memory Virtualization Basics
VMware vSphere® has three
layers of memory:
• Guest operating system virtual
memory is presented to
applications by the operating
system.
• Guest operating system
physical memory is presented
to the virtual machine by the
VMkernel.
• Host machine memory that is
managed by the VMkernel
provides a contiguous,
addressable memory space
that is used by the virtual
machine.
Guest OS
Virtual Memory
Guest OS
Physical Memory
ESXi Host
Machine Memory
Application
Operating
System
ESXi Host
Virtual Machine

Virtual Machine Memory Overcommitment
Memory is overcommitted
when the combined working
memory footprint of all virtual
machines exceeds that of the
host memory sizes.
Virtual machines do not
always use their full allocated
memory.
An ESXi host transfers
memory from idle virtual
machines to virtual machines
that need more memory to
improve memory utilization.
Memory overhead is stored in
a swap file (.vswp).
VM 1
.vswp
VM 2
.vswp
VM 3
.vswp
On On On Off
1GB 1GB 1GB 1GB
Host machine memory = 2 GB
Total configured VM memory = 4 GB
vmxd-*.vswp

Memory Reclamation Techniques
Economize use of physical memory pages:
• Transparent page sharing allows pages with identical contents to be stored
only once.
Deallocate memory from one virtual machine for another:
• Ballooning mechanism, active when memory is scarce, forces virtual machines
to use their own paging areas.
Memory compression:
• Attempts to reclaim some memory performance when memory contention is
high.
Host-level SSD swapping:
• Use of a solid-state drive (SSD) on the host for a host cache swap file might
increase performance.
Page virtual machine memory out to disk:
• Use of VMkernel swap space is the last resort due to the poor performance.

Virtual SMP
uni
VM
dual
VM
quad
VM
Virtual
Physical
Thread
Core
Socket
LCPU LCPU LCPU LCPU LCPU LCPU LCPU LCPU
Single-Core Dual-
Socket System
Dual-Core Single-
Socket System
Quad-Core Single-
Socket System
vCPU vCPU vCPU vCPU vCPU vCPU vCPU

Hyperthreading
Hyperthreading enables a core to
execute two threads, or sets of
instructions, at the same time.
Hyperthreading provides more scheduler
throughput.
To enable hyperthreading:
1. Verify that the host system supports
hyperthreading.
2. Enable hyperthreading in the system
BIOS.
3. Ensure that hyperthreading for the ESXi
host is turned on.
4. Hyperthreading is enabled by default.
uni
VM
dual
VM
LCPU LCPU
Dual-Core Single-Socket
System with Hyperthreading
LCPU LCPU

CPU Load Balancing
uni
VM
dual
VM
LCPU LCPU
Hyperthreaded Dual-Core,
Dual-Socket System
LCPU LCPU
LCPU LCPU
LCPU LCPU
uni
VM
uni
VM

• Discuss CPU and memory concepts in a virtualized environment
• Describe what overcommitment of a resource means
• Identify additional technologies that improve memory utilization
• Describe how VMware vSphere® Virtual Symmetric Multiprocessing works and
how hyperthreading is used by the VMkernel

Distributed Resource
Scheduler

Learner Objectives
By the end of this module, you should be able to meet the following
objectives:
• Describe the functions of a VMware vSphere® Distributed Resource
Scheduler™ cluster
• Create a vSphere DRS cluster
• View information about a vSphere DRS cluster
• Remove a host from a vSphere DRS cluster

vSphere DRS Cluster Prerequisites
vSphere DRS works best when the virtual machines meet VMware
vSphere® vMotion® migration requirements.
To use vSphere DRS for load balancing, the hosts in the cluster must be
part of a vSphere vMotion migration network.
• If not, vSphere DRS can still make initial placement recommendations.
To use shared storage, configure all hosts in the cluster:
• VMFS datastores must be accessible by source and destination hosts.
• Ensure that VMFS datastores are sufficiently large to store all virtual disks for
your virtual machine.

vSphere DRS Cluster Settings: Automation Level
Configure the automation level for the initial placement of virtual
machines and dynamic balancing while virtual machines are running.
Automation Level Settings
Migration threshold guides selection
of virtual machines for migration.

Other Cluster Settings: Swap File Location
By default, swap files for a virtual machine are located on a datastore in
the folder containing other virtual machine files.
You may place virtual machine swap files on an alternative datastore.

vSphere DRS Cluster Settings: Virtual Machine Affinity
vSphere DRS affinity rules
specify that selected virtual
machines be placed either
on the same host (affinity) or
on separate hosts (anti-
affinity).
Affinity rules:
• Use for multi-virtual machine
systems where virtual
machines communicate
heavily with one another.
Anti-affinity rules:
• Use for multi-virtual machine
systems where load balance
or high availability is desired.
Options:
• Keep Virtual Machines Together
• Separate Virtual Machines
• Virtual Machines to Hosts

vSphere DRS Cluster Settings: DRS Groups
DRS groups are used in defining
VM-Host affinity rules.
Types of DRS groups:
• A group of virtual machines
– A virtual machine can belong to multiple
virtual machine DRS groups.
• A group of hosts
– A host can belong to multiple host DRS
groups.
A virtual machine can belong to
multiple virtual machine DRS groups.
A host can belong to multiple host
DRS groups.

vSphere DRS Cluster Settings: VM-Host Affinity Rules
A VM-Host affinity rule:
• Specifies an affinity (or anti-
affinity) relationship between a
virtual machine DRS group and
a host DRS group
• Is either a required rule or a
preferential rule
Other options:
Must run on hosts in group,
Must not run on hosts in group,
Should not run on hosts in group

VM-Host Affinity Rule: Preferential
A preferential rule is softly enforced and can be violated if necessary.
Example: Separate virtual machines on different blade systems.
Group A Group B
Blade Chassis A Blade Chassis B
vSphere DRS Cluster
X X

VM-Host Affinity Rule: Required
A required rule is strictly enforced and can never be violated.
Example: Enforce host-based ISV licensing.
Group A
ISV-Licensed
vSphere DRS Cluster
X X

vSphere DRS Cluster Settings: Automation at the Virtual Machine
Level
You can customize the automation level for individual virtual machines in
a cluster to override the automation level set on the entire cluster.

Adding a Host to a Cluster
When adding a host or moving a host into a vSphere DRS cluster, you
can keep the resource pool hierarchy of the existing host.
• If vSphere DRS is not enabled, host resources pools are lost.
For example, add sc-quail04 to Lab Cluster.
When adding the host, choose to create a
resource pool for this host’s virtual
machines and resource pools.

Viewing vSphere DRS Cluster Information
The cluster Summary tab provides
information specific to vSphere DRS.
Clicking the vSphere DRS link on the
Monitor tab displays CPU and memory
utilization per host.

Viewing vSphere DRS Recommendations
The vSphere DRS tab displays information about the vSphere DRS
recommendations made for the cluster, the faults that occurred in
applying such recommendations, and the history of vSphere DRS
actions.
Apply a subset of
recommendations.
Apply all
recommendations.
Refresh recommendations.

Monitoring Cluster Status
Select the cluster object in the inventory to view the state of the cluster
and all its hosts and virtual machines.
You can view the cluster’s Tasks and Events tabs for more information.

Maintenance Mode and Standby Mode
To service a host in a cluster, for example, to install more memory, or
remove a host from a cluster, you must place the host in maintenance
mode:
• Virtual machines on the host should be migrated to another host or shut down.
• You cannot power on virtual machines or migrate virtual machines to a host
entering maintenance mode.
• While in maintenance mode, the host does not allow you to deploy or power on
a virtual machine.
When a host is placed in standby mode, it is powered off:
• This mode is used by VMware vSphere® Distributed Power Management™ to
optimize power usage.

Removing a Host from the vSphere DRS Cluster
Before removing a host
from a vSphere DRS
cluster, consider the
following issues:
• The resource pool
hierarchy remains with the
cluster.
• Because a host must be in
maintenance mode, all
virtual machines running
on that host are powered
off.
• The resources available
for the cluster decrease.

Disabling vSphere DRS and Restoring a Resource Pool Tree
You may turn off vSphere DRS for a cluster.
If you disable vSphere DRS, the resource pools are removed from the
cluster.
When vSphere DRS is disabled, the cluster’s resource pool hierarchy
and affinity rules are not reestablished when DRS is turned back on.
To avoid losing the resource pools, save a snapshot of the resource pool
tree on your local machine.
You can restore a previously saved resource pool tree snapshot when
you enable vSphere DRS.

Improving Virtual Machine Performance Methods
Broad
Fine
Use a vSphere DRS cluster.
Use storage multipathing.
Use NIC teaming.
Modify the resource pool’s CPU and
memory limits and reservations.
Use network
traffic shaping.
Modify the virtual
machine’s CPU and memory
reservations.

Using vSphere HA with vSphere DRS
Reasons why VMware vSphere® High Availability might not be able to
fail over virtual machines:
• vSphere HA admission control is disabled, and there are insufficient resources
in the remaining hosts to power on all of the failed VMs.
• Required VM-Host affinity rules prevent vSphere HA from failing over.
• Sufficient aggregated resources exist, but they are fragmented across hosts. In
such cases, vSphere HA uses vSphere DRS to try to adjust the cluster by
migrating virtual machines to defragment the resources.

• Describe the functions of a VMware vSphere® Distributed Resource
Scheduler™ cluster
• Create a vSphere DRS cluster
• View information about a vSphere DRS cluster
• Remove a host from a vSphere DRS cluster

Key Points
• vSphere DRS clusters provide automated resource management for multiple
VMware ESXi™ hosts.
• vSphere DRS works best if the virtual machines meet vSphere vMotion
migration requirements.
Questions?

VMware Advance Troubleshooting Workshop - Day 5

Recommended

More Related Content

What's hot (20)

Viewers also liked (17)

Similar to VMware Advance Troubleshooting Workshop - Day 5 (20)

Recently uploaded (20)

VMware Advance Troubleshooting Workshop - Day 5