azarpara.vahid@gmail.

com

VMware vSphere:
Optimize and Scale
Lecture Manual
ESXi 7 and vCenter Server 7

VMware® Education Services

VMware, Inc.
www.vmware.com/education

mcse2012.blogfa.com
 [email protected]
VMware vSphere:
Optimize and Scale
Lecture Manual
ESXi 7 and vCenter Server 7
Part Number EDU-EN-VSOS7-LECT (4/2020)
Copyright © 2020 VMware, Inc. All rights reserved. This manual and its accompanying materials
are protected by U.S. and international copyright and intellectual property laws. VMware products
are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware,
Project Photon OS™, vCenter Linked Mode, VMware Certificate Authority, VMware ESX®,
VMware ESXi™, VMware Horizon®, VMware Horizon® 7, VMware Horizon® 7 on VMware
Cloud™ on AWS, VMware Horizon® View™, VMware Host Client™, VMware NSX®, VMware
NSX® Manager™, VMware NSX-T™ Data Center, VMware Photon™, VMware Platform
Services Controller™, VMware PowerCLI™, VMware Tanzu™, VMware Tanzu™ Kubernetes
Grid™, VMware Tanzu™ Kubernetes Grid™ service for vSphere®, VMware Tools™, VMware
vCenter Server®, VMware vCenter® Lifecycle Manager™, VMware vCenter® Server
Appliance™, VMware vCenter® Single Sign-On, VMware vCloud Director®, VMware vCloud
Director® for Service Providers, VMware vCloud®, VMware Verify™, VMware View®, VMware
vRealize® Suite Lifecycle Manager™, VMware vSAN™, VMware vShield Endpoint™, VMware
vShield™, VMware vSphere®, VMware vSphere® API, VMware vSphere® API for Storage
Awareness™, VMware vSphere® Authentication Proxy, VMware vSphere® Auto Deploy™,
VMware vSphere® Client™, VMware vSphere® Command-Line Interface, VMware vSphere®
Distributed Resource Scheduler™, VMware vSphere® Distributed Switch™, VMware vSphere®
ESXi™ Dump Collector, VMware vSphere® ESXi™ Image Builder CLI, VMware vSphere®
ESXi™ Shell, VMware vSphere® Fault Tolerance, VMware vSphere® High Availability, VMware
vSphere® Network I/O Control, VMware vSphere® PowerCLI™, VMware vSphere® Storage
APIs - Array Integration, VMware vSphere® Storage APIs - Data Protection, VMware vSphere®
Storage DRS™, VMware vSphere® Storage I/O Control, VMware vSphere® Storage vMotion®,
VMware vSphere® Trust Authority™, VMware vSphere® Update Manager™, VMware vSphere®
Virtual Volumes™, VMware vSphere® VMFS, VMware vSphere® vMotion®, VMware vSphere®
Web Client, VMware vSphere® with Kubernetes, and VMware Workstation™ are registered
trademarks or trademarks of VMware, Inc. in the United States and/or other jurisdictions. All
other marks and names mentioned herein may be trademarks of their respective companies.
The training material is provided “as is,” and all express or implied conditions, representations,
and warranties, including any implied warranty of merchantability, fitness for a particular purpose
or noninfringement, are disclaimed, even if VMware, Inc., has been advised of the possibility of
such claims. This training material is designed to support an instructor-led training course and is
intended to be used for reference purposes in conjunction with the instructor-led training course.
The training material is not a standalone training tool. Use of the training material for self-study
without class attendance is not recommended.
These materials and the computer programs to which it relates are the property of, and embody
trade secrets and confidential information proprietary to, VMware, Inc., and may not be
reproduced, copied, disclosed, transferred, adapted or modified without the express written
approval of VMware, Inc.

www.vmware.com/education

mcse2012.blogfa.com
CONTENTS

Module 1 Course Introduction 1

1-2 Course Introduction................................................................................................ 2
1-3 Importance ............................................................................................................. 3
1-4 Learner Objectives .................................................................................................. 4
1-5 Course Outline ........................................................................................................ 5
1-6 Typographical Conventions .................................................................................... 6
1-7 References (1)......................................................................................................... 7
1-8 References (2)......................................................................................................... 8
1-9 VMware Online Resources ..................................................................................... 9
1-10 Review of vSphere User Interfaces ......................................................................10
1-11 About Scalability ...................................................................................................11
1-12 About Optimization ..............................................................................................12
1-13 About Security ......................................................................................................13
1-14 Data Center Layout ...............................................................................................14
1-15 VMware Education Overview ...............................................................................16
1-16 VMware Certification Overview ...........................................................................17
1-17 VMware Digital Badge Overview ..........................................................................18
1-18 About VMware Skyline .........................................................................................19

Module 2 Network Scalability 21

2-2 Importance ...........................................................................................................22
2-3 Module Lessons ....................................................................................................23
2-4 Lesson 1: Creating vSphere Distributed Switches ................................................24
2-5 Learner Objectives ................................................................................................25
2-6 About Distributed Switches ..................................................................................26
2-7 Distributed Switch Architecture ...........................................................................27
2-8 Standard and Distributed Switches: Shared Features ..........................................29
2-9 Distributed Switch Features .................................................................................30

Contents i
2-10 Viewing VDS in vSphere Client .............................................................................31
2-11 Load-Based NIC Teaming ......................................................................................32
2-12 Discovery Protocols ..............................................................................................33
2-13 Configuring CDP or LLDP.......................................................................................34
2-14 About Port Binding ...............................................................................................36
2-15 About the Traffic Filtering and Marking Policy .....................................................38
2-16 Creating Network Traffic Rules.............................................................................39
2-17 Network Traffic Rule: Example 1 ..........................................................................40
2-18 Network Traffic Rule: Example 2 ..........................................................................41
2-19 Marking Network Traffic (1) .................................................................................42
2-20 Marking Network Traffic (2) .................................................................................44
2-21 Access to NSX Virtual Distributed Switch (1) ........................................................45
2-22 Access to NSX Virtual Distributed Switch (2) ........................................................46
2-23 Access to NSX Virtual Distributed Switch (3) ........................................................47
2-24 Activity: Distributed Switch Features (1) ..............................................................48
2-25 Activity: Distributed Switch Features (2) ..............................................................49
2-26 Labs .......................................................................................................................50
2-27 Lab 1: Accessing the Lab Environment .................................................................51
2-28 Lab 2: Configuring vSphere Distributed Switches (1) ...........................................52
2-29 Lab 3: Configuring vSphere Distributed Switches (2) ...........................................53
2-30 Review of Learner Objectives ...............................................................................54
2-31 Lesson 2: Managing vSphere Distributed Switches ..............................................55
2-32 Learner Objectives ................................................................................................56
2-33 About Health Check for Distributed Switches ......................................................57
2-34 Health Check Example ..........................................................................................58
2-35 Enabling Health Check ..........................................................................................60
2-36 Monitoring Health Check Results .........................................................................61
2-37 Backing Up and Restoring Configurations ............................................................62
2-38 Exporting Distributed Switch Configurations .......................................................63
2-39 Restoring and Importing Distributed Switch Configurations ...............................65
2-40 Rollback and Recovery of the Management Network .........................................66

ii Contents
2-41 Rollback Details ....................................................................................................67
2-42 Recovery Through the DCUI .................................................................................68
2-43 Lab 4: Managing vSphere Distributed Switches ...................................................69
2-44 Review of Learner Objectives ...............................................................................70
2-45 Lesson 3: Using Network I/O Control in Distributed Switches .............................71
2-46 Learner Objectives ................................................................................................72
2-47 About Network I/O Control (1) .............................................................................73
2-48 About Network I/O Control (2) .............................................................................74
2-49 About the Bandwidth Allocation Model for System Traffic .................................75
2-50 Configuring Bandwidth Allocations for System Traffic .........................................76
2-51 Reserving Bandwidth for System Traffic (1) .........................................................78
2-52 Reserving Bandwidth for System Traffic (2) .........................................................79
2-53 Bandwidth Aggregation for Network Resource Pools ..........................................80
2-54 Creating Network Resource Pools ........................................................................81
2-55 Bandwidth Allocation for VM Traffic ....................................................................82
2-56 Bandwidth Allocation for Individual VMs .............................................................83
2-57 Bandwidth Admission Control in vSphere DRS (1) ...............................................84
2-58 Bandwidth Admission Control in vSphere DRS (2) ...............................................85
2-59 Bandwidth Admission Control in vSphere HA ......................................................86
2-60 Activity: Network I/O Control (1) .........................................................................87
2-61 Activity: Network I/O Control (2) .........................................................................88
2-62 Review of Learner Objectives ...............................................................................89
2-63 Lesson 4: NetFlow and Port Mirroring in Distributed Switches ...........................90
2-64 Learner Objectives ................................................................................................91
2-65 About NetFlow .....................................................................................................92
2-66 About Network Flows ...........................................................................................93
2-67 Network Flow Analysis (1) ....................................................................................95
2-68 Network Flow Analysis (2) ....................................................................................96
2-69 Configuring NetFlow on Distributed Switches .....................................................97
2-70 Enabling NetFlow on a Distributed Port Group ....................................................99
2-71 About Port Mirroring ......................................................................................... 100

Contents iii
2-72 Port Mirroring Session Types ............................................................................ 101
2-73 Port Mirroring Session Properties ..................................................................... 103
2-74 Source and Destination in a Port Mirroring Session ......................................... 104
2-75 Lab 5: Using Port Mirroring ............................................................................... 105
2-76 Review of Learner Objectives ............................................................................ 106
2-77 Key Points .......................................................................................................... 107

Module 3 Storage Scalability 109

3-2 Importance ........................................................................................................ 110
3-3 Module Lessons ................................................................................................. 111
3-4 Lesson 1: VMFS Datastores ............................................................................... 112
3-5 Learner Objectives ............................................................................................. 113
3-6 About VMFS Datastores .................................................................................... 114
3-7 VMFS5 and VMFS6 Datastores .......................................................................... 115
3-8 VMFS6 Datastores ............................................................................................. 116
3-9 VMFS 6 New Features in vSphere 7 .................................................................. 117
3-10 ESXi Support for VMFS5 and VMFS6 ................................................................. 118
3-11 Storage Device Formats..................................................................................... 119
3-12 Storage Devices Supported by VMFS Datastores .............................................. 121
3-13 Limitations of Using 4Kn Drives ......................................................................... 122
3-14 Snapshot Formats for VMFS .............................................................................. 123
3-15 Migrating Your Data from VMFS5 to VMFS6 ..................................................... 124
3-16 Activity: VMFS6 Features (1) ............................................................................. 125
3-17 Activity: VMFS6 Features (2) ............................................................................. 126
3-18 Review of Learner Objectives ............................................................................ 127
3-19 Lesson 2: vSphere Storage APIs - Array Integration .......................................... 128
3-20 Learner Objectives ............................................................................................. 129
3-21 About vSphere Storage APIs - Array Integration ............................................... 130
3-22 Hardware Acceleration APIs for Block Storage ................................................. 131
3-23 Hardware Acceleration for NAS ........................................................................ 133
3-24 Array Thin Provisioning APIs .............................................................................. 135

iv Contents
3-25 About Space Reclamation (1) ............................................................................ 136
3-26 About Space Reclamation (2) ............................................................................ 137
3-27 Space Reclamation with VMFS .......................................................................... 138
3-28 Performing Manual Space Reclamation on a VMFS5 Datastore ....................... 139
3-29 Enabling Automatic Space Reclamation on a VMFS6 Datastore ....................... 141
3-30 Configuring the Automatic Unmap Rate ........................................................... 142
3-31 Space Reclamation Support for Guest Operating Systems (1) .......................... 143
3-32 Space Reclamation Support for Guest Operating Systems (2) .......................... 144
3-33 Space Reclamation Support for Guest Operating Systems (3) .......................... 145
3-34 Activity: vSphere Storage APIs – Array Integration (1) ...................................... 146
3-35 Activity: vSphere Storage APIs – Array Integration (2) ...................................... 147
3-36 Review of Learner Objectives ............................................................................ 148
3-37 Lesson 3: vSphere Storage APIs for Storage Awareness and vSphere APIs
for I/O Filtering .................................................................................................. 149
3-38 Learner Objectives ............................................................................................. 150
3-39 About vSphere API for Storage Awareness ....................................................... 151
3-40 Benefits of Storage Providers ............................................................................ 153
3-41 Comparison of Storage Provider Versions ........................................................ 154
3-42 Storage Provider Categories .............................................................................. 155
3-43 Registering a Storage Provider .......................................................................... 156
3-44 About vSphere APIs for I/O Filtering ................................................................. 157
3-45 Types of I/O Filters ............................................................................................ 158
3-46 How I/O Filters Work (1) ................................................................................... 159
3-47 How I/O Filters Work (2) ................................................................................... 160
3-48 I/O Filter Providers ............................................................................................ 161
3-49 Displaying I/O Filter Provider Information ........................................................ 163
3-50 Activity: I/O Filters (1) ....................................................................................... 164
3-51 Activity: I/O Filters (2) ....................................................................................... 165
3-52 Review of Learner Objectives ............................................................................ 166
3-53 Lesson 4: Storage Policy Based Management Overview ................................... 167
3-54 Learner Objectives ............................................................................................. 168

Contents v
3-55 About Storage Policy Based Management (1) ................................................... 169
3-56 About Storage Policy Based Management (2) ................................................... 170
3-57 About VM Storage Policies ................................................................................ 171
3-58 About Storage Policy Rules ................................................................................ 172
3-59 Storage Policy Example (1) ................................................................................ 173
3-60 Storage Policy Example (2) ................................................................................ 174
3-61 About Storage Policy Components .................................................................... 175
3-62 Storage Policy Component Example ................................................................. 176
3-63 Activity: Storage Policy Rules (1) ....................................................................... 177
3-64 Activity: Storage Policy Rules (2) ....................................................................... 178
3-65 Review of Learner Objectives ............................................................................ 179
3-66 Lesson 5: Using VM Storage Policies ................................................................. 180
3-67 Learner Objectives ............................................................................................. 181
3-68 Creating and Managing Storage Policies ........................................................... 182
3-69 Step 1: Configuring Storage to Be Used for Storage Policies ............................ 183
3-70 Step 2: Creating Storage Policy Components .................................................... 184
3-71 Step 3: Creating VM Storage Policies ................................................................ 185
3-72 Example: Creating a Silver Tier Storage Policy (1) ............................................. 186
3-73 Example: Creating a Silver Tier Storage Policy (2) ............................................. 187
3-74 Example: Creating a Silver Tier Storage Policy (3) ............................................. 188
3-75 Example: Creating a Silver Tier Storage Policy (4) ............................................. 189
3-76 Step 4: Applying the Storage Policy to the VM ................................................. 190
3-77 Step 5: Checking Compliance for the VM Storage Policy .................................. 191
3-78 Lab 6: Using Policy Based Storage ..................................................................... 192
3-79 Review of Learner Objectives ............................................................................ 193
3-80 Lesson 6: Storage Policies for vSAN................................................................... 194
3-81 Learner Objectives ............................................................................................. 195
3-82 About vSAN (1) .................................................................................................. 196
3-83 About vSAN (2) .................................................................................................. 197
3-84 vSAN Disk Architecture ...................................................................................... 198
3-85 Object-Based Storage ........................................................................................ 199

vi Contents
3-86 vSAN Storage Providers ..................................................................................... 200
3-87 vSAN Storage Policies ........................................................................................ 201
3-88 Creating vSAN Storage Policies.......................................................................... 202
3-89 vSAN Capabilities (1) ......................................................................................... 203
3-90 vSAN Capabilities (2) ......................................................................................... 204
3-91 Lab 7: Creating vSAN Storage Policies ............................................................... 205
3-92 Review of Learner Objectives ............................................................................ 206
3-93 Lesson 7: Storage Policies for vSphere Virtual Volumes ................................... 207
3-94 Learner Objectives ............................................................................................. 208
3-95 About vSphere Virtual Volumes ........................................................................ 209
3-96 vSphere Virtual Volumes Architecture .............................................................. 210
3-97 vSphere Virtual Volumes Storage Providers ..................................................... 211
3-98 Protocol Endpoints ............................................................................................ 213
3-99 Storage Containers and vSphere Virtual Volumes Datastores .......................... 214
3-100 Virtual Volumes ................................................................................................. 215
3-101 Mapping VM Files to Virtual Volumes ............................................................... 216
3-102 vSphere Virtual Volumes Replication ................................................................ 217
3-103 Virtual Volume Storage Policies ........................................................................ 219
3-104 Virtual Volume Storage Policy: Examples .......................................................... 220
3-105 Creating Virtual Volumes Storage Policies (1) ................................................... 221
3-106 Creating Virtual Volumes Storage Policies (2) ................................................... 222
3-107 Activity: Displaying Storage Capabilities (1) ...................................................... 223
3-108 Activity: Displaying Storage Capabilities (2) ...................................................... 224
3-109 Review of Learner Objectives ............................................................................ 225
3-110 Lesson 8: Storage I/O Control............................................................................ 226
3-111 Learner Objectives ............................................................................................. 227
3-112 About Storage I/O Control................................................................................. 228
3-113 Enabling Storage I/O Control (1) ....................................................................... 229
3-114 Enabling Storage I/O Control (2) ....................................................................... 230
3-115 Monitoring Latency ........................................................................................... 231
3-116 Automatic Threshold Detection ........................................................................ 232

Contents vii
3-117 Storage Policy Components for Storage I/O Control......................................... 233
3-118 Configuring Storage I/O Control ........................................................................ 234
3-119 Storage I/O Control Requirements .................................................................... 235
3-120 Activity: Storage I/O Control (1) ........................................................................ 236
3-121 Activity: Storage I/O Control (2) ........................................................................ 237
3-122 Review of Learner Objectives ............................................................................ 238
3-123 Lesson 9: Datastore Clusters and vSphere Storage DRS ................................... 239
3-124 Learner Objectives ............................................................................................. 240
3-125 About Datastore Clusters .................................................................................. 241
3-126 Datastore Cluster Requirements ....................................................................... 242
3-127 Host Cluster to Datastore Cluster Relationships ............................................... 243
3-128 vSphere Storage DRS Overview ......................................................................... 244
3-129 Initial Disk Placement ........................................................................................ 245
3-130 vSphere Storage DRS Affinity Rules ................................................................... 246
3-131 Storage Migration Recommendations .............................................................. 247
3-132 Datastore Correlation Detector......................................................................... 248
3-133 Configuring vSphere Storage DRS Runtime Settings ......................................... 249
3-134 vSphere Storage DRS Maintenance Mode ........................................................ 251
3-135 Backups and vSphere Storage DRS .................................................................... 252
3-136 Activity: vSphere Storage DRS (1) ...................................................................... 253
3-137 Activity: vSphere Storage DRS (2) ...................................................................... 254
3-138 Review of Learner Objectives ............................................................................ 255
3-139 Key Points .......................................................................................................... 256

Module 4 Host and Management Scalability 257

4-2 Importance ........................................................................................................ 258
4-3 Module Lessons ................................................................................................. 259
4-4 Lesson 1: VMware Certificate Management ..................................................... 260
4-5 Learner Objectives ............................................................................................. 261
4-6 Certificate Use Case ........................................................................................... 262
4-7 About Certificate Authorities ............................................................................ 263

viii Contents
4-8 About VMware CA ............................................................................................. 265
4-9 About the VMware Endpoint Certificate Store ................................................. 266
4-10 vSphere Certificate Types .................................................................................. 267
4-11 Chain of Trust (1) ............................................................................................... 268
4-12 Chain of Trust (2) ............................................................................................... 269
4-13 Chain of Trust (3) ............................................................................................... 270
4-14 Activity: ESXi Certificates (1) ............................................................................. 271
4-15 Activity: ESXi Certificates (2) ............................................................................. 272
4-16 VMware CA Modes ............................................................................................ 273
4-17 VMware CA: Root CA Mode .............................................................................. 274
4-18 Replacing the VMware CA Certificate in Root CA Mode ................................... 275
4-19 VMware CA: Subordinate CA Mode .................................................................. 276
4-20 Replacing the VMware CA Certificate with a Subordinate CA Certificate......... 277
4-21 VMware CA: Hybrid Mode................................................................................. 278
4-22 Hybrid Mode: Replacing a vCenter Server Machine Server Certificate with
a Custom Certificate .......................................................................................... 279
4-23 Certificate Replacement Options ...................................................................... 280
4-24 Certificate Manager ........................................................................................... 281
4-25 Replacing All VMware CA Certificates with Custom Certificates ...................... 282
4-26 ESXi Certificate Replacement Options ............................................................... 283
4-27 Renewing or Refreshing ESXi Certificates ......................................................... 284
4-28 VMware CA Availability ..................................................................................... 285
4-29 Lab 8: Working with Certificates ....................................................................... 286
4-30 Review of Learner Objectives ............................................................................ 287
4-31 Lesson 2: vCenter Server Identity Federation ................................................... 288
4-32 Learner Objectives ............................................................................................. 289
4-33 About vCenter Server IDP Federation for ADFS ................................................ 290
4-34 vCenter Server IDP Federation Use Cases ......................................................... 291
4-35 vCenter Server IDP Federation Components .................................................... 292
4-36 Prerequisites for vCenter Server IDP Federation .............................................. 293
4-37 User Login with the vCenter Server Built-In Identity Provider .......................... 294

Contents ix
4-38 Configuring vCenter Server IDP Federation in vSphere (1) ............................... 295
4-39 Configuring vCenter Server IDP Federation in vSphere (2) ............................... 296
4-40 Configuring vCenter Server IDP Federation in vSphere (3) ............................... 297
4-41 Configuring vCenter Server IDP Federation in vSphere (4) ............................... 298
4-42 Login Workflow of the vCenter Server IDP Federation for ADFS ...................... 299
4-43 Enabling ADFS on an Existing Enhanced Linked Mode Configuration ............... 300
4-44 Lab 9: Configuring Identity Federation to Use Microsoft ADFS ........................ 301
4-45 Review of Learner Objectives ............................................................................ 302
4-46 Lesson 3: vSphere Trust Authority .................................................................... 303
4-47 Learner Objectives ............................................................................................. 304
4-48 Overview of VM Encryption .............................................................................. 305
4-49 UEFI Secure Boot for ESXi Hosts ........................................................................ 306
4-50 UEFI Secure Boot Sequence for ESXi Hosts ....................................................... 307
4-51 TPM 2.0 on ESXi Hosts ....................................................................................... 308
4-52 About Remote Attestation ................................................................................ 309
4-53 Remote Attestation Process .............................................................................. 310
4-54 Requirements for Using TPM 2.0 ...................................................................... 311
4-55 About vSphere Trust Authority ......................................................................... 312
4-56 About vSphere Trust Authority Administrator .................................................. 313
4-57 Use Case for vSphere Trust Authority ............................................................... 314
4-58 vSphere Trust Authority Components (1) ......................................................... 315
4-59 vSphere Trust Authority Components (2) ......................................................... 316
4-60 vSphere Trust Authority Workflow ................................................................... 317
4-61 ESXi Components............................................................................................... 318
4-62 ESXi Components............................................................................................... 319
4-63 Attestation Service ............................................................................................ 320
4-64 Attestation Service Trust ................................................................................... 321
4-65 Key Provider Service .......................................................................................... 322
4-66 Trusted Infrastructure Agent ............................................................................. 323
4-67 vCenter Server Topologies (1) ........................................................................... 324
4-68 vCenter Server Topologies (2) ........................................................................... 325

x Contents
4-69 About Enabling vSphere Trust Authority ........................................................... 326
4-70 About VM Encryption with vSphere Trust Authority ........................................ 327
4-71 VM Encryption with a Trusted Key Provider ..................................................... 328
4-72 Identifying VM Encryption Key Providers .......................................................... 329
4-73 Migrating a VM Encrypted with a Trusted Key Provider (1).............................. 330
4-74 Migrating a VM Encrypted with a Trusted Key Provider (2).............................. 331
4-75 Labs .................................................................................................................... 332
4-76 Lab 10: Assigning a vSphere Trust Authority Administrator ............................. 333
4-77 Lab 11: Enabling and Configuring vSphere Trust Authority .............................. 334
4-78 Lab 12: Encrypting a VM with a Trusted Key Provider ...................................... 335
4-79 Review of Learner Objectives ............................................................................ 336
4-80 Lesson 4: Host Profiles....................................................................................... 337
4-81 Learner Objectives ............................................................................................. 338
4-82 About Host Profiles............................................................................................ 339
4-83 Implementing Host Profiles ............................................................................... 341
4-84 Monitoring for Compliance ............................................................................... 343
4-85 Applying Host Profiles ....................................................................................... 344
4-86 Host Customization ........................................................................................... 345
4-87 Managing Host Customizations (1) ................................................................... 347
4-88 Managing Host Customizations (2) ................................................................... 348
4-89 Standardization Across Multiple vCenter Server Instances .............................. 349
4-90 Lab 13: Using Host Profiles ................................................................................ 350
4-91 Review of Learner Objectives ............................................................................ 351
4-92 Lesson 5: Content Libraries ............................................................................... 352
4-93 Learner Objectives ............................................................................................. 353
4-94 About Content Libraries .................................................................................... 354
4-95 Content Library Benefits.................................................................................... 355
4-96 Content Library Types ....................................................................................... 356
4-97 Storing Content in Content Libraries ................................................................. 358
4-98 Creating Content Libraries................................................................................. 359
4-99 Advanced Configuration .................................................................................... 360

Contents xi
4-100 Populating Content Libraries ............................................................................. 361
4-101 Using the Content Library to Deploy VMs (1) ................................................... 362
4-102 Using the Content Library to Deploy VMs (2) ................................................... 363
4-103 Updating Content Library Items ........................................................................ 364
4-104 Publishing Content Libraries for External Use ................................................... 365
4-105 Content Library Subscriptions ........................................................................... 366
4-106 Published and Subscribed Content Libraries ..................................................... 367
4-107 Synchronizing and Versioning Content Libraries ............................................... 369
4-108 Content Library Maximums ............................................................................... 371
4-109 Lab 14: Creating a Content Library .................................................................... 372
4-110 Review of Learner Objectives ............................................................................ 373
4-111 Lesson 6: Resource Pools .................................................................................. 374
4-112 Learner Objectives ............................................................................................. 375
4-113 About Resource Pools ....................................................................................... 376
4-114 Resource Pool Attributes ................................................................................... 377
4-115 Reasons to Use Resource Pools ......................................................................... 378
4-116 Resource Pool Example ..................................................................................... 379
4-117 Resource Pool Example: CPU Shares ................................................................. 380
4-118 Resource Pool Example: CPU Contention ......................................................... 381
4-119 About Expandable Reservations........................................................................ 382
4-120 Example of Expandable Reservation (1) ............................................................ 383
4-121 Example of Expandable Reservation (2) ............................................................ 384
4-122 Resource Pool Summary Tab ............................................................................. 385
4-123 Resource Reservation Details ............................................................................ 386
4-124 Resource Pool Shares ........................................................................................ 387
4-125 Diluted Shares ................................................................................................... 389
4-126 About Scalable Shares (1) .................................................................................. 390
4-127 About Scalable Shares (2) .................................................................................. 391
4-128 Enabling Scalable Shares ................................................................................... 392
4-129 Lab 15: Managing Resource Pools ..................................................................... 393
4-130 Review of Learner Objectives ............................................................................ 394

xii Contents
4-131 Key Points .......................................................................................................... 395

Module 5 CPU Optimization 397

5-2 Importance ........................................................................................................ 398
5-3 Module Lessons ................................................................................................. 399
5-4 Lesson 1: CPU Virtualization Concepts .............................................................. 400
5-5 Learner Objectives ............................................................................................. 401
5-6 About the CPU Scheduler .................................................................................. 402
5-7 About VM Worlds .............................................................................................. 403
5-8 CPU Scheduler Features .................................................................................... 404
5-9 CPU Scheduler Support for SMP VMs ............................................................... 405
5-10 Relaxed Co-Scheduling ...................................................................................... 406
5-11 CPU Performance Factors.................................................................................. 407
5-12 Warning Sign: Ready Time ................................................................................. 408
5-13 Transitioning Between World States ................................................................. 409
5-14 Activity: CPU Scheduler (1) ................................................................................ 410
5-15 Activity: CPU Scheduler (2) ................................................................................ 411
5-16 Security Boundaries ........................................................................................... 412
5-17 Review of Learner Objectives ............................................................................ 413
5-18 Lesson 2: NUMA Architecture in vSphere ......................................................... 414
5-19 Learner Objectives ............................................................................................. 415
5-20 How the ESXi NUMA Scheduler Works ............................................................. 416
5-21 NUMA-Aware ESXi ............................................................................................. 417
5-22 Terminology: vCPUs, pCPUs, Cores, and More ................................................. 418
5-23 About the NUMA Client ..................................................................................... 419
5-24 Wide VM NUMA Support .................................................................................. 420
5-25 Virtual NUMA Topology .................................................................................... 421
5-26 Optimizing the Number of Cores per Socket for VMs (1) ................................. 422
5-27 Optimizing the Number of Cores per Socket for VMs (2) ................................. 423
5-28 Optimizing the Number of Cores per Socket for VMs (3) ................................. 424
5-29 Activity: VM Cores Per Socket (1) ...................................................................... 425

Contents xiii
5-30 Activity: VM Cores Per Socket (2) ...................................................................... 426
5-31 Memory Considerations with vNUMA (1) ......................................................... 427
5-32 Memory Considerations with vNUMA (2) ......................................................... 428
5-33 Memory Considerations with vNUMA (3) ......................................................... 429
5-34 Memory Considerations with vNUMA (4) ......................................................... 430
5-35 vNUMA Considerations ..................................................................................... 431
5-36 Review of Learner Objectives ............................................................................ 432
5-37 Lesson 3: Using esxtop ...................................................................................... 433
5-38 Learner Objectives ............................................................................................. 434
5-39 About esxtop (1) ................................................................................................ 435
5-40 About esxtop (2) ................................................................................................ 436
5-41 Getting Help in esxtop ....................................................................................... 437
5-42 Customizing Fields in esxtop ............................................................................. 438
5-43 Review of Learner Objectives ............................................................................ 439
5-44 Lesson 4: Monitoring CPU Activity .................................................................... 440
5-45 Learner Objectives ............................................................................................. 441
5-46 CPU Key Performance Indicators for ESXi Hosts ............................................... 442
5-47 CPU Key Performance Indicators for VMs ......................................................... 443
5-48 Using esxtop to Monitor ESXi Host CPU ............................................................ 445
5-49 Using esxtop to View CPU Metrics per VM ....................................................... 446
5-50 Using esxtop to View Single CPU Statistics........................................................ 448
5-51 Using esxtop to Monitor VM vCPU.................................................................... 449
5-52 Important Metrics to Monitor ........................................................................... 450
5-53 Example: Identifying CPU Overcommitment..................................................... 451
5-54 Host CPU Saturation .......................................................................................... 452
5-55 Resolving Host CPU Saturation .......................................................................... 453
5-56 Activity: CPU Metrics (1).................................................................................... 454
5-57 Activity: CPU Metrics (2).................................................................................... 455
5-58 Guest CPU Saturation ........................................................................................ 456
5-59 Using One vCPU in an SMP VM ......................................................................... 457
5-60 Low Guest CPU Use ........................................................................................... 459

xiv Contents
5-61 Introduction to Lab 8: Monitoring CPU Performance ....................................... 460
5-62 Lab 16: Monitoring CPU Performance............................................................... 461
5-63 Review of Lab: Monitoring CPU Performance ................................................... 462
5-64 Review of Learner Objectives ............................................................................ 463
5-65 Key Points .......................................................................................................... 464

Module 6 Memory Optimization 465

6-2 Importance ........................................................................................................ 466
6-3 Module Lessons ................................................................................................. 467
6-4 Lesson 1: Memory Virtualization Concepts ....................................................... 468
6-5 Learner Objectives ............................................................................................. 469
6-6 Virtual Memory Overview ................................................................................. 470
6-7 Application and Guest OS Memory Management ............................................ 471
6-8 VM Memory Management ................................................................................ 472
6-9 Memory Reclamation ........................................................................................ 473
6-10 Reason for Reclaiming Memory ........................................................................ 474
6-11 Guest OS Memory Terminology ........................................................................ 475
6-12 VM Memory Reclamation Techniques .............................................................. 477
6-13 Transparent Page Sharing (1) ............................................................................ 479
6-14 Transparent Page Sharing (2) ............................................................................ 481
6-15 Impact of Hardware MMU on Transparent Page Sharing ................................. 482
6-16 Impact of NUMA Architecture on Transparent Page Sharing ........................... 484
6-17 Transparent Page Sharing and Intra-VM Memory Sharing ............................... 485
6-18 Using Salting with Transparent Page Sharing (1) .............................................. 486
6-19 Using Salting with Transparent Page Sharing (2) .............................................. 487
6-20 Using Salting with Transparent Page Sharing (3) .............................................. 488
6-21 Using Salting with Transparent Page Sharing (4) .............................................. 489
6-22 Using Salting with Transparent Page Sharing (5) .............................................. 490
6-23 Memory Ballooning in the Guest OS ................................................................. 491
6-24 Reclaiming Memory with Ballooning ................................................................ 493
6-25 Memory Compression ....................................................................................... 494

Contents xv
6-26 Reclaiming Memory with Host Swapping ......................................................... 496
6-27 Activity: Identifying Memory Reclamation Features (1) ................................... 497
6-28 Activity: Identifying Memory Reclamation Features (2) ................................... 498
6-29 Sliding Scale Mem.MemMinFreePct ................................................................. 499
6-30 Criteria for Reclaiming Host Memory ................................................................ 500
6-31 Transitioning from One Memory State to Another (1) ..................................... 502
6-32 Transitioning from One Memory State to Another (2) ..................................... 504
6-33 Review of Learner Objectives ............................................................................ 506
6-34 Lesson 2: Monitoring Memory Activity ............................................................. 507
6-35 Learner Objectives ............................................................................................. 508
6-36 Memory Usage Metrics in the Guest OS ........................................................... 509
6-37 About Consumed Host Memory and Active Guest Memory............................. 510
6-38 Using esxtop to Monitor Memory Usage .......................................................... 511
6-39 Host Ballooning Activity in esxtop (1) ............................................................... 513
6-40 Host Ballooning Activity in esxtop (2) ............................................................... 515
6-41 Host Memory Compression Activity in esxtop .................................................. 516
6-42 Monitoring Host Cache Swapping ..................................................................... 518
6-43 Host Swapping Activity in esxtop: Memory Screen........................................... 519
6-44 Host Swapping Activity in esxtop: CPU Screen .................................................. 520
6-45 Causes of Active Host-Level Swapping .............................................................. 521
6-46 Activity: Identifying esxtop Fields (1) ................................................................ 523
6-47 Activity: Identifying esxtop Fields (2) ................................................................ 524
6-48 Resolving Host-Level Swapping ......................................................................... 525
6-49 Reducing Memory Overcommitment................................................................ 526
6-50 Enabling the Balloon Driver in VMs ................................................................... 528
6-51 Reducing VM Memory Reservation................................................................... 529
6-52 Dedicating Memory to Critical VMs .................................................................. 530
6-53 Introduction to Lab: Monitoring Memory Performance ................................... 531
6-54 Lab 17: Monitoring Memory Performance ....................................................... 532
6-55 Review of Lab: Monitoring Memory Performance............................................ 533
6-56 Review of Learner Objectives ............................................................................ 534

xvi Contents
6-57 Key Points .......................................................................................................... 535

Module 7 Storage Optimization 537

7-2 Importance ........................................................................................................ 538
7-3 Module Lessons ................................................................................................. 539
7-4 Lesson 1: Storage Virtualization Concepts ........................................................ 540
7-5 Learner Objectives ............................................................................................. 541
7-6 Storage Performance Overview ........................................................................ 542
7-7 Storage Protocol Performance .......................................................................... 543
7-8 About SAN Configuration .................................................................................. 544
7-9 About Storage Queues ...................................................................................... 546
7-10 Network Storage: iSCSI and NFS ........................................................................ 547
7-11 Review of Learner Objectives ............................................................................ 548
7-12 Lesson 2: Enhancements to NVMe Storage Systems ........................................ 549
7-13 Learner Objectives ............................................................................................. 550
7-14 Enhanced Support for NVMe ............................................................................ 551
7-15 Supported NVMe Topologies ............................................................................ 552
7-16 FC-NVMe and NVMe over RDMA (RoCE v2)...................................................... 553
7-17 Support for FC-NVMe ........................................................................................ 554
7-18 Requirements for FC-NVMe .............................................................................. 555
7-19 Support for NVMe over RDMA (RoCE v2) ......................................................... 556
7-20 Requirements for NVMe over RDMA (RoCE v2)................................................ 557
7-21 Configuring ESXi for NVMe over RDMA (RoCE v2) (1) ....................................... 558
7-22 Configuring ESXi for NVMe over RDMA (RoCE v2) (2) ....................................... 559
7-23 Configuring ESXi for NVMe over RDMA (RoCE v2) (3) ....................................... 560
7-24 Configuring ESXi for NVMe over RDMA (RoCE v2) (4) ....................................... 561
7-25 Pluggable Storage Architecture ......................................................................... 562
7-26 High-Performance Plug-In for NVMe ................................................................ 563
7-27 Support for High-Performance Plug-In Multipathing ........................................ 564
7-28 High-Performance Plug-In Best Practices .......................................................... 565
7-29 Review of Learner Objectives ............................................................................ 566

Contents xvii
7-30 Lesson 3: Support for iSER ................................................................................. 567
7-31 Learner Objectives ............................................................................................. 568
7-32 Support for iSER-Enabled vSphere Virtual Volumes Arrays .............................. 569
7-33 Configuring iSER ................................................................................................ 570
7-34 Requirements for iSER ....................................................................................... 571
7-35 Running iSER ...................................................................................................... 572
7-36 Review of Learner Objectives ............................................................................ 573
7-37 Lesson 4: Monitoring Storage Activity .............................................................. 574
7-38 Learner Objectives ............................................................................................. 575
7-39 Where Storage Problems Can Occur ................................................................. 576
7-40 Storage Key Performance Indicators ................................................................. 577
7-41 Monitoring Disk Throughput by Storage Adapter (1)........................................ 578
7-42 Monitoring Disk Throughput by Storage Adapter (2)........................................ 580
7-43 Correlating Storage Devices to Datastores (1) .................................................. 581
7-44 Correlating Storage Devices to Datastores (2) .................................................. 582
7-45 Monitoring Path Statistics for a Specific Storage Device (1) ............................. 583
7-46 Monitoring Path Statistics for a Specific Storage Device (2) ............................. 584
7-47 Monitoring Individual VMDK Statistics for a VM (1) ......................................... 585
7-48 Monitoring Individual VMDK Statistics for a VM (2) ......................................... 586
7-49 Disk Latency Metrics .......................................................................................... 587
7-50 Monitoring Disk Latency with esxtop ................................................................ 589
7-51 Monitoring Commands and Command Queuing .............................................. 590
7-52 Example: Disk Latency and Queuing ................................................................. 591
7-53 Example: Bad Disk Throughput ......................................................................... 593
7-54 Activity: Hardware Disk Latency (1) .................................................................. 594
7-55 Activity: Hardware Disk Latency (2) .................................................................. 595
7-56 Overloaded Storage and Command Aborts ...................................................... 596
7-57 Unexpected Increase in I/O Latency on Shared Storage ................................... 597
7-58 Storage Performance Best Practices ................................................................. 599
7-59 Introduction to Lab: Monitoring Storage Performance .................................... 600
7-60 Lab 18: Monitoring Storage Performance ......................................................... 601

xviii Contents
7-61 Review of Lab: Monitoring Storage Performance ............................................. 602
7-62 Review of Learner Objectives ............................................................................ 603
7-63 Key Points .......................................................................................................... 604

Module 8 Network Optimization 605

8-2 Importance ........................................................................................................ 606
8-3 Module Lessons ................................................................................................. 607
8-4 Lesson 1: Network Virtualization Concepts....................................................... 608
8-5 Learner Objectives ............................................................................................. 609
8-6 Network I/O Virtualization Overhead ............................................................... 610
8-7 VMXNET Network Adapters .............................................................................. 612
8-8 Virtual Network Adapters.................................................................................. 614
8-9 About VMXNET3 Optimization .......................................................................... 616
8-10 Configuring a Single CPU Thread per vNIC ........................................................ 618
8-11 Configuring Multiple CPU Threads per vNIC ..................................................... 619
8-12 About TCP Segmentation Offload ..................................................................... 621
8-13 About Jumbo Frames......................................................................................... 623
8-14 About SplitRx Mode........................................................................................... 624
8-15 About SplitTx Mode ........................................................................................... 626
8-16 Activity: vSphere Networking Features (1) ....................................................... 627
8-17 Activity: vSphere Networking Features (2) ....................................................... 628
8-18 Review of Learner Objectives ............................................................................ 629
8-19 Lesson 2: Monitoring Network I/O Activity ....................................................... 630
8-20 Learner Objectives ............................................................................................. 631
8-21 Monitoring for Performance in the Network I/O Stack .................................... 632
8-22 Network Capacity Metrics ................................................................................. 633
8-23 About esxtop Network Statistics ....................................................................... 634
8-24 About esxtop Network Output .......................................................................... 635
8-25 Dropped Network Packets................................................................................. 637
8-26 Dropped Packets in vNICs.................................................................................. 639
8-27 Dropped Received Packets in Physical NICs ...................................................... 641

Contents xix
8-28 Dropped Transmit Packets in Physical NICs ...................................................... 642
8-29 Unexpected Increases in Data Transfer Rate .................................................... 643
8-30 Networking Best Practices ................................................................................. 644
8-31 Introduction to Lab: Monitoring Network Performance (1) ............................. 646
8-32 Introduction to Lab: Monitoring Network Performance (2) ............................. 648
8-33 Lab 19: Monitoring Network Performance ....................................................... 649
8-34 Review of Lab: Monitoring Network Performance ........................................... 650
8-35 Review of Learner Objectives ............................................................................ 651
8-36 Key Points .......................................................................................................... 652

Module 9 vCenter Server Performance Optimization 653

9-2 Importance ........................................................................................................ 654
9-3 Lesson 1: vCenter Server Performance Optimization ....................................... 655
9-4 Learner Objectives ............................................................................................. 656
9-5 vCenter Server Performance Considerations .................................................... 657
9-6 vCenter Server CPU Performance ..................................................................... 658
9-7 vCenter Server Memory Performance .............................................................. 659
9-8 Monitoring CPU and Memory Use with the VAMI ............................................ 660
9-9 Using vimtop to Monitor vCenter Server Appliance Resources ........................ 661
9-10 vimtop Commands and Columns ...................................................................... 662
9-11 Monitoring CPU Use with vimtop...................................................................... 663
9-12 Activity: CPU and Memory Use (1) .................................................................... 664
9-13 Activity: CPU and Memory Use (2) .................................................................... 665
9-14 Monitoring Memory Use with vimtop .............................................................. 666
9-15 Viewing Service Memory Allocation with cloudvm-ram-size ............................ 667
9-16 Viewing Service Heap Size ................................................................................. 668
9-17 Changing Service Heap Size ............................................................................... 669
9-18 vCenter Server Network Performance .............................................................. 670
9-19 Monitoring Network Activity with the VAMI .................................................... 671
9-20 Monitoring Network Activity with vimtop ........................................................ 672
9-21 vCenter Server Database Performance (1) ........................................................ 673

xx Contents
9-22 vCenter Server Database Performance (2) ........................................................ 674
9-23 Effects of Changing Statistics Level on Database Traffic ................................... 675
9-24 vCenter Server Appliance Disk Use (1) .............................................................. 677
9-25 vCenter Server Appliance Disk Use (2) .............................................................. 678
9-26 vCenter Server Appliance Disk Use (3) .............................................................. 679
9-27 Monitoring Disk Use with the VAMI .................................................................. 680
9-28 Monitoring Disk Usage with the df Command .................................................. 681
9-29 Monitoring Disk Activity with vimtop ................................................................ 682
9-30 Monitoring Database Activity with the VAMI ................................................... 683
9-31 Activity: Network Performance (1).................................................................... 684
9-32 Activity: Network Performance (2).................................................................... 685
9-33 Activity: vCenter Server Tools (1) ...................................................................... 686
9-34 Activity: vCenter Server Tools (2) ...................................................................... 687
9-35 Review of Learner Objectives ............................................................................ 688
9-36 Key Points .......................................................................................................... 689

Module 10 Appendix: vSphere Auto Deploy 691

10-2 Importance ........................................................................................................ 692
10-3 Appendix Lessons .............................................................................................. 693
10-4 Lesson 1: vSphere ESXi Image Builder ............................................................... 694
10-5 Learner Objectives ............................................................................................. 695
10-6 ESXi Images ........................................................................................................ 696
10-7 vSphere Installation Bundles ............................................................................. 697
10-8 ESXi Image Deployment .................................................................................... 698
10-9 vSphere ESXi Image Builder Overview .............................................................. 699
10-10 vSphere ESXi Image Builder Architecture .......................................................... 700
10-11 Building ESXi Images .......................................................................................... 701
10-12 Step 1: Starting the Image Builder Service ........................................................ 702
10-13 Step 2: Connecting to a Software Depot ........................................................... 703
10-14 Step 3: Creating an Image Profile ...................................................................... 704
10-15 Step 4: Generating the New ESXi Image............................................................ 705

Contents xxi
10-16 Review of Learner Objectives ............................................................................ 706
10-17 Lesson 2: Overview of vSphere Auto Deploy .................................................... 707
10-18 Learner Objectives ............................................................................................. 708
10-19 About vSphere Auto Deploy .............................................................................. 709
10-20 vSphere Auto Deploy Modes ............................................................................. 711
10-21 vSphere Auto Deploy Stateless Mode ............................................................... 712
10-22 vSphere Auto Deploy Mode: Stateless Caching ................................................ 713
10-23 vSphere Auto Deploy Mode: Stateful Installation ............................................. 714
10-24 Locations for Configuration and State Information .......................................... 715
10-25 vSphere Auto Deploy Architecture .................................................................... 717
10-26 Rules Engine Basics ............................................................................................ 719
10-27 PXE Boot Infrastructure Setup ........................................................................... 721
10-28 Initial Boot of an Autodeployed ESXi Host (1) ................................................... 722
10-29 Initial Boot of an Autodeployed ESXi Host (2) ................................................... 723
10-30 Initial Boot of an Autodeployed ESXi Host (3) ................................................... 724
10-31 Initial Boot of an Autodeployed ESXi Host (4) ................................................... 725
10-32 Initial Boot of an Autodeployed ESXi Host (5) ................................................... 726
10-33 Subsequent Boot of a Stateless ESXi Host (1) ................................................... 727
10-34 Subsequent Boot of a Stateless ESXi Host (2) ................................................... 728
10-35 Subsequent Boot of a Stateless ESXi Host (3) ................................................... 729
10-36 Subsequent Boot of a Stateless ESXi Host (4) ................................................... 730
10-37 Activity: vSphere Auto Deploy Modes (1) ......................................................... 731
10-38 Activity: vSphere Auto Deploy Modes (2) ......................................................... 732
10-39 Running Custom Scripts on Stateless Hosts ...................................................... 733
10-40 Booting Many Stateless Hosts ........................................................................... 734
10-41 Stateless Caching Host Profile Configuration .................................................... 735
10-42 Subsequent Boot of a Stateless Caching Host ................................................... 736
10-43 Stateful Installation Host Profile Configuration ................................................ 738
10-44 Subsequent Boot of a Stateful Installation Host ............................................... 739
10-45 Stateful Installation Using vSphere Lifecycle Manager ..................................... 740
10-46 Review of Learner Objectives ............................................................................ 741

xxii Contents
10-47 Lesson 3: Configuring vSphere Auto Deploy ..................................................... 742
10-48 Learner Objectives ............................................................................................. 743
10-49 Configuring a vSphere Auto Deploy Environment............................................. 744
10-50 Step 1: Preparing the DHCP Server ................................................................... 745
10-51 Step 2: Starting the vSphere Auto Deploy Service ............................................ 746
10-52 Step 3: Preparing the TFTP Server ..................................................................... 747
10-53 Step 4: Creating Deployment Rules ................................................................... 748
10-54 Step 5: Activating Deployment Rules ................................................................ 749
10-55 Managing the vSphere Auto Deploy Environment............................................ 750
10-56 Using vSphere Auto Deploy with vSphere Update Manager ............................ 751
10-57 Review of Learner Objectives ............................................................................ 752
10-58 Key Points .......................................................................................................... 753

Module 11 Appendix: vSphere Security 755

11-2 Importance ........................................................................................................ 756
11-3 Appendix Lessons .............................................................................................. 757
11-4 Lesson 1: Configuring ESXi Host Access and Authentication............................. 758
11-5 Learner Objectives ............................................................................................. 759
11-6 About ESXi Host Security Properties ................................................................. 760
11-7 Configuring the ESXi Firewall............................................................................. 761
11-8 Configuring ESXi Services .................................................................................. 762
11-9 About Lockdown Mode (1) ................................................................................ 763
11-10 About Lockdown Mode (2) ................................................................................ 764
11-11 Normal Lockdown Mode Behavior .................................................................... 765
11-12 Strict Lockdown Mode Behavior ....................................................................... 766
11-13 Integrating ESXi with Active Directory .............................................................. 767
11-14 VMware vSphere Authentication Proxy Service ............................................... 769
11-15 Review of Learner Objectives ............................................................................ 770
11-16 Lesson 2: Securing vSphere ............................................................................... 771
11-17 Learner Objectives ............................................................................................. 772
11-18 About Securing vSphere .................................................................................... 773

Contents xxiii
11-19 About the vSphere Security Configuration Guide ............................................. 774
11-20 Controlling Access to vCenter Server Systems .................................................. 775
11-21 vCenter Server Access Control .......................................................................... 776
11-22 Securing vCenter Server Appliance ................................................................... 777
11-23 Securing vSphere Management Networks ........................................................ 779
11-24 Securing ESXi Hosts ........................................................................................... 781
11-25 VM Protection Overview ................................................................................... 783
11-26 Review of Learner Objectives ............................................................................ 784
11-27 Lesson 3: Security Standards and Protocols ...................................................... 785
11-28 Learner Objectives ............................................................................................. 786
11-29 Securing vCenter Server with TLS 1.2 ................................................................ 787
11-30 Activity: ESXi Secure Boot (1) ............................................................................ 788
11-31 Activity: ESXi Secure Boot (2) ............................................................................ 789
11-32 vSphere Compliance with FIPS 140-2 (1)........................................................... 790
11-33 vSphere Compliance with FIPS 140-2 (2)........................................................... 791
11-34 FIPS 140-2 Architecture ..................................................................................... 792
11-35 Enabling and Disabling FIPS 140-2 Mode on ESXi Hosts ................................... 793
11-36 Enabling and Disabling FIPS 140-2 Mode on vCenter Server Appliance ........... 794
11-37 Activity: Security Technologies (1) .................................................................... 795
11-38 Activity: Security Technologies (2) .................................................................... 796
11-39 Review of Learner Objectives ............................................................................ 797
11-40 Lesson 4: Securing VMs ..................................................................................... 798
11-41 Learner Objectives ............................................................................................. 799
11-42 VM Protection with UEFI Secure Boot............................................................... 800
11-43 Using vTPM in a VM (1) ..................................................................................... 802
11-44 Using vTPM in a VM (2) ..................................................................................... 803
11-45 Prerequisites for Adding vTPM .......................................................................... 804
11-46 Virtualization-Based Security for Windows Guest Operating Systems ............. 805
11-47 VBS Requirements and Limitations ................................................................... 807
11-48 Enabling VBS ...................................................................................................... 808
11-49 About SGX .......................................................................................................... 809

xxiv Contents
11-50 vSGX Use Cases .................................................................................................. 810
11-51 SGX Terminology ............................................................................................... 811
11-52 Intel SGX Architecture ....................................................................................... 812
11-53 vSGX Requirements ........................................................................................... 813
11-54 vSGX Restrictions ............................................................................................... 814
11-55 Enabling and Disabling vSGX for a VM .............................................................. 815
11-56 Review of Learner Objectives ............................................................................ 816
11-57 Lesson 5: VM Encryption ................................................................................... 817
11-58 Learner Objectives ............................................................................................. 818
11-59 About VM Encryption ........................................................................................ 819
11-60 Overview of VM Encryption .............................................................................. 820
11-61 Advantages of VM Encryption ........................................................................... 821
11-62 Business Use Case: Securing VMs...................................................................... 822
11-63 VM Encryption Interoperability (1) ................................................................... 823
11-64 VM Encryption Interoperability (2) ................................................................... 825
11-65 VM Encryption Architecture .............................................................................. 827
11-66 VM Encryption Prerequisites ............................................................................. 828
11-67 Encrypting New VMs ......................................................................................... 829
11-68 Encrypting Existing VMs .................................................................................... 830
11-69 Unlocking Encrypted VMs (1) ............................................................................ 831
11-70 Unlocking Encrypted VMs (2) ............................................................................ 832
11-71 About the KMS .................................................................................................. 833
11-72 KMIP Client Certificate ...................................................................................... 834
11-73 Key Management Clusters................................................................................. 835
11-74 Summary of Keys ............................................................................................... 836
11-75 Role of vCenter Server in VM Encryption .......................................................... 837
11-76 KMS and vCenter Server Communication ......................................................... 838
11-77 Making an ESXi Host Cryptographically Safe ..................................................... 839
11-78 VM Encryption Process Flow (1) ........................................................................ 840
11-79 VM Encryption Process Flow (2) ........................................................................ 841
11-80 Activity: VM Keys (1).......................................................................................... 842

Contents xxv
11-81 Activity: VM Keys (2).......................................................................................... 843
11-82 Managing VM Encryption .................................................................................. 844
11-83 vCenter Server Role: No Cryptography Administrator ...................................... 845
11-84 Activity: Cryptographic Privileges (1) ................................................................ 846
11-85 Activity: Cryptographic Privileges (2) ................................................................ 847
11-86 Pushing Keys to Hosts........................................................................................ 848
11-87 Cross vCenter Migration and Cloning of an Encrypted VM ............................... 849
11-88 Architecture for Cross vCenter Server Migrations and Clones (1) .................... 850
11-89 Architecture for Cross vCenter Server Migrations and Clones (2) .................... 851
11-90 Architecture for Cross vCenter Server Migrations and Clones (3) .................... 852
11-91 Performing a Shallow Recrypt of VMs with Snapshots ..................................... 853
11-92 VM Encryption Operations When Cloning a VM ............................................... 854
11-93 Cloning an Encrypted VM with Instant Clone Technology ................................ 855
11-94 Cloning a VM: Encrypting or Decrypting the Destination VM ........................... 856
11-95 System Requirements When Cloning a VM....................................................... 857
11-96 Backing Up Encrypted VMs (1) .......................................................................... 858
11-97 Backing Up Encrypted VMs (2) .......................................................................... 859
11-98 About Encrypted Core Dumps ........................................................................... 860
11-99 Core Dump Encryption ...................................................................................... 861
11-100 Providing Passwords for Encrypted Core Dumps .............................................. 862
11-101 VM Encryption Events and Alarms .................................................................... 863
11-102 Warning: Encrypted Core Dump Exists (1) ........................................................ 864
11-103 Warning: Encrypted Core Dump Exists (2) ........................................................ 865
11-104 Critical Event: Insufficient Disk Space................................................................ 866
11-105 Critical Event: Key Generation Failed ................................................................ 867
11-106 Critical Event: KMS Unreachable ....................................................................... 868
11-107 About Encrypted vSphere vMotion ................................................................... 869
11-108 Configuring Encrypted vSphere vMotion .......................................................... 871
11-109 Encrypted vSphere vMotion Requirements ...................................................... 872
11-110 Review of Learner Objectives ............................................................................ 873
11-111 Key Points .......................................................................................................... 874

xxvi Contents
11-112 Auto Deploy 1 .................................................................................................... 878
11-113 Auto Deploy 2 .................................................................................................... 879
11-114 vSphere Security 1 ............................................................................................. 882
11-115 vSphere Security 2 ............................................................................................. 883

Contents xxvii
xxviii Contents
Module 1
Course Introduction

Module 1: Course Introduction1

1-2 Course Introduction

2 Module 1: Course Introduction

1-3 Importance

Module 1: Course Introduction3

1-4 Learner Objectives

4 Module 1: Course Introduction

1-5 Course Outline

Those who participate in the VMware vSphere: Optimize and Scale instructor-led training receive
additional recorded lecture material as an exclusive benefit from VMware Learning Zone. This
material is available to participants who sign up at the VMware Learning Zone for a free trial
access period at
https://vmwarelearningzone.vmware.com/oltpublish/site/program.do?dispatch=showCourseSessio
n&id=0dd4d914-733a-11ea-9f48-0cc47adeb5f8

Module 1: Course Introduction5

1-6 Typographical Conventions

6 Module 1: Course Introduction

1-7 References (1)

Module 1: Course Introduction7

1-8 References (2)

8 Module 1: Course Introduction

1-9 VMware Online Resources

Module 1: Course Introduction9

1-10 Review of vSphere User Interfaces

The Flash-based client, known as the vSphere Web Client, is no longer available.

10 Module 1: Course Introduction

1-11 About Scalability

Module 1: Course Introduction11

1-12 About Optimization

12 Module 1: Course Introduction

1-13 About Security

Module 1: Course Introduction13

1-14 Data Center Layout

The data center used in the labs consists of Site A and Site B networks that share a domain
controller (DC-RRAS).
Site A has three ESXi 7.0 hosts (SA-ESXi-04, SA-ESXi-05, and SA-ESXi-06), a vCenter Server
Appliance (SA-VCSA-01), a Key Server (SA-KMS-01), and an administrator system (Student-a-
01). These servers are managed through the SA Management Network (172.20.10.0/24).
Site B has one ESXi 7.0 host (SB-ESXi-01), a vCenter Server Appliance (SB-VCSA-01), and a
Key Server (SB-KMS-01). These servers are managed through the SB Management Network
(172.20.110.0/24).
These servers are connected through the following networks:

• SA Production Network (172.20.11.0/24)

• vMotion Network (172.20.12.0/24)

14 Module 1: Course Introduction

• vSAN Network (172.20.13.0/24)

• SB Production Network (172.20.111.0/24)

Module 1: Course Introduction15

1-15 VMware Education Overview

16 Module 1: Course Introduction

1-16 VMware Certification Overview

Module 1: Course Introduction17

1-17 VMware Digital Badge Overview

Digital badges contain metadata with skill tags and accomplishments, and are based on Mozilla's
Open Badges standard.

18 Module 1: Course Introduction

1-18 About VMware Skyline

VMware Skyline shortens the time it takes to resolve a problem so that you can get back to
business quickly. The VMware Technical Support engineers can use VMware Skyline to view
your environment's configuration and the specific, data-driven analytics to help speed up problem
resolution.

Module 1: Course Introduction19

20 Module 1: Course Introduction
Module 2
Network Scalability

Module 2: Network Scalability21

2-2 Importance

22 Module 2: Network Scalability

2-3 Module Lessons

Module 2: Network Scalability23

2-4 Lesson 1: Creating vSphere Distributed Switches

24 Module 2: Network Scalability

2-5 Learner Objectives

Module 2: Network Scalability25

2-6 About Distributed Switches

Having the network configuration at the data center level (vSphere Distributed Switch), not at the
host level (standard switch), gives distributed switches the following advantages:

• Data center setup and administration are simplified through this centralized network
configuration. For example, adding a host to a cluster and making it compatible with vSphere
vMotion is much easier than with a standard switch.

• Distributed ports migrate with their clients. For example, when you migrate a VM with
vSphere vMotion, the distributed port statistics and policies move with the VM, which
simplifies debugging and troubleshooting.

26 Module 2: Network Scalability

2-7 Distributed Switch Architecture

Managed by vCenter Server, a distributed switch is a logical entity that you can use to create and
maintain a consistent virtual networking configuration throughout all your ESXi hosts.
The distributed switch architecture consists of the control plane and the I/O plane.
The control plane resides in vCenter Server. The control plane configures distributed switches,
distributed port groups, distributed ports, uplinks, NIC teaming, and so on. The control plane also
coordinates the migration of the ports and is responsible for the switch configuration.
The I/O plane is implemented as a hidden virtual switch in the VMkernel of each ESXi host. The
I/O plane manages the I/O hardware on the host and is responsible for forwarding packets. vCenter
Server oversees the creation of these hidden virtual switches.
Each distributed switch includes distributed ports. You can connect any networking entity, such as
a VM or a VMkernel interface, to a distributed port. vCenter Server stores the state of distributed
ports in the vCenter Server database.

Module 2: Network Scalability27

With a distributed port group, you can logically group distributed ports to simplify configuration.
A distributed port group specifies port configuration options for each member port on a distributed
switch. Ports can also exist without port groups.
Uplinks are abstractions of vmnics from multiple hosts to a single distributed switch. An uplink is
to a distributed switch what a vmnic is to a standard switch. Two VMs on different hosts can
communicate with each other only if both VMs have uplinks in the same broadcast domain.

28 Module 2: Network Scalability

2-8 Standard and Distributed Switches: Shared Features

Module 2: Network Scalability29

2-9 Distributed Switch Features

During a vSphere vMotion migration, a distributed switch tracks the virtual networking state (for
example, counters and port statistics) as the VM moves from host to host. This tracking provides a
consistent view of a virtual network interface, regardless of the VM location or vSphere vMotion
migration history.
Tracking simplifies network monitoring and troubleshooting activities where vSphere vMotion is
used to migrate VMs between hosts.

30 Module 2: Network Scalability

2-10 Viewing VDS in vSphere Client

Module 2: Network Scalability31

2-11 Load-Based NIC Teaming

Load-based NIC teaming checks the real load of the uplinks and reduces the load on overloaded
uplinks. No changes on the physical switch are required.
The distributed switch calculates uplinks for VMs by checking the VM port ID and the number of
uplinks in the NIC team. The distributed switch tests the uplinks every 30 seconds. If the load of
an uplink exceeds 75 percent of usage, the port ID of the VM with the highest I/O is moved to a
different uplink.
Load-based NIC teaming is not the default teaming policy. You must configure the policy to use
it.

32 Module 2: Network Scalability

2-12 Discovery Protocols

Discovery protocols are available in vSphere virtual switches.

Cisco Discovery Protocol (CDP) was developed by Cisco Systems to broadcast connected devices
information at network layer 2. CDP is supported in vSphere since version 4.0.
Link Layer Display Protocol (LLDP) was introduced in vSphere 5.0. LLDP supports the
standards-based IEEE 802.1AB discovery protocol.
Network devices use CDP or LLDP to advertise their identity, capabilities, and neighbors on a
network.

Module 2: Network Scalability33

2-13 Configuring CDP or LLDP

With CDP and LLDP, the vSphere Client can identify properties of a physical switch, such as
switch name, port number, and port speed or duplex settings. You can also configure CDP or
LLDP so that information about physical adapters and ESXi host names is passed to the CDP- or
LLDP-compatible switches.
You can configure the discovery protocol to use one of the following modes of operation:

• Listen (default): The ESXi host detects and displays information about the associated physical
switch port, but information about the virtual switch is not available to the physical switch
administrator.

• Advertise: The ESXi host provides information about the virtual switch to the physical switch
administrator but does not detect and display information about the physical switch.

• Both: The ESXi host detects and displays information about the associated physical switch
and provides information about the virtual switch to the physical switch administrator.

34 Module 2: Network Scalability

You can use the esxcli command to enable CDP on a standard switch.

Module 2: Network Scalability35

2-14 About Port Binding

When you connect a VM to a port group that is configured with static binding, a port is
immediately assigned and reserved for the VM, guaranteeing connectivity at all times. The port is
disconnected only when the VM is removed from the port group. Static binding is recommended
for general use.
If you select static binding, the default number of ports is set to eight. Elastic is the default port
allocation setting.
With ephemeral binding, a port is created and assigned to a VM when the VM is powered on and
its NIC is in a connected state. The port is deleted when the VM is powered off or the VM NIC is
disconnected.
You can make ephemeral port assignments through the ESXi host and vCenter Server, providing
flexibility in managing VM connections through the host when vCenter Server is down. Only an
ephemeral binding allows you to modify VM network connections when vCenter Server is down.
However, network traffic is unaffected by a vCenter Server failure, regardless of the port binding
type.

36 Module 2: Network Scalability

Use ephemeral port groups only for recovery purposes when you want to provision ports directly
on an ESXi host, bypassing vCenter Server, and not in any other case.

Module 2: Network Scalability37

2-15 About the Traffic Filtering and Marking Policy

The traffic filtering and marking policy consists of one or more network traffic rules, defined at
the distributed port group or uplink port group level.
Use the traffic filtering and marking policy to create a set of rules for security and QoS tagging of
packets flowing through distributed switch ports.
The distributed switch applies rules on traffic at different places in the data stream. It can apply
network traffic rules on the data path between the VM network adapter and the distributed port. It
can also apply network traffic rules between the uplink port and the physical network adapter.
The traffic filtering and marking policy is supported only on distributed switches.

38 Module 2: Network Scalability

2-16 Creating Network Traffic Rules

In general, a network traffic rule consists of a qualifier for traffic and an action for restricting or
prioritizing the matching traffic.
In a network traffic rule, the Allow action allows traffic to pass through the distributed switch port
or uplink port, and the Drop action blocks the traffic. The Tag action marks (or tags) traffic
passing through the distributed switch port or uplink port.
Traffic direction is for the distributed switch. The direction can be ingress (traffic entering the
distributed switch), egress (traffic leaving the distributed switch), or both. The direction influences
how you identify the traffic source and destination.
A qualifier represents a set of matching criteria related to a networking layer. You can match
traffic based on the system traffic type, layer 2 traffic properties, or layer 3 traffic properties. You
can use the qualifier for a specific networking layer, or you can combine qualifiers to match
packets more precisely.

Module 2: Network Scalability39

2-17 Network Traffic Rule: Example 1

Use a MAC traffic qualifier to define matching criteria for the layer 2 (data link layer) properties
of packets. These properties include MAC address, VLAN ID, and a protocol that consumes the
frame payload (IPv4, IPv6, or Address Resolution Protocol).
Locating traffic with a VLAN ID on a distributed port group works with Virtual Guest Tagging.
To match traffic to VLAN ID if Virtual Switch Tagging is active, use a rule on an uplink port
group or uplink port.

40 Module 2: Network Scalability

2-18 Network Traffic Rule: Example 2

Use the system traffic qualifier to match packets to the type of virtual infrastructure data that is
flowing through the ports of the distributed port group.
In this example, the rule enforces that the traffic through the pg-SA Production port group must be
VM traffic. The distributed switch drops any other types of traffic that are sent to this port group.
You can select the type of traffic through the ports of the group that carries system data, such as
traffic for management from vCenter Server, iSCSI storage, and vSphere vMotion.
When you use qualifiers for the system data type, both layer 2 and layer 3 packet attributes set the
properties that packets must have to match the rule.

Module 2: Network Scalability41

2-19 Marking Network Traffic (1)

Marking, or priority tagging, is a way to mark traffic that has higher Quality of Service (QoS)
demands. With priority tagging, the network can recognize different classes of traffic. Network
devices can handle the traffic from each class according to its priority and requirements.
Similar to traffic filtering, marking also requires that you classify the traffic first, based on these
qualifiers: system, MAC, and IP. After you define your traffic qualifiers, you can decide how to
mark your traffic.
You can prioritize traffic by using a Class of Service (CoS) priority tag, a Differentiated Serviced
Code Point (DSCP) tag, or both.
You can mark traffic with a CoS priority tag in the layer 2 packet header. Accepted values are 0
through 7.
You can mark traffic with a DSCP tag in the layer 3 packet header. Accepted values are 0 through
63.

42 Module 2: Network Scalability

You can assign priority tags to traffic, such as VoIP and streaming video, that has higher
networking requirements for bandwidth, low latency, and so on.

Module 2: Network Scalability43

2-20 Marking Network Traffic (2)

This example marks VoIP traffic. VoIP flows have special requirements for QoS in terms of low
loss and delay. The traffic related to Session Initiation Protocol (SIP) for VoIP usually has a DSCP
tag equal to 26.

44 Module 2: Network Scalability

2-21 Access to NSX Virtual Distributed Switch (1)

Module 2: Network Scalability45

2-22 Access to NSX Virtual Distributed Switch (2)

Starting with vSphere 7, NSX port groups behave as regular distributed port groups so VMkernel
adapters can be added to NSX port groups from the vSphere Client.
The vSphere Client does not support the following NSX port group actions:

• Delete

• Modify settings

• Use in port mirroring session

• Include configuration in VDS exports

46 Module 2: Network Scalability

2-23 Access to NSX Virtual Distributed Switch (3)

Module 2: Network Scalability47

2-24 Activity: Distributed Switch Features (1)

48 Module 2: Network Scalability

2-25 Activity: Distributed Switch Features (2)

Module 2: Network Scalability49

2-26 Labs

50 Module 2: Network Scalability

2-27 Lab 1: Accessing the Lab Environment

Module 2: Network Scalability51

2-28 Lab 2: Configuring vSphere Distributed Switches (1)

52 Module 2: Network Scalability

2-29 Lab 3: Configuring vSphere Distributed Switches (2)

Module 2: Network Scalability53

2-30 Review of Learner Objectives

54 Module 2: Network Scalability

2-31 Lesson 2: Managing vSphere Distributed Switches

Module 2: Network Scalability55

2-32 Learner Objectives

56 Module 2: Network Scalability

2-33 About Health Check for Distributed Switches

Health check is a feature that detects certain inconsistencies between the physical and the virtual
networks. Key parameters, such as VLAN tags, MTUs, and the NIC teaming configuration, must
be configured consistently on the physical and the virtual switches. An inconsistent configuration
can lead to network connectivity problems.
Health check searches for configuration inconsistencies and reports them to the administrator. The
default check interval is one minute.
As a best practice, enable health check for brief time periods because this feature creates many
MAC addresses and can affect production traffic.

Module 2: Network Scalability57

2-34 Health Check Example

In addition to the distributed switch, two physical switches with their switch port configurations
are shown.
Compare the virtual port group (VLAN 10) with the settings on physical switch 2. The VLAN ID
and the MTU settings are identical. The virtual port group is configured with the port ID teaming
selection. But because the configuration is internal to the virtual switch and requires no
configuration on the physical switch, the virtual and the physical settings are consistent and should
not experience connectivity issues.
Compare the virtual port group (VLAN 20) to the settings on physical switch 1. The VLAN IDs
and the MTU settings are different. The virtual port group teaming setting is set to IP hash. For IP
hash to operate properly, EtherChannel or Link Aggregation Control Protocol (LACP) must be
configured on the physical switch.
Health check can detect and report the configuration differences between the port group and the
switch port configuration by using layer 2 Ethernet packets. At one-minute intervals (by default),

58 Module 2: Network Scalability

request and acknowledge packets are sent back and forth between the virtual interface and the
switch. When packets are dropped, a configuration warning appears in the vSphere Client.

Module 2: Network Scalability59

2-35 Enabling Health Check

Health check has the following requirements:

• For VLAN and MTU checks, at least two physical uplinks must be connected to the
distributed virtual switch.

• For the teaming policy check, at least two active uplinks must be in the team, and at least two
ESXi hosts must be connected to the virtual distributed switch.

See vSphere Networking at https://docs.vmware.com/en/VMware-vSphere/7.0/vsphere-esxi-

vcenter-server-70-networking-guide.pdf.

60 Module 2: Network Scalability

2-36 Monitoring Health Check Results

The health status details are organized into three categories: VLAN, MTU, and Teaming and
Failover.

Module 2: Network Scalability61

2-37 Backing Up and Restoring Configurations

You perform these operations by using the export, import, and restore functions available for
distributed switches.
You have other available restore options if a switch configuration is lost, for example, when
vCenter Server database corruption occurs, or if the virtual switch or port group settings are
misconfigured.
Ways to restore the virtual switch include restoring the database completely or rebuilding the
switch. Although both measures restore the switch, they can be time-consuming.

62 Module 2: Network Scalability

2-38 Exporting Distributed Switch Configurations

You export the distributed switch and distributed port group configuration to a file on the system
that is running the vSphere Client. The file preserves valid network configurations, which can then
be distributed to other deployments.
By making periodic backups with the export function, you can preserve your distributed switch
configuration if a failure occurs, such as the corruption of the vCenter Server database.
You can use the template created from the export function to create similar distributed switch
configurations on other vCenter Server systems.
You can keep revisions by saving the distributed switch configuration after each change. By
keeping revisions, you can restore the current configuration to an older configuration if necessary.
You can automate this task with the following PowerCLI cmdlets:

Module 2: Network Scalability63

• Export-VDSwitch: Exports the configuration of a specified vSphere distributed switch to
a ZIP file.

• Export-VDPortGroup: Exports the configuration of a specified distributed port group to

a specified ZIP file.

64 Module 2: Network Scalability

2-39 Restoring and Importing Distributed Switch
Configurations

With the restore and import functions, you can recreate a distributed switch configuration.
Restoring a distributed switch configuration overwrites the current settings of the distributed
switch and its port groups. Full functionality is restored if network settings fail or the vCenter
Server database is corrupted. The restore function does not delete port groups that are not part of
the configuration file.
When you import a distributed switch configuration, you can create multiple copies of an existing
deployment.

Module 2: Network Scalability65

2-40 Rollback and Recovery of the Management Network

Automatic rollback and recovery is a vSphere feature that helps prevent management network
outages. You protect the management network in the following ways:

• The automatic rollback feature detects any configuration changes on the management
network. If the host cannot reach vCenter Server, the changes do not take effect.

• You can also reconfigure the management network on the distributed virtual switch by using
the Direct Console User Interface (DCUI). Using the DCUI to change the management
network parameters of the switch changes those parameters for all hosts connected to the
switch.

66 Module 2: Network Scalability

2-41 Rollback Details

The following types of updates can trigger a rollback:

• If a change in the host networking configuration causes connectivity problems between the
host and the vCenter Server system, a rollback occurs. Examples of misconfigurations include
a mismatch in the speed and duplex of the physical NIC, a mismatch in the jumbo frame’s
configuration, and a misconfiguration in the IP settings of the management network.

• If misconfiguration of the distributed virtual switch causes connectivity issues between the
hosts and the vCenter Server system, a rollback can occur. Port group and distributed virtual
port misconfigurations can cause network downtime.

You can disable rollback by changing the vCenter Server config.vpxd.network.rollback advanced
setting to false. After you modify the advanced setting, you must restart the vCenter Server system
for the new setting to take effect.

Module 2: Network Scalability67

2-42 Recovery Through the DCUI

To roll back the configuration manually from the DCUI, take these steps:
1. In the DCUI, press the F2 key.
2. When prompted, log in to the DCUI with the appropriate credentials.
3. Use the down arrow key to scroll to Network Restore Options and press Enter.
4. Under Network Restore Options, select Restore vDS to roll back the distributed network
settings.
You can either restore the host settings to the default blocked setting or restore the default
teaming policy of the host.
5. Select the appropriate option and press Enter.

68 Module 2: Network Scalability

2-43 Lab 4: Managing vSphere Distributed Switches

Module 2: Network Scalability69

2-44 Review of Learner Objectives

70 Module 2: Network Scalability

2-45 Lesson 3: Using Network I/O Control in Distributed
Switches

Module 2: Network Scalability71

2-46 Learner Objectives

72 Module 2: Network Scalability

2-47 About Network I/O Control (1)

Module 2: Network Scalability73

2-48 About Network I/O Control (2)

For more information about the procedure for upgrading Network I/O Control, see vSphere
Networking at https://docs.vmware.com/en/VMware-vSphere/7.0/vsphere-esxi-vcenter-server-70-
networking-guide.pdf.

74 Module 2: Network Scalability

2-49 About the Bandwidth Allocation Model for System
Traffic

The amount of bandwidth that is available to a system traffic type is determined by its relative
shares and the amount of data that the other system features transmit.
Reserved bandwidth that is unused becomes available to other types of system traffic. However,
Network I/O Control 3.0 does not redistribute unused system traffic capacity to VM placement.
For example, you configure a reservation of 2 Gbps for iSCSI. Because iSCSI uses a single path,
the distributed switch might never impose this reservation on a physical adapter. The vacant
bandwidth is not allocated to VM system traffic.
Network I/O Control 3.0 cannot safely meet a potential need for bandwidth for system traffic. This
failure to reallocate unused reserved bandwidth to VM system traffic is especially true of a new
iSCSI path where you must provide bandwidth to a new VMkernel adapter.

Module 2: Network Scalability75

2-50 Configuring Bandwidth Allocations for System Traffic

You can use Network I/O Control 3.0 to configure bandwidth allocation for the traffic that is
related to the main features of vSphere:

• VMs

• Management

• vSphere vMotion

• NFS

• vSphere Fault Tolerance

• iSCSI

• vSphere Replication

76 Module 2: Network Scalability

• vSAN

• vSphere Data Protection

Module 2: Network Scalability77

2-51 Reserving Bandwidth for System Traffic (1)

You can guarantee a minimum bandwidth to a system feature for optimal operation according to
the capacity of the physical adapters.
For a distributed switch that is connected to ESXi hosts with 10 GbE network adapters, you can
configure a reservation to guarantee 1 Gbps for vSphere vMotion traffic, 1 Gbps for vSphere Fault
Tolerance, 0.5 Gbps for VM traffic, and so on.
You might leave more capacity unreserved so that the host can dynamically allocate bandwidth
according to shares, limits, and use, and to reserve only enough bandwidth for the operation of a
system feature.
You can reserve up to 75 percent of the bandwidth of a physical network adapter. For example, if
your ESXi host uses a 40 GbE network adapter, 30 Gbps (.75 x 40) is reserved.

78 Module 2: Network Scalability

2-52 Reserving Bandwidth for System Traffic (2)

Module 2: Network Scalability79

2-53 Bandwidth Aggregation for Network Resource Pools

Network I/O Control 3.0 allocates bandwidth for VMs across the entire distributed switch and on
the physical adapter carrying the VM traffic.
To use Network I/O Control 3.0 to enable bandwidth allocation for VMs, configure the VM
system traffic. The bandwidth reservation for VM traffic is also used in admission control. When
you power on a VM, admission control verifies that enough bandwidth is available.

80 Module 2: Network Scalability

2-54 Creating Network Resource Pools

For example, if the VM system traffic has 0.5 Gbps reserved for each 10 GbE uplink on a
distributed switch that has 10 uplinks, the total aggregated bandwidth that is available for VM
reservation on this switch is 5 Gbps. Each network resource pool can reserve a quota of this 5
Gbps capacity.
The bandwidth quota that is dedicated to a network resource pool is shared among the distributed
port groups associated with the pool. A VM receives bandwidth from the pool through the
distributed port group that the VM is connected to.

Module 2: Network Scalability81

2-55 Bandwidth Allocation for VM Traffic

A network resource pool provides a reservation quota to VMs. The quota represents a portion of
the bandwidth that is reserved for VM system traffic on the physical adapters connected to the
distributed switch.
You can set aside bandwidth from the quota for the VMs that are associated with the pool. The
reservation from the network adapters of powered-on VMs that are associated with the pool must
not exceed the quota of the pool.

82 Module 2: Network Scalability

2-56 Bandwidth Allocation for Individual VMs

To guarantee bandwidth, Network I/O Control 3.0 implements a traffic placement engine that
becomes active if a VM has bandwidth reservation configured. The distributed switch tries to
direct the traffic from a VM network adapter to a physical adapter that can supply the required
bandwidth and is in the scope of the active teaming policy.
The real limit and reservation also depend on the traffic-shaping policy on the distributed port
group that the adapter is connected to. For example, if a VM network adapter asks for a limit of
200 Mbps and the average bandwidth configured in the traffic-shaping policy is 100 Mbps, the
effective limit becomes 100 Mbps.

Module 2: Network Scalability83

2-57 Bandwidth Admission Control in vSphere DRS (1)

If you power on a VM that is in a cluster, vSphere DRS places that VM on a host with the capacity
to guarantee the bandwidth that is reserved for the VM, according to the active teaming policy.
To use admission control in vSphere DRS, you must perform the following tasks:

• Configure bandwidth allocation for the VM system traffic on the distributed switch.

• Configure the bandwidth requirements of a VM that is connected to the distributed switch.

If a VM reservation cannot be met, the VM is not powered on.

84 Module 2: Network Scalability

2-58 Bandwidth Admission Control in vSphere DRS (2)

In this example, Host1 loses an uplink, which leaves Host1 with one working uplink. Because
Host2's uplinks still work, vSphere DRS migrates either VM1 or VM2 to Host2 to meet the VM’s
network reservation requirement.

Module 2: Network Scalability85

2-59 Bandwidth Admission Control in vSphere HA

Bandwidth admission control prevents a VM from starting if the bandwidth reservation for that
VM cannot be met.
To use admission control in vSphere HA, you must perform the following tasks:

• Allocate bandwidth for VM system traffic.

• Configure the bandwidth requirements of a VM that is connected to the distributed switch.

86 Module 2: Network Scalability

2-60 Activity: Network I/O Control (1)

Module 2: Network Scalability87

2-61 Activity: Network I/O Control (2)

88 Module 2: Network Scalability

2-62 Review of Learner Objectives

Module 2: Network Scalability89

2-63 Lesson 4: NetFlow and Port Mirroring in Distributed
Switches

90 Module 2: Network Scalability

2-64 Learner Objectives

Module 2: Network Scalability91

2-65 About NetFlow

NetFlow is a protocol that Cisco Systems developed for analyzing network traffic. NetFlow is an
industry-standard specification for collecting network data for monitoring and reporting. The data
sources are network devices, such as switches and routers. For ESXi deployments, NetFlow
supports detailed monitoring and analysis of VM network traffic.
Standard switches do not support NetFlow.
NetFlow collectors are available from third-party providers.

92 Module 2: Network Scalability

2-66 About Network Flows

Network flows give you a complete view of VM traffic, which can be collected for historical
views and used for multiple purposes.
Internal flows are generated from intrahost VM traffic, that is, traffic between VMs on the same
hosts.
External flows are generated from traffic between VMs on different hosts or VMs on different
distributed switches. External flows are also generated from traffic between physical machines and
VMs.
A flow is a sequence of packets that share properties such as source and destination IP addresses,
source and destination ports, input and output interface IDs, and protocol.
A flow is unidirectional. Flows are processed and stored as flow records by supported network
devices, such as a distributed switch. The flow records are sent to a NetFlow collector for
additional analysis.

Module 2: Network Scalability93

Although flow processing is an efficient method, NetFlow can strain the distributed switch.
NetFlow requires both additional processing and storage on the host for the flow records to be
processed and exported.

94 Module 2: Network Scalability

2-67 Network Flow Analysis (1)

NetFlow sends aggregated network flow data to a NetFlow collector. Third-party vendors have
NetFlow collector products.
A NetFlow collector accepts and stores the completed network flow records. NetFlow collectors
vary in functionality by vendor. A NetFlow collector provides the following features:

• Analysis software to mine, aggregate, and report on the collected data

• A storage system for long-term storage so that you can archive the network flow data

• A customized user interface, often based on a web browser

The NetFlow collector can report on various kinds of networking information:

• The current top network flows consuming the most bandwidth in a particular (virtual) switch

• The IP addresses that are behaving irregularly

• The number of bytes that a particular VM sent and received in the past 24 hours

Module 2: Network Scalability95

2-68 Network Flow Analysis (2)

With NetFlow data, you can investigate the causes of excessive use of network bandwidth,
bottlenecks, and unexpected application traffic. The historical records that you put in long-term
storage can help you diagnose what caused these outages or breaches.
Because NetFlow data comes from NetFlow-enabled network devices, additional network probes
to collect the flow-based data are not required. NetFlow collectors and analyzers can provide a
detailed set of network performance data. Given enough storage on the NetFlow collector, flow
data can be archived for a long time, providing a long-term record of network behavior.

96 Module 2: Network Scalability

2-69 Configuring NetFlow on Distributed Switches

To configure NetFlow on a distributed switch, specify the following NetFlow settings:

• Collector IP address and Collector port: The IP address and port number that are used to
communicate with the NetFlow collector system. The values are assigned by the collector.

• Observation Domain ID: A value that identifies the information related to the switch.

• Switch IP address: An optional IP address that is used to identify the source of the network
flow to the NetFlow collector. The IP address is not associated with a network port, and it
does not have to be pingable.

This IP address is used to fill the source IP field of NetFlow packets. With the address, the
NetFlow collector can interact with the distributed switch as a single switch, rather than as a
separate, unrelated switch for each associated host. If this IP address is not configured, the
host’s management IP address is used instead.

Module 2: Network Scalability97

For a description of the advanced NetFlow settings, see vSphere Networking at
https://docs.vmware.com/en/VMware-vSphere/7.0/vsphere-esxi-vcenter-server-70-networking-
guide.pdf.

98 Module 2: Network Scalability

2-70 Enabling NetFlow on a Distributed Port Group

Module 2: Network Scalability99

2-71 About Port Mirroring

On physical switches, administrators often must mirror traffic to special ports when
troubleshooting network-related problems. Port mirroring is commonly used for network
appliances that require the monitoring of network traffic, for example, intrusion detection systems.
Many network switch vendors implement port mirroring in their products. For example, port
mirroring on a Cisco Systems switch is called Switched Port Analyzer (SPAN).
With port mirroring, you do not need to enable promiscuous mode on a distributed switch to
troubleshoot network issues. If you enable promiscuous mode on a distributed port, the port sees
all the network traffic passing through the distributed switch. You cannot select the port or port
group that a promiscuous port is allowed to see. The promiscuous port sees all traffic that is on the
broadcast domain.
Port mirroring supports Cisco Remote Switch Port Analyzer (RSPAN) and Encapsulated Remote
Switch Port Analyzer (ERSPAN). With RSPAN, mirrored traffic can be directed to a remote
monitor. The RSPAN session can span multiple switch hops on a network. With ERSPAN, the
session can span an IP network.

100 Module 2: Network Scalability

2-72 Port Mirroring Session Types

You can select from the following port mirroring session options:

• Distributed Port Mirroring: Port mirroring between distributed ports can only be local. If
the source and the destination ports are on different hosts because of vSphere vMotion
migration, mirroring between them does not work. However, if the source and destination
ports move to the same host, port mirroring works.

• Remote Mirroring Source: When a source-distributed port is moved from host A to host B,
the original mirroring path from the source port to A’s destination uplink is removed on A. A
new mirroring path from the source port to B’s destination uplink is created on B. The uplink
that is used is determined by the uplink name specified in the session.

Module 2: Network Scalability101

• Remote Mirroring Destination: When a destination-distributed port is moved from host A to
host B, all original mirroring paths from source VLANs to the destination port are moved
from A to B.

• Encapsulated Remote Mirroring (L3) Source: When a source-distributed port is moved

from host A to host B, all original mirroring paths from the source port to the destination IP
addresses are moved from A to B.

The type of source and destination involved in port mirroring depends on the session type that is
selected for port mirroring.

102 Module 2: Network Scalability

2-73 Port Mirroring Session Properties

A distributed port mirroring type session can have the following properties:

• TCP/IP stack type (default or mirror)

• Normal I/O on destination ports (can be allowed or disallowed)

• Mirrored packet length (in bytes)

• Rate at which packets are sampled

• Description

Every port mirroring session is uniquely identified by its name. A session can also have a
description.
For a description of the advanced session properties, see vSphere Networking at
https://docs.vmware.com/en/VMware-vSphere/7.0/vsphere-esxi-vcenter-server-70-networking-
guide.pdf.

Module 2: Network Scalability103

2-74 Source and Destination in a Port Mirroring Session

When creating a port mirroring session, you must configure the source and the destination. When
configuring the source, you must specify the traffic direction.
Traffic direction is categorized as ingress or egress. Ingress traffic direction is traffic flowing from
the source VM into the distributed switch. Egress traffic direction is traffic flowing from the
distributed switch into the source VM.
To avoid flooding the network with mirrored traffic, port mirroring has the following restrictions:

• In a session, a port cannot be a source and a destination.

• A port cannot be a destination for more than one session.

• A promiscuous port cannot be an egress source or destination.

• An egress source cannot be a destination for sessions, to avoid cycles of mirroring paths.

104 Module 2: Network Scalability

2-75 Lab 5: Using Port Mirroring

Module 2: Network Scalability105

2-76 Review of Learner Objectives

106 Module 2: Network Scalability

2-77 Key Points

Module 2: Network Scalability107

108 Module 2: Network Scalability
Module 3
Storage Scalability

Module 3: Storage Scalability109

3-2 Importance

110 Module 3: Storage Scalability

3-3 Module Lessons

Module 3: Storage Scalability111

3-4 Lesson 1: VMFS Datastores

112 Module 3: Storage Scalability

3-5 Learner Objectives

Module 3: Storage Scalability113

3-6 About VMFS Datastores

114 Module 3: Storage Scalability

3-7 VMFS5 and VMFS6 Datastores

For more information about the hot extend feature, see vSphere Storage at
https://docs.vmware.com/en/VMware-vSphere/7.0/vsphere-esxi-vcenter-server-70-storage-
guide.pdf.

Module 3: Storage Scalability115

3-8 VMFS6 Datastores

116 Module 3: Storage Scalability

3-9 VMFS 6 New Features in vSphere 7

Affinity 2.0 Manager is a smarter and more efficient way to allocate resources and minimize
overhead in the I/O path:

• Creates a cluster-wide view of the resource allocation status of the resource clusters (RC),
dividing disk space into “regions.”

• Ranks regions into Region Maps based on the number of blocks allocated by the current host
as well as other hosts.

Module 3: Storage Scalability117

3-10 ESXi Support for VMFS5 and VMFS6

118 Module 3: Storage Scalability

3-11 Storage Device Formats

The storage industry is moving towards advanced format drives to provide large capacity drives to
servers and storage arrays.
4K drives have advantages over 512-byte sector drives:

• Improved total cost of ownership: 4K drives require less space for error correction codes
(ECCs) than regular 512-byte sector drives. This space saving results in greater data density
on 4K drives, which leads to an improved total cost of ownership.

• Improved availability: 4K drives have a larger ECC field, which improves error correction
and data integrity.

• Improved performance: 4K drives are expected to have better performance than 512-byte
sector drives. However, this improved performance is true only when the guest operating
system is configured to issue I/O that are aligned to the 4K sector size.

Module 3: Storage Scalability119

Because legacy applications and operating systems do not support 4K native drives, the drive
industry introduced 4K sector size drives in 512-byte emulation mode to support legacy
applications. The drives have a physical sector size of 4K, but the logical sector size is 512 bytes.
These drives are called 512e drives, where the letter e stands for emulation mode. 512e drives help
drive adoption of 4K drives. The goal is to move to 4K native drives.

120 Module 3: Storage Scalability

3-12 Storage Devices Supported by VMFS Datastores

For more information about vSphere support for 512e and 4K native drives, see VMware
knowledge base article 2091600 at http://kb.vmware.com/kb/2091600.

Module 3: Storage Scalability121

3-13 Limitations of Using 4Kn Drives

122 Module 3: Storage Scalability

3-14 Snapshot Formats for VMFS

When you take a snapshot, the state of the virtual disk is preserved, preventing the guest operating
system from writing to the disk. A delta or child disk is created. The delta disk represents the
difference between the current state of the virtual disk and the state that existed when the previous
snapshot was taken.
VMFSsparse is implemented on top of VMFS. The VMFSsparse layer processes I/O issued to a
snapshot VM. Technically, VMFSsparse is a redo-log that starts empty, immediately after a VM
snapshot is taken. The redo-log expands to the size of its base VM disk (VMDK), when the entire
VMDK is rewritten with new data after the VM snapshots are taken. This redo-log is only a file in
the VMFS datastore. After the snapshot is taken, the base VMDK that is attached to the VM is
changed to the newly created sparse VMDK.
With SEsparse space reclamation, blocks that are deleted by the guest OS are marked, and
commands are issued to the SEsparse layer in the hypervisor to unmap those blocks. This
operation helps reclaim the space allocated by SEsparse after the guest OS deletes that data.
SEsparse is the recommended disk format for VMware Horizon environments. In these
environments, reclamation of storage space is critical because of the many tenants sharing storage.

Module 3: Storage Scalability123

3-15 Migrating Your Data from VMFS5 to VMFS6

For more information, see VMware knowledge base article 2147824 at

https://kb.vmware.com/s/article/2147824.

124 Module 3: Storage Scalability

3-16 Activity: VMFS6 Features (1)

Module 3: Storage Scalability125

3-17 Activity: VMFS6 Features (2)

126 Module 3: Storage Scalability

3-18 Review of Learner Objectives

Module 3: Storage Scalability127

3-19 Lesson 2: vSphere Storage APIs - Array Integration

128 Module 3: Storage Scalability

3-20 Learner Objectives

Module 3: Storage Scalability129

3-21 About vSphere Storage APIs - Array Integration

vSphere Storage APIs is a family of APIs that are used by third-party hardware, software, and
storage vendors to develop components that enhance several vSphere features and solutions.
In a virtualized environment, virtual disks are files on a VMFS datastore. Disk arrays cannot
interpret the VMFS datastore’s on-disk data layout, so the VMFS datastore cannot use hardware
functions per VM or per virtual disk file.
vSphere Storage APIs - Array Integration plug-ins can improve data transfer performance, and
they are transparent to the end user.

130 Module 3: Storage Scalability

3-22 Hardware Acceleration APIs for Block Storage

With storage hardware assistance, your host performs array operations faster and consumes less
CPU, memory, and storage fabric bandwidth.
ESXi hardware acceleration for block storage devices supports the following array operations:

• Full copy: Storage arrays can make full copies of data in the array. The host does not need to
read and write the data. This operation reduces the time and network load when cloning VMs,
provisioning from a template, or migrating with vSphere Storage vMotion.

• Block zeroing: Storage arrays can zero out several blocks to provide newly allocated storage,
free of previously written data. This operation reduces the time and network load when
creating VMs and formatting virtual disks. This operation is activated when the Thick
Provision Eager Zero option is used for a disk.

• Hardware-assisted locking: This option supports discrete VM locking without use of SCSI
reservations. Disk locking is per sector, unlike SCSI reservations, which locks the entire
logical unit number (LUN).

Module 3: Storage Scalability131

The hardware acceleration features are enabled by default and are used on storage arrays that
support these features. You can disable these features, but it is not recommended. Older arrays, or
arrays with outdated firmware, might not support hardware acceleration. In such cases, the host
falls back on older and more CPU-intensive techniques to perform the same tasks.
For more information about vSphere Storage APIs - Array Integration, see VMware knowledge
base article 1021976 at http://kb.vmware.com/kb/1021976.

132 Module 3: Storage Scalability

3-23 Hardware Acceleration for NAS

With hardware acceleration, your host can integrate with NAS devices and use several hardware
operations that NAS storage provides:

• Full file clone: This operation is similar to VMFS block cloning, except that NAS devices
clone entire files instead of file segments.

• Reserve space: Storage arrays allocate space for a virtual disk file in thick format.
Typically, when you create a virtual disk on an NFS datastore, the NAS server determines the
allocation policy. The default allocation policy on most NAS servers is thin and does not
guarantee backing storage to the file. However, the reserve space operation can instruct the
NAS device to use vendor-specific mechanisms to reserve space for a virtual disk. As a result,
you can create thick virtual disks on the NFS datastore.

Module 3: Storage Scalability133

• Native snapshot support: This operation enables View to offload the creation of linked clones
to a NAS array.

• Extended file statistics: Storage arrays can accurately report space utilization.

Hardware acceleration for NAS is implemented through vendor-specific NAS plug-ins. Vendors
typically create and support these plug-ins and distribute them as vSphere installation bundles
(VIBs) through a webpage.

134 Module 3: Storage Scalability

3-24 Array Thin Provisioning APIs

Traditional LUNs that arrays present to the ESXi host are thick-provisioned. The entire physical
space that is required to back each LUN is allocated in advance.
ESXi also supports thin-provisioned LUNs. When a LUN is thin-provisioned, the storage array
reports the LUN logical size, which might be larger than the real physical capacity backing that
LUN.
When the datastore expands, or when vSphere Storage vMotion is used to migrate VMs to a thin-
provisioned LUN, the array thin provisioning APIs enable the host to communicate with the LUN.
The host can warn about breaches in physical space or about out-of-space conditions
No installation steps are required for the array thin provisioning extensions. Array thin
provisioning works on all VMFS5 and VMFS6 volumes. Device firmware that is enabled for this
API must use the array thin provisioning features. ESXi continuously checks for firmware that is
compatible with array thin provisioning. After firmware is upgraded, ESXi starts using the array
thin provisioning features.

Module 3: Storage Scalability135

3-25 About Space Reclamation (1)

136 Module 3: Storage Scalability

3-26 About Space Reclamation (2)

Deleting or removing files from a VMFS datastore frees space within the file system. This free
space is mapped to a storage device until the file system releases or unmaps it.

Module 3: Storage Scalability137

3-27 Space Reclamation with VMFS

When you run the unmap command, an ESXi host informs the storage array that the VM files are
removed or deleted from a thin-provisioned VMFS datastore. At this notification, the array can
reclaim the freed blocks.

138 Module 3: Storage Scalability

3-28 Performing Manual Space Reclamation on a VMFS5
Datastore

On VMFS5 datastores, you must run the esxcli storage vmfs unmap command
manually. The target server prompts you for a user name and password. Other connection options,
such as a configuration file and a session file, are supported.
This command can be run without a maintenance window:

• You can specify the reclaim size in blocks, instead of a percentage value, to make the reclaim
size more intuitive to calculate.

• Dead space is reclaimed in increments, instead of all at once, to avoid possible performance
issues.

When you run the command, it might send many unmap requests at a time, which can lock some
of the resources during this operation.

Module 3: Storage Scalability139

For more information about this command, see vSphere Storage at
https://docs.vmware.com/en/VMware-vSphere/7.0/vsphere-esxi-vcenter-server-70-storage-
guide.pdf.

140 Module 3: Storage Scalability

3-29 Enabling Automatic Space Reclamation on a VMFS6
Datastore

By default, the space reclamation granularity equals the block size, which is 1 MB. Storage sectors
smaller than 1 MB are not reclaimed.
By default, the LUN performs the space reclamation operation at a low rate. You can also set the
space reclamation priority to None to disable the operation for the datastore.

Module 3: Storage Scalability141

3-30 Configuring the Automatic Unmap Rate

142 Module 3: Storage Scalability

3-31 Space Reclamation Support for Guest Operating
Systems (1)

ESXi supports the unmap commands that are issued directly from a guest operating system to
reclaim storage space.

Module 3: Storage Scalability143

3-32 Space Reclamation Support for Guest Operating
Systems (2)

144 Module 3: Storage Scalability

3-33 Space Reclamation Support for Guest Operating
Systems (3)

VMFS6 generally supports automatic space reclamation requests from the guest operating systems
and passes these requests to the array.
Many guest operating systems can send the unmap command and do not require any additional
configuration.
The guest operating systems that do not support automatic unmaps might require user
intervention.
For information about guest operating systems that support automatic space reclamation on
VMFS6, contact your vendor.

Module 3: Storage Scalability145

3-34 Activity: vSphere Storage APIs – Array Integration (1)

146 Module 3: Storage Scalability

3-35 Activity: vSphere Storage APIs – Array Integration (2)

Module 3: Storage Scalability147

3-36 Review of Learner Objectives

148 Module 3: Storage Scalability

3-37 Lesson 3: vSphere Storage APIs for Storage
Awareness and vSphere APIs for I/O Filtering

Module 3: Storage Scalability149

3-38 Learner Objectives

150 Module 3: Storage Scalability

3-39 About vSphere API for Storage Awareness

In vCenter Server, vSphere administrators cannot access the storage capabilities of the storage
array on which their VMs are stored. VMs are provisioned to a storage black box. vSphere
administrators see only a LUN identifier, such as a Network Address Authority ID (NAA ID) or a
T10 identifier.
A storage vendor can use vSphere API for Storage Awareness to provide information about its
storage array.
The storage provider is written by the storage array vendor. The storage provider can exist on
either the storage array processor or on a standalone host. The storage vendor decides.
The storage provider acts as a server in the vSphere environment. vCenter Server connects to the
provider to obtain information about available storage topology, capabilities, and state. The
information is viewed in the vSphere Client.
A storage provider can report information about one or more storage devices. It can support
connections to a single vCenter Server instance or to multiple vCenter Server instances.

Module 3: Storage Scalability151

For information about the vSphere API for Storage Awareness program, see APIs for Storage
Awareness (VASA) Package at https://developercenter.vmware.com/web/dp/other-
programs/storage/vasa.

152 Module 3: Storage Scalability

3-40 Benefits of Storage Providers

A storage provider provides capability information about the storage configuration, status, and
storage data services in your environment. vSphere Storage DRS uses the information supplied by
a storage provider to make VM placement and migration decisions that are compatible with the
storage system.
A storage provider provides capability information:

• Storage capabilities: Information about storage characteristics and services, such as the
number and type of spindles for a volume, the I/O operations in megabytes per second, the
type of compression that is used, and whether thick-provisioned format is used.

• Storage status: Status of storage entities and alarms and events for configuration changes.

Module 3: Storage Scalability153

3-41 Comparison of Storage Provider Versions

154 Module 3: Storage Scalability

3-42 Storage Provider Categories

Module 3: Storage Scalability155

3-43 Registering a Storage Provider

From the Storage Providers pane, you can register a storage provider. All system storage
capabilities that are presented by the storage provider appear in the vSphere Client.
To register a storage provider, the storage vendor provides a URL, a login account, and a
password.
Users log in to the provider to get array information. vCenter Server must trust the provider host,
so a security certificate from the provider must be installed on the vCenter Server system.
For information about how to Register Storage Providers see
https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.storage.doc/GUID-
B50E19B8-C1AD-4FE1-AD90-4DD6D29831FE.html.

156 Module 3: Storage Scalability

3-44 About vSphere APIs for I/O Filtering

I/O filters can gain direct access to the VM I/O path. You can enable the I/O filter for an
individual virtual disk.
VMware offers certain categories of I/O filters. In addition, third-party vendors can create the I/O
filters. Typically, these I/O filters are distributed as packages that provide an installer for
deploying the filter components on vCenter Server and ESXi host clusters.
I/O filters can support all datastore types:

• VMFS

• NFS 3

• NFS 4.1

• vSphere Virtual Volumes

• vSAN

Module 3: Storage Scalability157

3-45 Types of I/O Filters

VMware provides certain categories of I/O filters that are installed on your ESXi hosts. In
addition, VMware partners can create the I/O filters through the vSphere APIs for I/O Filtering
developer program. The I/O filters can serve multiple purposes.

158 Module 3: Storage Scalability

3-46 How I/O Filters Work (1)

Filters operate by intercepting data in the VMkernel before the data is sent out to physical storage.
The I/O filter framework exists outside the virtual SCSI layer, which means that offline I/O to the
disk (that is, modifications when a VM is powered off) can also be intercepted. The framework
defines which filters should be applied and in which order.
I/O filters run in the VM’s user world. A world is an execution context that is scheduled on a
processor. A world is like a process in conventional operating systems.

Module 3: Storage Scalability159

3-47 How I/O Filters Work (2)

The order in which I/O passes through these filter categories is predefined. This order, as shown
on the slide, is important, especially if you plan to implement a replication filter and an encryption
filter.
Data passes from the filters in this predefined order before being forwarded to the storage on
which the VM files or objects reside.
You can install several filters from the same category on your ESXi host. However, you can have
only one filter from the same category per virtual disk.
Because the replication filter comes before the encryption filter, the data that is replicated is in
plain text. Additional security measures must be taken to protect these transmissions and the data
on the replication target.

160 Module 3: Storage Scalability

3-48 I/O Filter Providers

After I/O filters are installed on ESXi hosts, the I/O filter framework configures and registers a
storage provider, also called an IOFilter provider, for each host.
The IOFilter provider advertises the new filter capabilities to vCenter Server. You can use these
capabilities to build storage policies.
Storage providers for I/O filtering are software components in vSphere. These providers integrate
with I/O filters and report data service capabilities that are supported by I/O filters to vCenter
Server.
If you use I/O filters provided by third parties, you must install the I/O filters in an ESXi host
cluster. You cannot install the filters on selected hosts. The filter packages are typically distributed
as VIBs, which can include I/O filter daemons, CIM providers, and other associated components.
Typically, to deploy the filters, you run installers provided by vendors.

Module 3: Storage Scalability161

The capabilities populate the VM Storage Policies pane in the vSphere Client and can be
referenced in a VM storage policy. You apply this policy to virtual disks so that the I/O filters can
process I/O for the disks.

162 Module 3: Storage Scalability

3-49 Displaying I/O Filter Provider Information

Module 3: Storage Scalability163

3-50 Activity: I/O Filters (1)

164 Module 3: Storage Scalability

3-51 Activity: I/O Filters (2)

Module 3: Storage Scalability165

3-52 Review of Learner Objectives

166 Module 3: Storage Scalability

3-53 Lesson 4: Storage Policy Based Management Overview

Module 3: Storage Scalability167

3-54 Learner Objectives

168 Module 3: Storage Scalability

3-55 About Storage Policy Based Management (1)

VMware partners and storage vendors can support these storage and data services.
Rather than integrating with each individual vendor or type of storage and data service, storage
policy based management (SPBM) provides a universal framework for many types of storage
entities.
When an application is to be provisioned, vSphere uses the Storage Policy Based Management
module to match the policy requirements against configured storage container capabilities. In this
way, vSphere determines where the VMDK files should exist.

Module 3: Storage Scalability169

3-56 About Storage Policy Based Management (2)

VM storage policies minimize the amount of storage planning that you must do for each VM.
For example, you can use VM storage policies to create basic storage tiers. Datastores with similar
capabilities are tagged to form gold, silver, and bronze tiers.
Redundant, high-performance storage might be tagged as the gold tier. Nonredundant, medium
performance storage might be tagged as the bronze tier.
The capabilities exposed are performance, capacity, protection, and so on.
The capabilities requested are aligned on VM boundaries.
The capabilities fulfilled are datastore provisioned, and the policy is kept with the datastore.

170 Module 3: Storage Scalability

3-57 About VM Storage Policies

VM storage policies are essential to VM provisioning.

You use VM storage policies during the provisioning of a VM to ensure that a VM’s disks are
placed on the storage that is best for its situation.
For example, with VM storage policies, you can ensure that the VM running a critical I/O
intensive database is placed in the gold tier.
Ideally, you want to create the best match of predefined VM storage requirements with available
physical storage properties.

Module 3: Storage Scalability171

3-58 About Storage Policy Rules

Capability-based placement rules describe how VM storage objects are allocated within a
datastore to achieve the required level of service. For example, the rules can list virtual volumes as
a destination and define the maximum recovery point objective for the virtual volume objects.
Storage policy based management finds the virtual volumes datastores that can match the rules and
satisfy the storage requirements of the VM.
Data service rules activate specific data services. Storage systems or other entities can provide
these services. They can also be installed on your hosts and vCenter Server. You include the data
service rules in the storage policy components.
Tag-based placement rules reference datastore tags. You can use the tag-based rules to fine-tune
your VM placement request. For example, you can exclude datastores with a Palo Alto tag from
the list of your virtual volumes datastores.

172 Module 3: Storage Scalability

3-59 Storage Policy Example (1)

Each array vendor has an identified namespace. The namespace contains the storage container
identifiers and assigned capabilities. The capabilities appear as options in the client interface. In
this example, the vSphere Client is used.

Module 3: Storage Scalability173

3-60 Storage Policy Example (2)

174 Module 3: Storage Scalability

3-61 About Storage Policy Components

The service provider can be a storage system, an I/O filter, or another entity.

Module 3: Storage Scalability175

3-62 Storage Policy Component Example

You must add the component to the storage policy and assign the policy to the VM. You cannot
assign a component directly to a VM or virtual disk.
You can define the policy components in advance and associate them with multiple VM storage
policies.

176 Module 3: Storage Scalability

3-63 Activity: Storage Policy Rules (1)

Module 3: Storage Scalability177

3-64 Activity: Storage Policy Rules (2)

178 Module 3: Storage Scalability

3-65 Review of Learner Objectives

Module 3: Storage Scalability179

3-66 Lesson 5: Using VM Storage Policies

180 Module 3: Storage Scalability

3-67 Learner Objectives

Module 3: Storage Scalability181

3-68 Creating and Managing Storage Policies

182 Module 3: Storage Scalability

3-69 Step 1: Configuring Storage to Be Used for Storage
Policies

Module 3: Storage Scalability183

3-70 Step 2: Creating Storage Policy Components

184 Module 3: Storage Scalability

3-71 Step 3: Creating VM Storage Policies

Module 3: Storage Scalability185

3-72 Example: Creating a Silver Tier Storage Policy (1)

186 Module 3: Storage Scalability

3-73 Example: Creating a Silver Tier Storage Policy (2)

Module 3: Storage Scalability187

3-74 Example: Creating a Silver Tier Storage Policy (3)

188 Module 3: Storage Scalability

3-75 Example: Creating a Silver Tier Storage Policy (4)

Module 3: Storage Scalability189

3-76 Step 4: Applying the Storage Policy to the VM

When you select a VM storage policy, the vSphere Client shows the datastores that are compatible
with the capabilities of that policy. You can select a datastore or a datastore cluster from the list. If
you select a datastore that does not match the storage policy, the vSphere Client shows that the
VM is using noncompliant storage.
When a VM storage policy is selected, the datastores are divided into two categories: compatible
and incompatible.
You can select datastores outside of the VM storage policy, but these datastores put the VM into a
noncompliant state.
By using VM storage policies, you can easily see whether storage is compatible or incompatible.
You do not need to ask the vSAN administrator, or refer to a spreadsheet of NAA IDs, every time
you deploy a VM.

190 Module 3: Storage Scalability

3-77 Step 5: Checking Compliance for the VM Storage
Policy

You can use VM storage policies during the ongoing management of VMs.
You can periodically check whether a VM is migrated to or created on inappropriate storage,
potentially making it noncompliant.
You can also use storage information to monitor the health and use of storage and report when the
VM’s storage is noncompliant.

Module 3: Storage Scalability191

3-78 Lab 6: Using Policy Based Storage

192 Module 3: Storage Scalability

3-79 Review of Learner Objectives

Module 3: Storage Scalability193

3-80 Lesson 6: Storage Policies for vSAN

194 Module 3: Storage Scalability

3-81 Learner Objectives

Module 3: Storage Scalability195

3-82 About vSAN (1)

In a vSAN environment, ESXi hosts are configured to form a vSAN cluster. All ESXi hosts
communicate through a dedicated network. At least three hosts in the cluster must have local
storage devices. Although it is not a best practice, hosts without local storage can share their
compute resources and use the clustered storage resources.
vSAN datastores help administrators use software-defined storage in the following ways:

• Storage policy per VM architecture: With multiple policies per datastore, each VM can have
different storage configurations in terms of availability, performance, and capacity.

• vSphere and vCenter Server integration: vSAN capability is built into the VMkernel and
requires no appliance.

• Scale-out storage: Up to 64 ESXi hosts can be in a cluster. You scale out by populating new
hosts in the cluster.

• Built-in resiliency: A default policy mirrors all objects for VMs that are configured for vSAN.

196 Module 3: Storage Scalability

3-83 About vSAN (2)

All-flash configurations can use features that are not available in hybrid configurations:

• Erasure coding: This feature provides the same levels of redundancy as mirroring (RAID 1)
while consuming less storage capacity. Use of erasure coding reduces capacity consumption
by as much as 50 percent versus mirroring at the same fault tolerance level.

• Deduplication and compression are space-saving features that can reduce the amount of
storage consumption by as much as seven times. Deduplication and compression results vary
based on the types of data stored on vSAN storage.

Module 3: Storage Scalability197

3-84 vSAN Disk Architecture

In the vSAN cluster, the local storage components of the ESXi hosts are combined to create a
vSAN datastore. Only one datastore can be created for each vSAN cluster.
Local storage is combined on each host to form a disk group. A disk group is a vSAN
management construct that includes one cache device and one to seven capacity devices.
At most, a vSAN host can have 5 disk groups, each containing up to 7 capacity devices, resulting
in a maximum of 35 capacity devices for each host.

198 Module 3: Storage Scalability

3-85 Object-Based Storage

The object store file system manages data as objects. Each object includes its own data, part of the
metadata, and a unique ID. With this unique ID, the object can be globally addressed by more than
the filename and path. The use of objects provides a detailed level of configuration on the object
level, for example, RAID type or disk use at a level higher than the physical disk blocks.
When you provision a VM on a vSAN datastore, a set of objects is created. These objects are of
the following types:

• VM home namespace: Stores the VM metadata (configuration files)

• VMDK: VM disk

• Snapshot delta: Created when snapshots of the VM are taken

• VM swap: VM swap file, which is created when the VM is powered on

• VM memory: A VM’s memory state when the VM is suspended or when a snapshot is taken
of a VM and its memory state is preserved

Module 3: Storage Scalability199

3-86 vSAN Storage Providers

vSAN storage providers report a set of underlying storage capabilities to vCenter Server. They
also communicate with the vSAN layer to report the storage requirements of the VMs.

200 Module 3: Storage Scalability

3-87 vSAN Storage Policies

VMs that are deployed to vSAN datastores are assigned at least one VM storage policy. If a
storage policy is not assigned to the VM that is provisioned, a default storage policy is applied to
the VM from the datastore. If a custom policy is not applied to the vSAN datastore, the vSAN
default storage policy is used.

Module 3: Storage Scalability201

3-88 Creating vSAN Storage Policies

202 Module 3: Storage Scalability

3-89 vSAN Capabilities (1)

Module 3: Storage Scalability203

3-90 vSAN Capabilities (2)

For more information about vSAN advanced options, see Administering VMware vSAN at
https://docs.vmware.com/en/VMware-vSphere/7.0/vsan-70-administration-guide.pdf.

204 Module 3: Storage Scalability

3-91 Lab 7: Creating vSAN Storage Policies

Module 3: Storage Scalability205

3-92 Review of Learner Objectives

206 Module 3: Storage Scalability

3-93 Lesson 7: Storage Policies for vSphere Virtual Volumes

Module 3: Storage Scalability207

3-94 Learner Objectives

208 Module 3: Storage Scalability

3-95 About vSphere Virtual Volumes

vSphere Virtual Volumes functionality changes the storage management paradigm from managing
space inside datastores to managing abstract storage objects handled by storage arrays.
Traditionally, vSphere storage management uses a datastore-centric approach. With this approach,
storage administrators and vSphere administrators discuss in advance the underlying storage
requirements for VMs. The storage administrator sets up LUNs or NFS shares and presents them
to ESXi hosts. The vSphere administrator creates datastores based on LUNs or NFS shares and
uses these datastores as VM storage. Typically, the datastore is the lowest level at which data
management occurs, from a storage perspective. However, a single datastore contains multiple
VMs that might have different requirements.
With the traditional approach, differentiation per VM is difficult.
With vSphere Virtual Volumes, you can differentiate VM services per application by offering a
new approach to storage management.
vSphere Virtual Volumes supports Fibre Channel (FC), FCoE, iSCSI, and NFS.

Module 3: Storage Scalability209

3-96 vSphere Virtual Volumes Architecture

The architecture of vSphere Virtual Volumes exists on the storage itself and on various
components in the vSphere environment.

210 Module 3: Storage Scalability

3-97 vSphere Virtual Volumes Storage Providers

The vSphere Virtual Volumes storage provider delivers storage information from the underlying
storage container so that storage container capabilities appear in vCenter Server and vSphere
Client.
A vSphere Virtual Volumes storage provider is implemented through vSphere API for Storage
Awareness and is used to manage all aspects of vSphere Virtual Volumes storage. The storage
provider integrates with the Storage Monitoring Service, shipped with vSphere, to communicate
with vCenter Server and ESXi hosts.
The storage provider communicates VM storage requirements, which you define in a storage
policy, to the storage layer. In this way, a virtual volume created in the storage layer meets the
requirements outlined in the policy.
Typically, vendors supply storage providers that can integrate with vSphere and provide support to
virtual volumes. Every storage provider must be certified by VMware and properly deployed. For
information about deploying the vSphere Virtual Volumes storage provider, contact your storage
vendor.

Module 3: Storage Scalability211

For information about vSphere Virtual Volumes storage partners, see the vSphere Virtual
Volumes product page at http://www.vmware.com/products/vsphere/features/virtual-
volumes.html.

212 Module 3: Storage Scalability

3-98 Protocol Endpoints

Although storage systems manage all aspects of virtual volumes, ESXi hosts have no direct access
to virtual volumes on the storage side. Instead, ESXi hosts use a logical I/O proxy, called the
protocol endpoint, to communicate with virtual volumes and virtual disk files that virtual volumes
encapsulate.
When the SCSI-based protocol is used, the protocol endpoint represents a LUN defined by a T10-
based LUN World Wide Name. For the NFS protocol, the protocol endpoint is a mount point, such
as an IP address or a DNS name and a share name.
When a VM on the host performs an I/O operation, the protocol endpoint directs the I/O to the
appropriate virtual volume. Typically, a storage system requires a few protocol endpoints.
A storage administrator configures protocol endpoints. Protocol endpoints are part of physical
storage and are exported, with associated storage containers, by the storage system through a
storage provider. After you map a storage container to a virtual volumes datastore, protocol
endpoints are discovered by ESXi and appear in the vSphere Client. Protocol endpoints can also
be discovered during a storage rescan.

Module 3: Storage Scalability213

3-99 Storage Containers and vSphere Virtual Volumes
Datastores

Unlike traditional LUN-based and NFS-based vSphere storage, vSphere Virtual Volumes
functionality does not require preconfigured volumes on a storage side. Instead, vSphere Virtual
Volumes uses a storage container, which is a pool of raw storage capacity or an aggregation of
storage capabilities that a storage system can provide to virtual volumes.
Typically, a storage administrator on the storage side defines storage containers. The number of
storage containers and their capacity depend on the vendor-specific implementation, but each
storage system requires at least one container.
After vCenter Server discovers storage containers exported by storage systems, you must map a
storage container to a virtual volumes datastore. The virtual volumes datastore that you create
corresponds directly to the specific storage container and becomes the container’s representation
in vCenter Server and the vSphere Client.

214 Module 3: Storage Scalability

3-100 Virtual Volumes

Virtual volumes are encapsulations of VM files, virtual disks, and their derivatives.
Virtual volumes are exported as objects by a compliant storage system and are managed by
hardware on the storage side. Typically, a unique GUID identifies a virtual volume.

Module 3: Storage Scalability215

3-101 Mapping VM Files to Virtual Volumes

A data virtual volume corresponds directly to each virtual disk (.vmdk) file. Like virtual disk files
on traditional datastores, virtual volumes are presented to VMs as SCSI disks.
A configuration virtual volume, or a home directory, represents a small directory that contains
metadata files for a VM. Metadata files include a VMX file, descriptor files for virtual disks, log
files, and so on. The configuration virtual volume is formatted with a file system. When ESXi uses
the SCSI protocol to connect to storage, configuration virtual volumes are formatted with VMFS.
With the NFS protocol, configuration virtual volumes are presented as an NFS directory.
Additional virtual volumes can be created for other VM components and virtual disk derivatives,
such as clones, snapshots, and replicas. These virtual volumes include a swap virtual volume to
hold VM swap files and a virtual memory volume to hold the contents of VM memory for a
snapshot.

216 Module 3: Storage Scalability

3-102 vSphere Virtual Volumes Replication

vSphere Virtual Volumes replication has the following properties:

• Available with vSphere API for Storage Awareness version 3.0

• Available for ESXi hosts, version 6.5 and later

• Requires the storage array to be replication-capable

Implementation of vSphere Virtual Volumes replication depends on your array and might vary
with different storage vendors. The following requirements generally apply to all vendors:

• The storage arrays that you use to implement replication must be compatible with vSphere
Virtual Volumes.

• The arrays must integrate with the version of the storage (VASA) provider that is compatible
with vSphere Virtual Volumes replication.

Module 3: Storage Scalability217

• The storage arrays must be replication-capable and configured to use vendor-provided
replication mechanisms. Typical configurations usually involve one or two replication targets.
Any required configurations, such as pairing of the replicated site and the target site, must
also be performed by the storage system.

• When applicable, replication groups and fault domains for virtual volumes must be
preconfigured by the storage system.

218 Module 3: Storage Scalability

3-103 Virtual Volume Storage Policies

Module 3: Storage Scalability219

3-104 Virtual Volume Storage Policy: Examples

220 Module 3: Storage Scalability

3-105 Creating Virtual Volumes Storage Policies (1)

Module 3: Storage Scalability221

3-106 Creating Virtual Volumes Storage Policies (2)

222 Module 3: Storage Scalability

3-107 Activity: Displaying Storage Capabilities (1)

Module 3: Storage Scalability223

3-108 Activity: Displaying Storage Capabilities (2)

224 Module 3: Storage Scalability

3-109 Review of Learner Objectives

Module 3: Storage Scalability225

3-110 Lesson 8: Storage I/O Control

226 Module 3: Storage Scalability

3-111 Learner Objectives

Module 3: Storage Scalability227

3-112 About Storage I/O Control

With Storage I/O Control, you can balance I/O load in a datastore cluster that is enabled for
vSphere Storage DRS.
Storage I/O Control extends the constructs of shares, limits, and reservations to handle storage I/O
resources. Storage I/O Control is a proportional-share, I/O operations per second (IOPS) scheduler
that, under contention, throttles IOPS. You can control the amount of storage I/O that is allocated
to VMs during periods of I/O congestion. Controlling storage I/O ensures that more important
VMs get preference over less important VMs for I/O resource allocation.
You can use Storage I/O Control with or without vSphere Storage DRS. Two thresholds exist: one
for Storage I/O Control and one for vSphere Storage DRS. For vSphere Storage DRS, Storage I/O
Control gathers latency statistics for an ESXi host, sent to vCenter Server, and stored in the
vCenter Server database. With these statistics, vSphere Storage DRS can decide whether a VM
should be migrated to another datastore.
For more information, see Managing Storage I/O Resources at
https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.resmgmt.doc/GUID-
7686FEC3-1FAC-4DA7-B698-B808C44E5E96.html.

228 Module 3: Storage Scalability

3-113 Enabling Storage I/O Control (1)

Storage I/O Control provides quality-of-service capabilities for storage I/O in the form of I/O
shares, limits, and reservations that are enforced across all VMs accessing a datastore, regardless
of which host they are running on.
When you enable Storage I/O Control on a datastore, ESXi begins to monitor the device latency
that hosts observe when communicating with that datastore. When device latency exceeds a
threshold, the datastore is considered to be congested, and each VM that accesses that datastore is
allocated I/O resources in proportion to its shares.

Module 3: Storage Scalability229

3-114 Enabling Storage I/O Control (2)

When you allocate storage I/O resources, you can limit the IOPS that are allowed for a VM. By
default, the number of IOPS that are allowed for a VM is unlimited. If the limit that you want to
set for a VM is in megabytes per second instead of IOPS, you can convert megabytes per second
into IOPS based on the typical I/O size for that VM. For example, a backup application has a
typical I/O size of 64 KB. To restrict a backup application to 10 MB per second, set a limit of 160
IOPS: 10 MB per second divided by 64 KB I/O size = 160 IOPS.
VMs VM1 and VM2 are running an I/O load generator called Iometer.
Each VM is running on a different host, but they are running the same type of workload, which is
16 KB random reads. The shares of VM2 are set to twice as many shares as VM1, which implies
that VM2 is more important than VM1. With Storage I/O Control disabled, the IOPS and the I/O
latency that each VM achieves are identical. However, with Storage I/O Control enabled, the IOPS
achieved by the VM with more shares (VM2) is greater than the IOPS of VM1. The example
assumes that each VM is running enough load to cause a bottleneck on the datastore.

230 Module 3: Storage Scalability

3-115 Monitoring Latency

Storage I/O Control automatically determines the optimal latency threshold by using injector-
based models. However, you can also override this threshold by setting a specific latency value.
This default latency setting might be fine for some storage devices, but other devices might reach
their latency threshold well before or after the default setting is reached.
For example, solid-state drives (SSDs) typically reach their contention point sooner than the
default setting protects against. Because not all devices are equal, the injector model is the
preferred option.

Module 3: Storage Scalability231

3-116 Automatic Threshold Detection

The I/O injector model determines the peak throughput of a datastore. The resulting peak
throughput measurement can be used to determine the peak latency of a datastore.

232 Module 3: Storage Scalability

3-117 Storage Policy Components for Storage I/O Control

Module 3: Storage Scalability233

3-118 Configuring Storage I/O Control

234 Module 3: Storage Scalability

3-119 Storage I/O Control Requirements

Automated storage tiering is the ability of an array (or a group of arrays) to migrate LUNs, virtual
volumes, or parts of LUNs or virtual volumes to different types of storage media (SSD, Fibre
Channel, SAS, SATA, and so on) based on user-set policies and current I/O patterns. No special
certification is required for arrays that do not have these automatic migration or tiering features,
including those features that allow you to manually migrate data between different types of storage
media.
To verify that your automated tiered storage array is certified as compatible with Storage I/O
Control, see the VMware Compatibility Guide at http://www.vmware.com/resources/compatibility.

Module 3: Storage Scalability235

3-120 Activity: Storage I/O Control (1)

236 Module 3: Storage Scalability

3-121 Activity: Storage I/O Control (2)

Module 3: Storage Scalability237

3-122 Review of Learner Objectives

238 Module 3: Storage Scalability

3-123 Lesson 9: Datastore Clusters and vSphere Storage
DRS

Module 3: Storage Scalability239

3-124 Learner Objectives

240 Module 3: Storage Scalability

3-125 About Datastore Clusters

The datastore cluster serves as a container or folder. You can store datastores in the container, but
the datastores work as separate entities.
A datastore cluster that is enabled for vSphere Storage DRS is a collection of datastores that work
as a single unit.

Module 3: Storage Scalability241

3-126 Datastore Cluster Requirements

A datastore cluster can contain a mix of datastores of different sizes and I/O capacities, as well as
datastores from different arrays and vendors. However, LUNs with different performance
characteristics can cause performance problems.
vSphere Storage DRS cannot move VMs between NFS and VMFS datastores.

242 Module 3: Storage Scalability

3-127 Host Cluster to Datastore Cluster Relationships

Host clusters and datastore clusters can coexist in the virtual infrastructure. A host cluster is a
vSphere DRS or a vSphere HA cluster.
Load balancing by vSphere DRS and vSphere Storage DRS can occur at the same time. vSphere
DRS balances VMs across hosts based on CPU usage, memory usage, and network usage (the
latter if Network I/O Control version 3 is used). vSphere Storage DRS load balances VMs across
storage, based on storage capacity and IOPS.
A standalone host (one that is not part of a host cluster) can also use a datastore cluster.

Module 3: Storage Scalability243

3-128 vSphere Storage DRS Overview

vSphere Storage DRS manages the placement of VMs in a datastore cluster, based on the space
usage of the datastores. It tries to keep usage as even as possible across datastores in the datastore
cluster.
vSphere Storage DRS uses vSphere Storage vMotion to migrate VMs to maintain the balance
across datastores.
Optionally, the user can configure vSphere Storage DRS to balance I/O latency across members of
the datastore cluster as a way to help mitigate performance problems that are caused by I/O
latency.
vSphere Storage DRS can be set up to work in either manual mode or fully automated mode:

• Manual mode presents migration and placement recommendations to the user, but nothing is
executed until the user accepts the recommendation.

• Fully automated mode automatically handles initial placement and migrations based on
runtime.

244 Module 3: Storage Scalability

3-129 Initial Disk Placement

When a VM is created, cloned, or migrated, and the user selects a datastore cluster, vSphere
Storage DRS selects a member datastore in the datastore cluster based on storage use. vSphere
Storage DRS tries to keep the member datastores evenly used.

Module 3: Storage Scalability245

3-130 vSphere Storage DRS Affinity Rules

By default, all of a VM's disks can be on the same datastore.

A user might want the virtual disks on different datastores. For example, a user can place a system
disk on one datastore and the data disks on another. In this case, the user can set up a VMDK anti-
affinity rule, which keeps a VM’s virtual disks on separate datastores.
VM anti-affinity rules keep VMs on separate datastores. This rule is useful when redundant VMs
must always be available.

246 Module 3: Storage Scalability

3-131 Storage Migration Recommendations

Storage migration recommendations work as follows:

• vSphere Storage DRS provides as many recommendations as necessary to balance the space
and, optionally, the IOPS resources of the datastore cluster.

• Reasons for migration recommendations include balancing space usage in the datastore,
reducing datastore I/O latency, and balancing datastore IOPS load.

• vSphere Storage DRS can also make mandatory recommendations based on the following
conditions:
– A datastore is out of space.
– vSphere Storage DRS anti-affinity rules or affinity rules are being violated.
– A datastore is entering maintenance mode.
To achieve optimal balance of VMs across datastores in the datastore cluster, vSphere Storage
DRS considers moving powered-off VMs to other datastores.

Module 3: Storage Scalability247

3-132 Datastore Correlation Detector

The purpose of datastore correlation is to help vSphere Storage DRS in determining where to
move a VM. For example, you gain little advantage by moving a VM from one datastore to
another if both datastores are supported by the same set of physical spindles on the array.
The datastore correlation detector uses the I/O injector to determine whether a source and a
destination datastore are using the same back-end spindles.
The datastore correlation detector works by monitoring the load on one datastore and monitoring
the latency on another. If latency increases on other datastores when a load is placed on one
datastore, the datastores are correlated.
The datastore correlation detector can also be used for anti-affinity rules, ensuring that VMs and
virtual disks are not only kept on separate datastores but also kept on different spindles on the
back end.
Correlation is determined by a long-running background process.
Datastore correlation is enabled by default.

248 Module 3: Storage Scalability

3-133 Configuring vSphere Storage DRS Runtime Settings

• When you create datastore cluster, you select or deselect the Enable I/O metric for SDRS
recommendations check box to enable or disable IOPS metrics use in recommendations or
automated migrations.

• When you enable I/O load balancing, Storage I/O Control is enabled for all the datastores in
the datastore cluster if it is not already enabled for the cluster.

Module 3: Storage Scalability249

• When this option is deselected, you disable the following functions:
- IOPS load balancing among datastores in the datastore cluster.
- Initial placement for virtual disks based on the IOPS metric. Space is the only
consideration when placement and balancing recommendations are made.

• vSphere Storage DRS runtime settings include the following options:

- I/O latency threshold: Indicates the minimum I/O latency for each datastore below which
I/O load-balancing moves are not considered. This setting is applicable only if the
Enable I/O metric for SDRS recommendations check box is selected.
- Space threshold: Determines the minimum levels of consumed space and free space for
each datastore that is the threshold for action.

250 Module 3: Storage Scalability

3-134 vSphere Storage DRS Maintenance Mode

m
co
y.
g sk
lo
2 .b
01

With vSphere Storage DRS, you can place a datastore in maintenance mode. vSphere Storage
DRS maintenance mode is available to datastores in a datastore cluster that is enabled for vSphere
e2

Storage DRS. Standalone datastores cannot be placed in maintenance mode.

cs

The datastore does not enter maintenance mode until all files on the datastore are moved. You
must manually move these files off the datastore so that the datastore can enter vSphere Storage
m

DRS maintenance mode.

If the datastore cluster is placed in fully automated mode, VMs are automatically migrated to other
datastores.
If the datastore cluster is placed in manual mode, migration recommendations appear in the
vSphere Client. The virtual disks cannot be moved until the recommendations are accepted.

Module 3: Storage Scalability251

3-135 Backups and vSphere Storage DRS

You can configure scheduled tasks to change vSphere Storage DRS behavior.
You can use scheduled tasks to change the vSphere Storage DRS configuration of the datastore
cluster to match enterprise activity.
For example, if the datastore cluster is configured to perform migrations based on I/O latency, you
might disable the use of I/O metrics by vSphere Storage DRS during the backup window.
You can reenable I/O metrics use after the backup window closes.

252 Module 3: Storage Scalability

3-136 Activity: vSphere Storage DRS (1)

Module 3: Storage Scalability253

3-137 Activity: vSphere Storage DRS (2)

254 Module 3: Storage Scalability

3-138 Review of Learner Objectives

Module 3: Storage Scalability255

3-139 Key Points

256 Module 3: Storage Scalability

Module 4
Host and Management Scalability

Module 4: Host and Management Scalability 257

4-2 Importance

258 Module 4: Host and Management Scalability

4-3 Module Lessons

Module 4: Host and Management Scalability 259

4-4 Lesson 1: VMware Certificate Management

260 Module 4: Host and Management Scalability

4-5 Learner Objectives

Module 4: Host and Management Scalability 261

4-6 Certificate Use Case

262 Module 4: Host and Management Scalability

4-7 About Certificate Authorities

Public key cryptographic systems use pairs of keys. The data is encrypted using a private key,
which is known only to the owner. The data can be decrypted only by using a public key, which
might be widely disseminated.
CAs are important in public key infrastructure systems. When a Secure Sockets Layer (SSL) or
Transport Layer Security (TLS) client connects to a server, the server sends its public key to the
client to authenticate the server. However, the server cannot send its public key in plain text
because such text is easily corruptible by attackers who can inject themselves into the
communication.
Instead, the server sends an X.509 certificate to the client. The X.509 certificate contains the
server’s name (the subject), its public key, and other information. X.509 certificates are signed by
a trusted CA, which means that they are encrypted with the CA’s private key.
The client trusts the CA because the client already has the CA’s public key, which is either
preinstalled or manually installed by an administrator. For example, browsers such as Safari,
Firefox, Internet Explorer, and Chrome have public keys of the most common CAs preinstalled.

Module 4: Host and Management Scalability 263

SSL or TLS use is not limited to websites and browsers, although these uses are common and well
known. Any client or server software solution that requires secure authentication of the parties can
use SSL or TLS in the manner described here.

264 Module 4: Host and Management Scalability

4-8 About VMware CA

By default, VMware CA acts as a root CA. It issues and validates certificates for vSphere
components such as ESXi hosts and solutions. VMware CA can handle all certificate management
for you. VMware CA provisions vCenter Server components and ESXi hosts with certificates that
use VMware CA as the root certificate authority. If you upgrade to vSphere 6.x from an earlier
version of vSphere, all self-signed certificates are replaced with certificates that are signed by
VMware CA.

Module 4: Host and Management Scalability 265

4-9 About the VMware Endpoint Certificate Store

ESXi certificates are stored locally on each host, not in the VMware Endpoint Certificate Store
(VECS).
VECS runs as part of the VMware Authentication Framework Daemon on vCenter Server. It holds
the keystores that contain the certificates and keys. With support for multiple keystores, the key
manager can manage multiple hosts that require different access credentials. Different certificates
for different host names can be served from the same IP address. Therefore, multiple secure
(HTTPS) websites (or any other service over TLS) can be served by the same IP address without
requiring all those sites to use the same certificate.
The vCenter Single Sign-On service stores the token-signing certificate and its SSL certificate on
disk. You can change the token-signing certificate from the vSphere Client.

266 Module 4: Host and Management Scalability

4-10 vSphere Certificate Types

vSphere components use SSL to communicate securely with one another and with ESXi hosts.
SSL communications ensure data confidentiality and integrity. Data is protected and cannot be
modified in transit without detection.
vCenter Server services, such as the vSphere Client, also use certificates for their initial
authentication to vCenter Single Sign-On. vCenter Single Sign-On provisions each component
with a SAML token that the component uses for authentication from that point on.
VMware products use standard X.509 version 3 (X.509v3) certificates to encrypt session
information that is sent over SSL between components.
Solution user certificates continue to be used, but they might not be displayed in the Certificate
Manager client.

Module 4: Host and Management Scalability 267

4-11 Chain of Trust (1)

268 Module 4: Host and Management Scalability

4-12 Chain of Trust (2)

The client trusts the server because the Certificate Authority (CA) issued the server its certificate
and the client trusts the CA.

Module 4: Host and Management Scalability 269

4-13 Chain of Trust (3)

The client trusts the server because the Intermediate Certificate Authority (ICA) issued the server
its certificate, the CA issued the ICA its certificate, and the client trusts the CA.

270 Module 4: Host and Management Scalability

4-14 Activity: ESXi Certificates (1)

Module 4: Host and Management Scalability 271

4-15 Activity: ESXi Certificates (2)

272 Module 4: Host and Management Scalability

4-16 VMware CA Modes

By default, VMware CA is configured to act as a root CA that can use its self-signed root
certificate to issue and sign other certificates. However, you can also configure VMware CA as a
subordinate CA. In subordinate CA mode, VMware CA is an intermediate CA in the chain of trust
that continues up to an enterprise CA or an external third-party CA.
If you do not want to use VMware CA, you can bypass VMware CA and install custom
certificates on all your systems.
Use subordinate CA mode if you want to issue and install your own certificates. In this situation,
you must issue a certificate for every component. Certificate management is your responsibility.
All your custom certificates (except for host certificates) must be installed in VECS.

Module 4: Host and Management Scalability 273

4-17 VMware CA: Root CA Mode

VMware CA uses a self-signed root certificate. It issues certificates to vCenter Server, ESXi, and
so on, and manages these certificates. These certificates have a chain of trust that stops at the
VMware CA root certificate. VMware CA is not a general-purpose CA, and its use is limited to
VMware components.

274 Module 4: Host and Management Scalability

4-18 Replacing the VMware CA Certificate in Root CA Mode

You are not required to replace the certificates that were issued and signed by the previous
VMware CA root CA certificate. When you replace the VMware CA root CA certificate, it does
not automatically invalidate the previous certificate. It also does not remove the old certificate, so
the old certificate remains valid in your vSphere environment.
However, when you want to replace the root CA certificate, you typically also want to replace all
your solution user certificates, your ESXi certificates, and so on, so that they are signed by the
current VMware CA root CA certificate. You can replace all these certificates at once by using the
vSphere Certificate Manager tool.

Module 4: Host and Management Scalability 275

4-19 VMware CA: Subordinate CA Mode

In subordinate CA mode, you replace the VMware CA root certificate with a third-party CA-
signed certificate that includes VMware CA in the certificate chain. The replacement certificate
can be signed by your enterprise CA or by a third-party, external CA.
All certificates that VMware CA generates include the full chain. You can replace existing
certificates with newly generated certificates, although this step is not required. The original
certificate is still valid if it has not expired or been revoked, and you can continue to use the
certificates that were signed with the original certificate.
The subordinate CA mode combines the security of a third-party, CA-signed certificate with the
convenience of the automated certificate management in VMware CA.

276 Module 4: Host and Management Scalability

4-20 Replacing the VMware CA Certificate with a
Subordinate CA Certificate

To configure your VMware CA as a subordinate CA, you must import an enterprise or third-party
CA certificate into VMware CA. You must generate a certificate signing request (CSR) that you
can present to your enterprise or third-party CA so that the CA can create an appropriate certificate
for you.

Module 4: Host and Management Scalability 277

4-21 VMware CA: Hybrid Mode

For more information about replacing certificates, see Certificate Replacement Overview at
https://docs.vmware.com/en/VMware-
vSphere/7.0/com.vmware.vsphere.authentication.doc/GUID-4469A6D3-048A-471C-9CB4-
518A15EA2AC0.html

278 Module 4: Host and Management Scalability

4-22 Hybrid Mode: Replacing a vCenter Server Machine
Server Certificate with a Custom Certificate

Module 4: Host and Management Scalability 279

4-23 Certificate Replacement Options

With the introduction of VMware CA in vSphere 6.0, managing your vSphere certificates is much
easier. The situations described on the slide are the three most common ones.

280 Module 4: Host and Management Scalability

4-24 Certificate Manager

With vSphere 7, you can use the vSphere Client to replace the certificates in your vSphere
environment and to reset all vSphere certificates. You can also use the vSphere Certificate
Manager.
To start the vSphere Certificate Manager, log in to your vCenter Server system and run the
commands at the command prompt:
/usr/lib/vmware-vmca/bin/certificate-manager

For more information about certificate management, see Certificate Management Overview at
https://docs.vmware.com/en/VMware-
vSphere/7.0/com.vmware.vsphere.authentication.doc/GUID-3D0DE463-D0EC-442E-B524-
64759D063E25.html.

Module 4: Host and Management Scalability 281

4-25 Replacing All VMware CA Certificates with Custom
Certificates

Replacing VMware CA issued certificates with custom enterprise or third-party certificates is not
recommended. However, your own site security policies might dictate that you use only a specific
CA for all your site’s systems.
When you replace the VMware CA certificates with custom certificates, you increase your
management tasks, which can increase the risk of errors that lead to security vulnerabilities. You
are responsible for all certificate management tasks. Because you do not use VMware CA, you
cannot rely on VMware CA to manage your certificates for you.

282 Module 4: Host and Management Scalability

4-26 ESXi Certificate Replacement Options

Module 4: Host and Management Scalability 283

4-27 Renewing or Refreshing ESXi Certificates

To renew or refresh CA certificates from the vSphere Client:

1 Browse to the host in the vSphere Client inventory.
2 Click the Manage tab and click Settings.
3 Under System, click Certificate.
You can view detailed information about the selected host's certificate.
4 Choose to Renew or Refresh.
5 Click Yes to confirm.

284 Module 4: Host and Management Scalability

4-28 VMware CA Availability

VMware CA is embedded with vCenter Server. When VMware CA is not available, you lose the
ability to manage your certificates. However, issued certificates are still valid. Your ongoing
vSphere services or functions should not be disrupted.

Module 4: Host and Management Scalability 285

4-29 Lab 8: Working with Certificates

286 Module 4: Host and Management Scalability

4-30 Review of Learner Objectives

Module 4: Host and Management Scalability 287

4-31 Lesson 2: vCenter Server Identity Federation

288 Module 4: Host and Management Scalability

4-32 Learner Objectives

Module 4: Host and Management Scalability 289

4-33 About vCenter Server IDP Federation for ADFS

290 Module 4: Host and Management Scalability

4-34 vCenter Server IDP Federation Use Cases

Module 4: Host and Management Scalability 291

4-35 vCenter Server IDP Federation Components

292 Module 4: Host and Management Scalability

4-36 Prerequisites for vCenter Server IDP Federation

vCenter Server supports only one external IDP (one ADFS source) and the local OS or
vsphere.local identity source.
You cannot use multiple external IDPs.
The vCenter Server IDP federation uses OpenID Connect (OIDC) for user login to vCenter Server.

Module 4: Host and Management Scalability 293

4-37 User Login with the vCenter Server Built-In Identity
Provider

Before using vCenter Server IDP Federation for ADFS, this is how the vCenter Server, vCenter
Server Single Sign-On Service, and AD would interact during the login workflow:
1. The user logs in to vCenter Server on the usual vCenter Server landing page.
2. vCenter Server redirects the authentication request to the vCenter Single Sign on Service
3. If needed, the vCenter Single Sign on Service prompts the user to log in with Active Directory
credentials
4. The Single Sign On service authenticates the user with Active Directory
5. The vCenter Single Sign-On service signs all tokens with a signing certificate, and stores the
token signing certificate on disk. The certificate for the service itself is also stored on disk.
6. vCenter Server uses the token to log in the user.

294 Module 4: Host and Management Scalability

4-38 Configuring vCenter Server IDP Federation in vSphere
(1)

Module 4: Host and Management Scalability 295

4-39 Configuring vCenter Server IDP Federation in vSphere
(2)

For more information about configuring vCenter Server IDP federation for vCenter Server, see the
VMware knowledge base article at <link TBD.>

296 Module 4: Host and Management Scalability

4-40 Configuring vCenter Server IDP Federation in vSphere
(3)

Module 4: Host and Management Scalability 297

4-41 Configuring vCenter Server IDP Federation in vSphere
(4)

298 Module 4: Host and Management Scalability

4-42 Login Workflow of the vCenter Server IDP Federation
for ADFS

vCenter Server, ADFS, and AD interact during the login workflow as follows:
1. The user logs in to vCenter Server on the usual vCenter Server landing page.
2. vCenter Server redirects the authentication request to ADFS.
3. If needed, ADFS prompts the user to log in with Active Directory credentials.
4. ADFS authenticates the user with Active Directory.
5. ADFS issues a security token with group information from Active Directory.
6. vCenter Server uses the token to log in the user.

If the external IDP is unreachable, the login process falls back to the vCenter Server landing page,
showing an appropriate information message. Administrative users can still log in using their local
accounts (local OS or vsphere.local identity sources).

Module 4: Host and Management Scalability 299

4-43 Enabling ADFS on an Existing Enhanced Linked Mode
Configuration

300 Module 4: Host and Management Scalability

4-44 Lab 9: Configuring Identity Federation to Use Microsoft
ADFS

Module 4: Host and Management Scalability 301

4-45 Review of Learner Objectives

302 Module 4: Host and Management Scalability

4-46 Lesson 3: vSphere Trust Authority

Module 4: Host and Management Scalability 303

4-47 Learner Objectives

304 Module 4: Host and Management Scalability

4-48 Overview of VM Encryption

The enhancements introduced in vSphere 7 improved the general functionality of VM encryption

and are based on competitive analysis and feedback from VMware customers and partners.

To mitigate the risk of an administrator accessing data with authorization, VMs can be encrypted
and access to encryption/decryption functionality can be provided to a subset of administrators.
This is achieved using the inbuilt cryptographic privileges in vSphere.

Module 4: Host and Management Scalability 305

4-49 UEFI Secure Boot for ESXi Hosts

Starting with vSphere 7, a vSphere Trust Authority Trusted Cluster must have UEFI Secure Boot
enabled.
To ensure that boot code is executed without modification, Unified Extensible Firmware Interface
(UEFI) uses a digital signature provided by a trusted code creator. The digital signature is
embedded in every executable code section. Using public-private key pairs, the code creator signs
the code with a private key, which can be checked against the public key in a preinstalled
signature before the code is executed. If the executable code is marked as modified or invalid, the
code is not executed in the boot path. The system might take an alternate boot path, or the user
might be notified to take remedial actions.

306 Module 4: Host and Management Scalability

4-50 UEFI Secure Boot Sequence for ESXi Hosts

Module 4: Host and Management Scalability 307

4-51 TPM 2.0 on ESXi Hosts

308 Module 4: Host and Management Scalability

4-52 About Remote Attestation

Module 4: Host and Management Scalability 309

4-53 Remote Attestation Process

When an ESXi host is added to, rebooted from, or reconnected to vCenter Server, vCenter Server
requests an attestation key from the host. Part of the attestation key creation process also involves
the verification of the TPM hardware itself, to ensure that a known (and trusted) vendor produced
it.
vCenter Server requests that the host sends an attestation report. By checking that the information
corresponds to a configuration it deems trusted, vCenter Server identifies the platform on a
previously untrusted host.
vCenter Server verifies the authenticity of the signed quote, infers the software versions, and
determines the trustworthiness of said software versions.

310 Module 4: Host and Management Scalability

4-54 Requirements for Using TPM 2.0

Module 4: Host and Management Scalability 311

4-55 About vSphere Trust Authority

312 Module 4: Host and Management Scalability

4-56 About vSphere Trust Authority Administrator

Module 4: Host and Management Scalability 313

4-57 Use Case for vSphere Trust Authority

In vSphere 7, you can create a trusted computing base, which consists of a secure, manageable set
of ESXi hosts. vSphere Trust Authority implements a remote attestation service for the ESXi hosts
you want to trust. Furthermore, vSphere Trust Authority improves upon TPM 2.0 attestation
support (added to vSphere in the 6.7 release), to implement access restrictions on encryption keys
and so better protect VM workload secrets. In addition, vSphere Trust Authority allows only
authorized Trust Authority administrators to configure vSphere Trust Authority services, and
configure Trust Authority hosts. The Trust Authority Administrator can be the same user as the
vSphere administrator user, or a separate user.

314 Module 4: Host and Management Scalability

4-58 vSphere Trust Authority Components (1)

Several concepts apply to vSphere Trust Authority:

• Trust Authority Cluster: A set of restricted ESXi hosts with a known good configuration.

• Administered by a smaller number of trusted administrators.

• Trusted Key Provider (also called key management server [KMS]): A key provider that only
the Trust Authority Cluster should know.

• Attested cluster: Also known as a Trusted Cluster. The attestation report for a cluster must be
validated by the Trust Authority Cluster.

Module 4: Host and Management Scalability 315

4-59 vSphere Trust Authority Components (2)

Before vSphere 7, in vSphere 6.x, vSphere Trust Authority worked in the following ways:

• Attestation is a read-only view, and no repercussions occur when it fails.

• The attester is vCenter Server.

• Key handoff occurs through vCenter.

• You cannot safely encrypt vCenter Server because it forms a dependency on key access.

316 Module 4: Host and Management Scalability

4-60 vSphere Trust Authority Workflow

Module 4: Host and Management Scalability 317

4-61 ESXi Components

All vSphere Trust Authority services run behind the Reverse Proxy and API Forwarder. The
Trusted Infrastructure Agent communicates over port 443 to the Trust Authority Hosts.
The following internal ports are used:

• Attestation Service : Port 7889

• Key Provider Service : Port 7888

• Trusted Infrastructure Agent : Port 7890

318 Module 4: Host and Management Scalability

4-62 ESXi Components

Module 4: Host and Management Scalability 319

4-63 Attestation Service

The Attestation Service performs the following functions:

• Generates a signed document that contains assertions describing the binary and configuration
state of the remote ESXi hosts in the Trusted Cluster.

• Attests the state of the ESXi hosts using a Trusted Platform Module (TPM) 2.0 chip for
software measurement and reporting. The TPM measures the software stack and configuration
data and reports it to the Attestation Service.

• Verifies that the software measurement signature can be attributed to a previously configured
trusted TPM endorsement key (EK).

• Ensures that the software measurement matches one of a set of previously blessed ESXi
images.

• Signs a SAML token that it issues to the ESXi host, providing the assertions about the
identity, validity, and configuration of the ESXi host.

320 Module 4: Host and Management Scalability

4-64 Attestation Service Trust

Module 4: Host and Management Scalability 321

4-65 Key Provider Service

With the Key Provider Service, vCenter Server and ESXi hosts do not require direct key
management server (KMS) credentials.
In vSphere Trust Authority, an ESXi host can access an encryption key by authenticating with the
Key Provider Service rather than with vCenter Server (as in vSphere 6.7).
For the Key Provider Service to connect to a KMS, the Trust Authority Administrator must
configure a trust setup. For most servers that are compliant with the Key Management
Interoperability Protocol (KMIP), you configure a trust setup by creating client and server
certificates.

322 Module 4: Host and Management Scalability

4-66 Trusted Infrastructure Agent

Module 4: Host and Management Scalability 323

4-67 vCenter Server Topologies (1)

324 Module 4: Host and Management Scalability

4-68 vCenter Server Topologies (2)

Module 4: Host and Management Scalability 325

4-69 About Enabling vSphere Trust Authority

326 Module 4: Host and Management Scalability

4-70 About VM Encryption with vSphere Trust Authority

Module 4: Host and Management Scalability 327

4-71 VM Encryption with a Trusted Key Provider

328 Module 4: Host and Management Scalability

4-72 Identifying VM Encryption Key Providers

The crypto-util dictionary describe /path/vm.vmx command does not indicate

whether the VM is encrypted with a standard or trusted key provider. However, the key provider
name is present in the output so that a correlation can be made.

Module 4: Host and Management Scalability 329

4-73 Migrating a VM Encrypted with a Trusted Key Provider
(1)

330 Module 4: Host and Management Scalability

4-74 Migrating a VM Encrypted with a Trusted Key Provider
(2)

Module 4: Host and Management Scalability 331

4-75 Labs

332 Module 4: Host and Management Scalability

4-76 Lab 10: Assigning a vSphere Trust Authority
Administrator

Module 4: Host and Management Scalability 333

4-77 Lab 11: Enabling and Configuring vSphere Trust
Authority

334 Module 4: Host and Management Scalability

4-78 Lab 12: Encrypting a VM with a Trusted Key Provider

Module 4: Host and Management Scalability 335

4-79 Review of Learner Objectives

336 Module 4: Host and Management Scalability

4-80 Lesson 4: Host Profiles

Module 4: Host and Management Scalability 337

4-81 Learner Objectives

338 Module 4: Host and Management Scalability

4-82 About Host Profiles

Over time, your vSphere environment expands and changes. New hosts are added to the data
center and must be configured to coexist with other hosts in the cluster, for example:

• Storage: Datastores such as VMFS and NFS, iSCSI initiators and targets, and multipathing.

• Networking: Virtual switches, port groups, physical NIC speed, security, NIC teaming
policies, and so on.

• Licensing: Edition selection, license key input, and so on.

• Date and time configuration: Network Time Protocol (NTP) server

• Security settings: Firewall settings, Active Directory configuration, ESXi services, and so on.

• DNS and routing: DNS server, IP default gateway, and so on.

Module 4: Host and Management Scalability 339

Traditionally you would modify the configuration of a single host:

• Manually, using the vSphere Client or the command prompt

• Automatically, using scripts

Using the Host Profiles feature, you can export configuration settings from a master reference host
and save them as a portable set of policies, called a host profile. You can use this profile to quickly
configure other hosts in the data center. Configuring hosts with this method reduces the setup time
of new hosts. Multiple steps are reduced to a single click. Host profiles also eliminate the need for
specialized scripts to configure hosts.
The vCenter Server system uses host profiles as host configuration baselines so that you can
monitor changes to the host configuration, detect discrepancies, and fix them.
As a licensed feature of vSphere, Host Profiles are available only with Enterprise Plus licensing. If
you see errors, ensure that you have the appropriate vSphere licensing for your hosts.

340 Module 4: Host and Management Scalability

4-83 Implementing Host Profiles

Follow this basic workflow to implement host profiles:

1. Set up and configure a host to be used as the reference host. A reference host is the host from
which the profile is created.
2. Use the Create Profile wizard to create a profile from the designated reference host. vCenter
Server retrieves the configuration settings of the reference host, including IP addressing, NFS
mounts, and vSphere vMotion ports. You can edit the profile to further customize the policies
in the profile.
3. Attach the host profile to a host or a cluster of hosts. You can attach a profile to standalone
hosts, or to one or more clusters of hosts.

Module 4: Host and Management Scalability 341

4. Check host compliance to the reference host’s profile. If all hosts are compliant with the
reference host, the hosts are correctly configured.
5. Check new hosts for compliance against the reference host’s profile. You can easily apply the
host profile of the reference host to other hosts or clusters of hosts that are not compliant.

You can also import and export a profile file to a host profile that is in the VMware profile format
(.vpf). This feature is useful for sharing host profiles across multiple vCenter Server instances.

342 Module 4: Host and Management Scalability

4-84 Monitoring for Compliance

After the host profile is created and associated with a set of hosts or clusters, you can verify the
compliance status in various places in the vSphere Client:

• Host profile’s main view: Displays the compliance status of hosts.

• Host Summary tab: Displays the compliance status of the selected host.

• Host profile’s Monitor tab: After you check compliance, this view displays the compliance
status of the selected hosts. This view also displays any inconsistencies between the host
values and the host profile values (shown on the slide).

You can also check compliance for individual components or settings. These built-in cluster
compliance checks are useful for vSphere DRS and vSphere DPM.

Module 4: Host and Management Scalability 343

4-85 Applying Host Profiles

After attaching the profile to an entity (a host or a cluster of hosts), configure the entity by
applying the configuration as contained in the profile. Before you apply the changes, notice that
vCenter Server displays a detailed list of actions that will be performed on the host to configure it
and bring it into compliance. If this list of actions shows parameters that you do not want, you can
cancel applying the host profile.
Certain host profile policy configurations require that the host is rebooted after remediation. In
those cases, you are prompted to place the host into maintenance mode. You might be required to
place hosts into maintenance mode before remediation. Hosts that are in a fully automated
vSphere DRS cluster are placed into maintenance mode at remediation. For other cases, the
remediation process stops if the host is not placed into maintenance mode before remediation.

344 Module 4: Host and Management Scalability

4-86 Host Customization

For hosts that are provisioned with vSphere Auto Deploy, vCenter Server manages the entire host
configuration, which is captured in a host profile. In most cases, the host profile information is
sufficient to store all configuration information. Sometimes, the user is prompted for input when
the host provisioned with vSphere Auto Deploy boots. The host customization mechanism
manages such cases. After the user specifies the information, the system generates a host-specific
customization and stores it with the vSphere Auto Deploy cache and the vCenter Server host
object.

NOTE

In some documentation, you might see references to a host profile answer file. An
answer file is equivalent to host customization.

You can check the status of a host customization. If the host customization has all the required
user input answers, the status is Complete. If the host customization is missing some of the

Module 4: Host and Management Scalability 345

required user input answers, the status is Incomplete. You can also update or change the user input
parameters for the host profile’s policies in the host customization.

346 Module 4: Host and Management Scalability

4-87 Managing Host Customizations (1)

Module 4: Host and Management Scalability 347

4-88 Managing Host Customizations (2)

If a host profile contains any customized attributes, you can export them to a CSV file on your
desktop. For security, sensitive data such as passwords, is not exported. After the file is saved to
your desktop, you can manually edit the file and save it to apply the customizations later.

348 Module 4: Host and Management Scalability

4-89 Standardization Across Multiple vCenter Server
Instances

Some organizations might have large environments that span many vCenter Server instances. If
you want to standardize a single host configuration across multiple vCenter Server environments,
you can export the host profile from one vCenter Server instance so that other vCenter Server
instances can import the profile for use in their environments. Host profiles are not replicated,
shared, or kept synchronized across vCenter Server instances connected by Enhanced Linked
Mode.
Files are exported in the VMware profile format (.vpf).

NOTE

When a host profile is exported, administrator and user profile passwords are not exported. This r
a security measure that stops passwords from being exported in plain text. After the profile is imp
are prompted to reenter values for the password and the password is applied to a host.

Module 4: Host and Management Scalability 349

4-90 Lab 13: Using Host Profiles

350 Module 4: Host and Management Scalability

4-91 Review of Learner Objectives

Module 4: Host and Management Scalability 351

4-92 Lesson 5: Content Libraries

352 Module 4: Host and Management Scalability

4-93 Learner Objectives

Module 4: Host and Management Scalability 353

4-94 About Content Libraries

Content libraries provide a simple and effective way to manage multiple content types in a small
vSphere environment or across several vCenter Server instances. A content library can be
synchronized across remotely located sites and against remotely located vCenter Server instances.
Content libraries allow subscriptions to published data sets, resulting in consistent data across
multiple sites. A vSphere administrator controls publication and maintains consistent data and
security on that data. Data can be synchronized automatically across sites or on demand when
needed.
The type of storage that is used for content libraries depends on the data use in the library.

354 Module 4: Host and Management Scalability

4-95 Content Library Benefits

Storage and consistency is a key reason to install and use a content library. Sharing content and
ensuring that this content is kept up to date is a major task. For example, you might create a
central content library to store the master copies of OVF templates, ISO images, or other file types
for a main vCenter Server instance.
You can publish this content library so that other libraries that are located across the world can
subscribe and download an exact copy of the data. If an OVF template is added, modified, or
deleted from the published catalog, the subscriber synchronizes with the publisher, and the
libraries are updated with the latest content.
Because content libraries are globally available for subscription, security might be a concern. To
resolve this security concern, content libraries can be password-protected during publication. This
password is a static password and no integration occurs with vCenter Single Sign-On or Active
Directory in the current release of vCenter Server.

Module 4: Host and Management Scalability 355

4-96 Content Library Types

Use an optimized published content library when the subscribed libraries reside on a remote
vCenter Server system and Enhanced Linked Mode is not used.
You cannot deploy VMs from templates in an optimized library. You must first create a
subscribed library and then deploy VMs from the subscribed library.
A local library is the simplest form of library. A local library is available for use in data centers
that are objects to the local vCenter. A published library is a local library that is available for
subscription. Using the vCenter Server database, the content library tracks version changes. No
option to use previous versions of content is available.
A subscribed library is configured to subscribe to a published library or an optimized published
library. An administrator cannot directly change the contents of the subscribed library. But the
administrator can control how the data is synchronized to the subscribed content library.
An optimized published library is a library that is optimized to ensure lower CPU use and faster
streaming of the content over HTTP. Use this library as a main content depot for your subscribed
libraries. You cannot deploy VMs from an optimized library. You must first configure a

356 Module 4: Host and Management Scalability

subscribed library to subscribe to the optimized published library and then deploy VM from the
subscribed library.

Module 4: Host and Management Scalability 357

4-97 Storing Content in Content Libraries

Content library storage can be a datastore in your vSphere inventory or it can be a local file system
on a vCenter Server system.
Content library storage can also be a remote storage location. If you use a Windows-based vCenter
Server system, the storage can be an SMB server. If you use vCenter Server Appliance, the storage
can be an NFS server.
For more information about how to increase the performance of content libraries, see the VMware
blog article "How to Efficiently Synchronize, Import, and Export Content in VMware vSphere
Content Library" at https://blogs.vmware.com/performance/2015/09/efficiently-synchronize-
import-export-content-vmware-vsphere-content-library.html.

358 Module 4: Host and Management Scalability

4-98 Creating Content Libraries

Module 4: Host and Management Scalability 359

4-99 Advanced Configuration

360 Module 4: Host and Management Scalability

4-100 Populating Content Libraries

Your source to import items into content libraries can be a file stored on your local machine or a
file stored on a web server. You can add an item that resides on your local system to a content
library.

Module 4: Host and Management Scalability 361

4-101 Using the Content Library to Deploy VMs (1)

You can use a VM template from a content library to deploy a VM to a host or a cluster in your
vSphere inventory. You can also apply a guest OS customization specification to the VM.

362 Module 4: Host and Management Scalability

4-102 Using the Content Library to Deploy VMs (2)

To install a guest OS and its applications on a new VM, you can connect the CD/DVD device to
an ISO file that is stored in a content library.
A VM can only use an ISO image that is stored in a content library if its ESXi host can access the
datastore that originally stored the ISO image.

Module 4: Host and Management Scalability 363

4-103 Updating Content Library Items

Managing and keeping your virtual environment up to date might require that you update the
content of a library item.

364 Module 4: Host and Management Scalability

4-104 Publishing Content Libraries for External Use

Module 4: Host and Management Scalability 365

4-105 Content Library Subscriptions

When a content library subscription is created, the administrator selects how the content
synchronizes with the published content library. Content can be downloaded immediately if space
for the content is not a concern.
Synchronization can be set to on-demand so that only the metadata is copied when the
subscription is created. The full payload of the content library is downloaded only as it is used.

366 Module 4: Host and Management Scalability

4-106 Published and Subscribed Content Libraries

The slide shows two vCenter Server instances, with both the Content Library Service and the
Transfer Service installed and running on them. The user creates a content library on the first
vCenter Server instance.

The content in the content library is divided into two categories:

• Templates: This category contains only OVF templates. You can deploy them directly from
the content library as VMs to hosts, clusters, virtual data centers, and so on.

• Other: This category contains all other file types, such as ISO images. You can connect the
CD/DVD device to an ISO file that is stored in a content library, for example, to install a
guest OS on a new VM.

The administrator can publish the library. Publishing generates a URL that points to the
lib.json file of that library. The administrator can enable authentication and assign a password.

Module 4: Host and Management Scalability 367

By default, the user name is vcsp. You cannot set a user name during the creation of the content
library, but you can change the user name in the content library properties after creation.
Another content library is created on a separate vCenter Server instance and subscribes to the
publisher using the URL and password through the Content Library Service. The Content Library
Service then calls the Transfer Service. The Transfer Service is responsible for the import and
export of content from the publisher to the subscriber over HTTP NFC.
For more information, see vSphere Virtual Machine Administration at
https://docs.vmware.com/en/VMware-vSphere/7.0/vsphere-esxi-vcenter-server-70-virtual-
machine-admin-guide.pdf.

368 Module 4: Host and Management Scalability

4-107 Synchronizing and Versioning Content Libraries

Synchronization occurs after a library is subscribed. Synchronization occurs in an environment

that has two content libraries. Both libraries have the same content, but a new ISO image is added.
When any item is changed in a content library, the item number is incremented to note the
modification. The version number for the content library as a whole is also incremented. This
process is called simple versioning. The Content Library Service determines whether
synchronization should occur by using simple versioning. Simple versioning does not store
multiple versions or offer rollback features, it is purely a numerical value that is assigned to the
content library and its content.
The content library's version number is checked first. If this number matches the one in the
published and subscribed libraries, no synchronization is required. If a discrepancy is found, the
version number of each item in the library is checked. If discrepancies are found, only items with
such discrepancies are synchronized.
When multiple vCenter Server instances are configured in an Enhanced Linked Mode
environment, the contents of the published library can be synchronized to a subscribed library by

Module 4: Host and Management Scalability 369

directly copying the files from the source datastore to the destination datastore. When Enhanced
Linked Mode is not available, the contents of a published library must be streamed through the
Transfer Service component. In such a case, three scenarios are possible, based on the storage
configuration:

• Both published and subscribed libraries reside on a datastore.

• Both published and subscribed libraries reside on an NFS file system mounted on the vCenter
Server instances.

• The published library resides on an NFS file system, whereas the subscribed library resides on
a datastore.

370 Module 4: Host and Management Scalability

4-108 Content Library Maximums

The content library can be supported by a datastore or stored to available storage on vCenter
Server.
Regardless of the option selected, the content library can be supported only by a single file system
or datastore.
The maximum size of a library item is 1 TB. A content library can hold a maximum of 1,000 items
and a total of 2,000 items across all libraries in a vCenter Server instance.
The maximum number of concurrent synchronization operations on the published library’s
vCenter Server instance is 16.
Automatic synchronization occurs once every 24 hours by default, but the time and frequency can
be configured through the API. The administrator can synchronize an entire content library or an
individual item at any time through the vSphere Client.

Module 4: Host and Management Scalability 371

4-109 Lab 14: Creating a Content Library

372 Module 4: Host and Management Scalability

4-110 Review of Learner Objectives

Module 4: Host and Management Scalability 373

4-111 Lesson 6: Resource Pools

374 Module 4: Host and Management Scalability

4-112 Learner Objectives

Module 4: Host and Management Scalability 375

4-113 About Resource Pools

A resource pool is a logical abstraction for managing resources. Resource pools can be grouped
into hierarchies and used to hierarchically partition available CPU and memory resources. In the
example on the slide, RP-QA is the parent resource pool for RP-QA-UI. RP-Marketing and RP-
QA are siblings.
With a resource pool, you can divide and allocate resources to VMs and other resource pools. You
can control the aggregate CPU and memory resources of the compute resource. The compute
resource can be either a standalone host or a vSphere DRS cluster. Resource pools are also used to
delegate privileges to other users and groups.
The topmost resource pool is called the root resource pool. Each standalone host and each vSphere
DRS cluster has an (invisible) root resource pool that groups the resources of that host or cluster.
The root resource pool does not appear because the resources of the host (or cluster) and the root
resource pool are always the same.

376 Module 4: Host and Management Scalability

4-114 Resource Pool Attributes

Like VMs, resource pools have reservation, limit, and share values for CPU and memory
resources.
Resource pools also have a setting that enables reservations to be expanded. With this expandable
reservation attribute, a resource pool that cannot satisfy a reservation request can search through
its hierarchy to find unreserved capacity to satisfy the reservation request.

Module 4: Host and Management Scalability 377

4-115 Reasons to Use Resource Pools

378 Module 4: Host and Management Scalability

4-116 Resource Pool Example

Module 4: Host and Management Scalability 379

4-117 Resource Pool Example: CPU Shares

Resource pools can be organized hierarchically. The root resource pool is the topmost resource
pool. The root resource pool includes the sum for all CPUs (in megahertz) and the sum of all the
installed RAM (in megabytes) available in the compute environment (standalone host or cluster).
On the slide, the root resource pool is a standalone host named Svr001. It has 12,000 MHz of CPU
and 4 GB of RAM, available for use by other resource pools or VMs.
Except for the root resource pool, every resource pool has a parent resource pool. A resource pool
might contain child resource pools or only VMs that are powered on in the pool.
A child resource pool is used to allocate resources from the parent resource pool for the child’s
consumers. Administrative control can also be delegated to individuals or organizations. A child
resource pool cannot exceed the capacity of the parent resource pool. By creating a child pool, you
reserve resources from the parent pool, whether or not VMs in the child pool are powered on.
Shares specify the relative priority or importance of either a resource pool or VM. If a resource
pool has twice as many shares of a resource as another resource pool, it is entitled to consume
twice as much of that resource. The same logic applies to VMs.

380 Module 4: Host and Management Scalability

4-118 Resource Pool Example: CPU Contention

The example on the slide uses approximations to explain how the number of shares affects the
amount of CPU allocated to a VM.
All VMs are running on the same physical CPU, Srv01.
The Engineering pool gets 33 percent of the CPU and splits its allotment between VMs Eng-Test
and Eng-Prod. Likewise, the Finance pool gets 67 percent of the CPU and splits its allotment
between VMs Fin-Test and Fin-Prod. A VM’s resource settings are constrained by the resources
of the resource pool to which the VM belongs.
The VM Eng-Test gets approximately 33 percent of the CPU allocation of the Engineering
resource pool [1,000/(1,000 + 2,000)]. This figure is equal to about 11 percent of the physical
CPU (33 percent of 33 percent equals approximately 11 percent). Each of the VMs gets a
percentage of the physical CPU allocated to its resource pool based on its individual share
allocation.

Module 4: Host and Management Scalability 381

4-119 About Expandable Reservations

In the example, eCommerce Apps is an expandable child resource pool.

The reservation of a child resource pool cannot exceed that of its parent. The search for unused
resources goes through the ancestry of the root resource pool or to the first resource pool where
the Expandable Reservation Type is not enabled.
Use expandable reservation carefully. A single child resource pool can use all its parent’s
available resources, leaving nothing directly available for other child resource pools. This issue
can occur if VMs are improperly moved around to get them to start based on resource
reservations.
You might want to disable the Expandable Reservation Type when you configure a fixed amount
of resources to a group. For example, an IT administrator whose customers are different
organizations in the company that have paid for a fixed amount of CPU and memory resources
might want to disable the Expandable Reservation Type.

382 Module 4: Host and Management Scalability

4-120 Example of Expandable Reservation (1)

Module 4: Host and Management Scalability 383

4-121 Example of Expandable Reservation (2)

384 Module 4: Host and Management Scalability

4-122 Resource Pool Summary Tab

The Summary tab provides important information:

• The Resource Settings pane displays CPU and Memory share settings.

• The Resource Consumers pane displays the number of VMs, number of powered-on VMs,
and child resource pools that are in the selected resource pool.

• The Tags pane displays tags assigned to objects that reside in the resource pool.

Module 4: Host and Management Scalability 385

4-123 Resource Reservation Details

For the resource pool’s CPU and memory resources, the following information is provided:

• Amount of CPU and memory reservation that is configured

• Type of reservation used (expandable or fixed)

• Amount of reservation in use by VMs and child pools

• Amount of CPU and memory that is available to be reserved

386 Module 4: Host and Management Scalability

4-124 Resource Pool Shares

Resource pools can be organized hierarchically. The root resource pool is the topmost resource
pool. The root resource pool includes the sum for all CPUs (in megahertz) and the sum of all the
installed RAM (in megabytes) available in the compute environment (standalone host or cluster).
A child resource pool is used to allocate resources from the parent resource pool for the child’s
consumers. A child resource pool cannot exceed the capacity of the parent resource pool. By
creating a child pool, you reserve resources from the parent pool. In the example, the root resource
pool is Cluster-A. Cluster-A has two child resource pools: the Engineering pool and the Finance
pool.
Shares specify the relative priority or importance of a resource pool. In the example, you want the
finance VMs to have twice as much processor time as the engineering VMs. You can configure
the Finance pool with twice as many CPU shares as the Engineering pool.
You can also specify shares per VM and distribute the resource allocation within the resource
pool. VM shares are relative only to other powered-on VMs within the same resource pool. In the
example, all VMs have an equal number of shares. However, the Finance VMs are entitled to

Module 4: Host and Management Scalability 387

about 67% of the cluster processor time. Each Finance VM gets about 33% of the entire cluster
CPU resources, or twice as much as for engineering VMs.

388 Module 4: Host and Management Scalability

4-125 Diluted Shares

When you power on additional VMs in a resource pool, the resource allocation per VM is
recalculated based on the value configured for shares. In the example, two more VMs are powered
on in the Finance pool. All VMs in the cluster have an equal number of shares. Therefore, the
resources are divided by four.
Because the finance VMs are entitled to about 68% of the cluster processor time, each finance VM
gets about 17% of the entire cluster CPU resources. The value of shares within the Finance pool is
diluted and is no longer twice as much as the engineering VMs.

Module 4: Host and Management Scalability 389

4-126 About Scalable Shares (1)

When you enable scalable shares on a resource pool, all child resource pools maintain the original
ratio of shares configured in the parent pool.
In the example, you enable scalable shares on Cluster-A to dynamically adjust the resource
allocation of child resource pools. All VMs have an equal number of shares, so the shares
configuration of the child resource pools adjusts to give every finance VM twice as much as the
engineering VMs. The shares configuration of the child resource pools adjusts to 80% for the
Finance pool, and 20% for the Engineering pool. Each finance VM gets 20% of the Cluster-A
processor time, and the engineering VMs get 10%.

390 Module 4: Host and Management Scalability

4-127 About Scalable Shares (2)

Module 4: Host and Management Scalability 391

4-128 Enabling Scalable Shares

392 Module 4: Host and Management Scalability

4-129 Lab 15: Managing Resource Pools

Module 4: Host and Management Scalability 393

4-130 Review of Learner Objectives

394 Module 4: Host and Management Scalability

4-131 Key Points

Module 4: Host and Management Scalability 395

Module 5
CPU Optimization

Module 5: CPU Optimization397

5-2 Importance

398 Module 5: CPU Optimization

5-3 Module Lessons

Module 5: CPU Optimization399

5-4 Lesson 1: CPU Virtualization Concepts

400 Module 5: CPU Optimization

5-5 Learner Objectives

Module 5: CPU Optimization401

5-6 About the CPU Scheduler

A CPU capable of simultaneous multithreading (SMT) has two logical cores per physical core.
Because most modern processors are equipped with multiple cores per processor, building a
system with tens of cores running hundreds of VMs is easy. In such a large system, allocating
CPU resources efficiently and fairly is critical.

402 Module 5: CPU Optimization

5-7 About VM Worlds

The role of the CPU scheduler is to assign execution contexts to processors in a way that meets
system objectives such as responsiveness, throughput, and use. On conventional operating
systems, the execution context corresponds to a process or a thread. On ESXi hosts, the execution
context corresponds to a world.
Non-VM worlds exist as well. These non-VM worlds are VMkernel worlds, which are used to
perform various system tasks. Examples of these non-VM worlds include the idle and
vmotionServer worlds.

Module 5: CPU Optimization403

5-8 CPU Scheduler Features

One of the main tasks of the CPU scheduler is to choose which world is to be scheduled to a
processor. If the target processor is already occupied, the scheduler must decide whether to
preempt the currently running world on behalf of the chosen world.
A world migrates from a busy processor to an idle processor. A world migration can be initiated
either by a physical CPU that becomes idle or by a world that becomes ready to be scheduled.
An ESXi host implements the proportional-share-based algorithm. When CPU resources are
overcommitted, the ESXi host time-slices the physical CPUs across all VMs so that each VM runs
as if it had its specified number of virtual processors. The ESXi host associates each world with a
share of CPU resource.
This association of resources is called entitlement. Entitlement is calculated from user-provided
resource specifications such as shares, reservation, and limits. When making scheduling decisions,
the ratio of the consumed CPU resource to the entitlement is used as the priority of the world. If a
world consumed less than its entitlement, the world is considered high priority and is likely to be
chosen to run next.

404 Module 5: CPU Optimization

5-9 CPU Scheduler Support for SMP VMs

A symmetric multiprocessing (SMP) VM presents the guest OS and applications with the illusion
that they are running on a dedicated physical multiprocessor. An ESXi host implements this
illusion by supporting co-scheduling of the virtual CPUs (vCPUs) in an SMP VM.
Without co-scheduling, the vCPUs associated with an SMP VM would be scheduled
independently, breaking the guest's assumptions regarding uniform progress.
The progress of each vCPU in an SMP VM is tracked individually. The skew is measured as the
difference in progress between the slowest vCPU and each of the other vCPUs.
The ESXi scheduler maintains a detailed cumulative skew value for each vCPU in an SMP VM. A
vCPU is considered to be making progress if it consumes CPU at the guest level or if it halts. The
time spent in the hypervisor is excluded from the progress. This exclusion means that the
hypervisor execution might not always be co-scheduled. This behavior is acceptable because not
all operations in the hypervisor benefit from being co-scheduled. When co-scheduling is
beneficial, the hypervisor makes explicit co-scheduling requests to achieve good performance.

Module 5: CPU Optimization405

5-10 Relaxed Co-Scheduling

With relaxed co-scheduling, only the vCPUs that are skewed must be co-started. Relaxed co-
scheduling ensures that, when any vCPU is scheduled, all other vCPUs that are behind are also
scheduled, reducing skew. This approach is called relaxed co-scheduling because only a subset of
a VM’s vCPUs must be scheduled simultaneously after skew is detected.
The vCPUs that advanced too much are individually stopped. After the lagging vCPUs catch up,
the stopped vCPUs can start individually. Co-scheduling all vCPUs is still attempted to maximize
the performance benefit of co-scheduling.
An idle CPU has no co-scheduling overhead because the skew does not increase when a vCPU
halts. An idle vCPU does not accumulate skew and is treated as if it were running for co-
scheduling purposes. This optimization ensures that idle guest vCPUs do not waste physical
processor resources, which can instead be allocated to other VMs. For example, an ESXi host with
two physical cores might be running one vCPU each from two different VMs, if their sibling
vCPUs are idling, without incurring co-scheduling overhead.

406 Module 5: CPU Optimization

5-11 CPU Performance Factors

You might think that idle VMs do not cost anything in terms of performance. However, timer
interrupts still must be delivered to these VMs. Lower timer interrupt rates can help guest OS
performance.
You should use as few vCPUs in your VM as possible to reduce the amount of timer interrupts
necessary, and to reduce any co-scheduling overhead that might be incurred. Also, use SMP
kernels, and not uniprocessor kernels, in SMP VMs.
CPU capacity is a finite resource. Even on a server that allows additional processors to be
configured, a maximum always exists as to the number of processors that can be installed. As a
result, performance problems might occur when insufficient CPU resources are available to satisfy
demand.

Module 5: CPU Optimization407

5-12 Warning Sign: Ready Time

To achieve best performance in a consolidated environment, you must consider ready time. Ready
time is the time that a VM must wait in the queue in a ready-to-run state before it can be scheduled
on a CPU.
When multiple processes are trying to use the same physical CPU, that CPU might not be
immediately available, and a process must wait before the ESXi host can allocate a CPU to it. The
CPU scheduler manages access to the physical CPUs on the host system.

408 Module 5: CPU Optimization

5-13 Transitioning Between World States

When first added, a world is either in the run state or the ready state, depending on the availability
of a physical CPU. A world in the ready state is dispatched by the CPU scheduler and enters the
run state. It can be later descheduled and enters either the ready state or the co-stop state. The co-
stopped world is co-started later and enters the ready state. A world in the run state enters the wait
state by blocking on a resource. It is woken up when the resource becomes available.
A world becoming idle enters the wait_idle state, a special type of wait state, although it is not
explicitly blocking on a resource. An idle world is woken up whenever it is interrupted.

Module 5: CPU Optimization409

5-14 Activity: CPU Scheduler (1)

410 Module 5: CPU Optimization

5-15 Activity: CPU Scheduler (2)

Module 5: CPU Optimization411

5-16 Security Boundaries

For an overview of the security issues, descriptions of each scheduler option, and the results of
performance testing with different scenarios, see Performance of vSphere 6.7
Scheduling Options at https://blogs.vmware.com/vsphere/2019/05/which-vsphere-cpu-scheduler-
to-choose.html

412 Module 5: CPU Optimization

5-17 Review of Learner Objectives

Module 5: CPU Optimization413

5-18 Lesson 2: NUMA Architecture in vSphere

414 Module 5: CPU Optimization

5-19 Learner Objectives

Module 5: CPU Optimization415

5-20 How the ESXi NUMA Scheduler Works

A home node is one of the system’s NUMA nodes. In selecting a home node for a VM, the ESXi
NUMA scheduler attempts to keep both the VM and its memory on the same node, thereby
maintaining good NUMA locality.
The ESXi NUMA scheduler might migrate a VM to a new home node to reduce processor load
imbalance. Because this event might cause more of the VM’s memory to be remote, the scheduler
might migrate the VM’s memory dynamically to its new home node to improve memory locality.
The NUMA scheduler might also swap VMs between nodes when doing so improves overall
memory locality.

416 Module 5: CPU Optimization

5-21 NUMA-Aware ESXi

A non-uniform memory access (NUMA) node contains processors and memory, much like a small
SMP system. However, an advanced memory controller allows a node to use memory on all other
nodes, creating a single system image.
In a NUMA host, the delay incurred when accessing memory varies for different memory
locations.
When poor NUMA locality occurs, the VM’s performance might be less than if its memory were
all local.

Module 5: CPU Optimization417

5-22 Terminology: vCPUs, pCPUs, Cores, and More

To understand how NUMA functions in vSphere, a review of CPU components at the virtual and
physical layers is helpful.

418 Module 5: CPU Optimization

5-23 About the NUMA Client

When a VM is powered on, the number of vCPUs in a VM are compared with the number of
physical cores in a NUMA node. If enough physical cores exist in the NUMA node to satisfy the
vCPU count, a single NUMA client is created. A uniform memory address space is presented to
the VM.

Module 5: CPU Optimization419

5-24 Wide VM NUMA Support

Two NUMA clients are created for this VM and the vCPUs are equally distributed across the two
NUMA clients.

Wide VMs are assigned two or more NUMA nodes and are preferentially allocated memory local
to those NUMA nodes.

Because vCPUs in these wide VMs might sometimes need to access memory outside their own
NUMA node, the VM might experience higher average memory access latencies than VMs that fit
entirely within a NUMA node.

420 Module 5: CPU Optimization

5-25 Virtual NUMA Topology

Memory latency can be mitigated by appropriately configuring virtual NUMA (vNUMA) on the
VMs. vNUMA enables the guest OS to assume part of the memory-locality management task.
vNUMA can provide significant performance benefits, although the benefits depend heavily on
the level of NUMA optimization in the guest OS and applications.
You can obtain the maximum performance benefits from vNUMA if your vSphere clusters are
composed entirely of hosts with matching NUMA architecture. When a VM that is enabled for
vNUMA is powered on, its vNUMA topology is set based in part on the NUMA topology of the
underlying physical host. After a VM’s vNUMA topology is initialized, it does not change unless
the number of vCPUs in that VM is changed. If a vNUMA VM is moved to a host with a different
NUMA topology, the VM’s vNUMA topology might no longer be optimal. This move might
result in reduced performance.
For more information on NUMA and vNUMA, see NUMA Deep Dive Part 5: ESXi VMkernel
NUMA Constructs on the Frank Denneman website at https://frankdenneman.nl/.

Module 5: CPU Optimization421

5-26 Optimizing the Number of Cores per Socket for VMs
(1)

When you create VM, the number of vCPUs that you specify is divided by the cores per socket
value to give you the number of sockets. The default Cores per Socket value is 1.
To ensure both an optimal vNUMA topology and performance, regardless of what vSphere
version you use, ensure that the VM vCPU count does not exceed the physical core count of a
single physical NUMA node.

422 Module 5: CPU Optimization

5-27 Optimizing the Number of Cores per Socket for VMs
(2)

Module 5: CPU Optimization423

5-28 Optimizing the Number of Cores per Socket for VMs
(3)

424 Module 5: CPU Optimization

5-29 Activity: VM Cores Per Socket (1)

Module 5: CPU Optimization425

5-30 Activity: VM Cores Per Socket (2)

426 Module 5: CPU Optimization

5-31 Memory Considerations with vNUMA (1)

Module 5: CPU Optimization427

5-32 Memory Considerations with vNUMA (2)

428 Module 5: CPU Optimization

5-33 Memory Considerations with vNUMA (3)

Module 5: CPU Optimization429

5-34 Memory Considerations with vNUMA (4)

430 Module 5: CPU Optimization

5-35 vNUMA Considerations

For more information about vNUMA rightsizing and vNUMA topology, see Virtual Machine
vCPU and vNUMA Rightsizing - Rules of Thumb at
https://blogs.vmware.com/performance/2017/03/virtual-machine-vcpu-and-vnuma-rightsizing-
rules-of-thumb.html and Decoupling of Cores per Socket from Virtual NUMA Topology in
vSphere 6.5 on the Frank Denneman website at https://frankdenneman.nl/.

Module 5: CPU Optimization431

5-36 Review of Learner Objectives

432 Module 5: CPU Optimization

5-37 Lesson 3: Using esxtop

Module 5: CPU Optimization433

5-38 Learner Objectives

434 Module 5: CPU Optimization

5-39 About esxtop (1)

You can run the esxtop utility using vSphere ESXi Shell to communicate with the management
interface of the ESXi host. To do this, you must have root user privileges.

Module 5: CPU Optimization435

5-40 About esxtop (2)

You use lowercase and uppercase letters to specify which columns appear in which order on the
CPU, memory, storage adapter, storage device, VM storage, and network panels. You can add or
remove columns containing data in the respective esxtop panels.

For more information about using esxtop or resxtop in interactive mode, see Using esxtop or
resxtop in Interactive Mode
at https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.monitoring.doc/GUID-
49D45588-AE12-4143-9CCD-3E77368C506E.html.

436 Module 5: CPU Optimization

5-41 Getting Help in esxtop

Module 5: CPU Optimization437

5-42 Customizing Fields in esxtop

For more information about the esxtop utility, see vSphere Monitoring and Performance at
https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.monitoring.doc/GUID-
A31249BF-B5DC-455B-AFC7-7D0BBD6E37B6.html.

438 Module 5: CPU Optimization

5-43 Review of Learner Objectives

Module 5: CPU Optimization439

5-44 Lesson 4: Monitoring CPU Activity

440 Module 5: CPU Optimization

5-45 Learner Objectives

Module 5: CPU Optimization441

5-46 CPU Key Performance Indicators for ESXi Hosts

The values obtained from esxtop are gathered based on different sampling intervals. By default,
esxtop uses a sampling interval of 5 seconds.

442 Module 5: CPU Optimization

5-47 CPU Key Performance Indicators for VMs

The following CPU statistics are shown for a VM:

• %USED: CPU use. The percentage of physical CPU core cycles used by a group of worlds
running VMs, resource pools, or other worlds.

• %RDY: Percentage of time that the VM was ready to run but was not provided CPU
resources on which to execute.

• %CSTP: Percentage of time that the vCPUs of a VM spent in the co-stop state, waiting to be
co-started. This value gives an indication of the co-scheduling overhead incurred by the VM.
If this value is low, any performance problems should be attributed to other issues and not to
the co-scheduling of the VM’s vCPUs.

• %MLMTD: Percentage of time that the VMkernel did not run the VM because that would
violate the VM’s limit setting.

• %VMWAIT: Percentage of time that the VM spent in a blocked state waiting for events.

Module 5: CPU Optimization443

• %SWPWT: Percentage of time that the VM spends waiting for the VMkernel to swap
memory.

• NWLD: Number of members in the VM of the world that is running.

444 Module 5: CPU Optimization

5-48 Using esxtop to Monitor ESXi Host CPU

Module 5: CPU Optimization445

5-49 Using esxtop to View CPU Metrics per VM

The esxtop tool often shows more information than you need for the specific performance
problem that you are troubleshooting. You can issue commands in the interactive mode to
customize the display. The following commands are the most useful when monitoring CPU
performance:

• Press the spacebar: Immediately updates the current screen.

• Enter s: Prompts you for the delay between updates, in seconds. The default value is 5
seconds and the minimum value is 2 seconds.

• Enter V: Displays VM instances only. Do not confuse this command with lowercase v which
shifts the view to the storage VM resource use screen. If you accidentally use lowercase, enter
c to return to the CPU resource use panel.

• Enter e: Toggles between CPU statistics being displayed, expanded, or unexpanded. The
expanded display includes CPU resource use statistics broken down by individual worlds

446 Module 5: CPU Optimization

belonging to a VM. All percentages of the individual worlds are percentages of a single
physical CPU.

Module 5: CPU Optimization447

5-50 Using esxtop to View Single CPU Statistics

The default esxtop window shows aggregate information for all vCPUs in a VM. Enter e and
enter the group ID to expand the group and display all the worlds associated with that group.
In the example, the VM group with group ID 299335 is expanded to show all the worlds
associated with that group.

448 Module 5: CPU Optimization

5-51 Using esxtop to Monitor VM vCPU

CPU use, ready time, wait time, and co-stop information for each vCPU can be tracked in this
esxtop panel.

Module 5: CPU Optimization449

5-52 Important Metrics to Monitor

Though high use is not necessarily an indication of poor CPU performance, it is often mistakenly
understood as such. Administrators in the physical space view high use as a warning sign because
of the limitations of physical architectures. However, one of the goals of virtualization is to
efficiently use the available resources. Detecting high CPU use, and queuing, might indicate a
performance problem.
As an analogy, consider a freeway. If every lane of the freeway is packed with cars but traffic is
moving at a reasonable speed, you have high use but good performance. The freeway is doing
what it was designed to do, moving many vehicles at its maximum capacity. Now, imagine the
same freeway at rush hour. More cars than the maximum capacity of the freeway are trying to
share the limited space. Traffic speed reduces to the point where cars line up (queue) at the on-
ramps while they wait for shared space to open up so that they can merge onto the freeway. This
condition indicates high use with queuing and that a performance problem exists.
Ready time reflects the idea of queuing in the CPU scheduler. Ready time is the amount of time
that a vCPU was ready to run but was asked to wait due to an overcommitted physical CPU.

450 Module 5: CPU Optimization

5-53 Example: Identifying CPU Overcommitment

The example uses esxtop to detect CPU overcommitment. Looking at the pCPU line near the
top of the window, you can determine that the host has four CPUs, three of which are moderately
used.
Three active VMs are shown: Linux01, ResourceHog01, and ResourceHog02. These VMs are
active because they have relatively high values in the %USED column. The values in the %USED
column alone do not necessarily indicate that the CPUs are overcommitted. In the %RDY column,
you see that the three active VMs have high values. Values greater than 10 percent are considered
high. High %RDY values plus high %USED values are a sure indicator that your CPU resources
are overcommitted.

Module 5: CPU Optimization451

5-54 Host CPU Saturation

In the esxtop tool, ready time is reported in a percentage format. A figure of 5 percent means that
the VM spent 5 percent of its last sample period waiting for available CPU resources.
Using esxtop values for ready time, you can interpret the value as follows:

• A ready time that is less than or equal to 5 percent is normal. Very small single-digit numbers
result in minimal effects on users.

• If ready time is between 5 and 10 percent, ready time merits attention.

• If ready time is greater than 10 percent, even though some systems continue to meet
expectations, double-digit ready time percentages often mean that action is required to address
performance problems.

452 Module 5: CPU Optimization

5-55 Resolving Host CPU Saturation

Populating an ESXi host with too many VMs running compute-intensive applications can make
supplying sufficient resources to any individual VM impossible.
When the host has a few VMs with high CPU demand, you might solve the problem by first
increasing the efficiency of CPU use in the VMs. VMs with high CPU demands are those most
likely to benefit from simple performance tuning. In addition, unless care is taken with VM
placement, moving VMs with high CPU demands to a new host might move the problem to that
host.
When the host has many VMs with moderate CPU demands, reducing the number of VMs on the
host by rebalancing the load is the simplest solution. Performance tuning on VMs with low CPU
demands yields smaller benefits. It is also time-consuming if the VMs are running different
applications.
When the host has a mix of VMs with high and low CPU demand, the appropriate solution
depends on available resources and skill sets. If additional ESXi hosts are available, rebalancing
the load might be the best approach. If additional hosts are not available or if expertise is available
for tuning the high-demand VMs, increasing efficiency might be the best approach.

Module 5: CPU Optimization453

5-56 Activity: CPU Metrics (1)

454 Module 5: CPU Optimization

5-57 Activity: CPU Metrics (2)

Module 5: CPU Optimization455

5-58 Guest CPU Saturation

Guest CPU saturation occurs when the application and OS running in a VM use all the CPU
resources that the ESXi host is providing to that VM. The occurrence of guest CPU saturation
does not necessarily indicate that a performance problem exists.
Compute-intensive applications commonly use all available CPU resources. Even less-intensive
applications might experience periods of high CPU demand without experiencing performance
problems. However, if a performance problem exists while guest CPU saturation occurs, steps
should be taken to eliminate the condition.
Adding CPU resources is often the easiest choice, particularly in a virtualized environment.
However, this approach misses inefficient behavior in the guest application and OS that might be
wasting CPU resources or, in the worst case scenario, that might be an indication of an error
condition in the guest. If a VM continues to experience CPU saturation despite adding CPU
resources, the tuning and behavior of the application and OS should be investigated.

456 Module 5: CPU Optimization

5-59 Using One vCPU in an SMP VM

When a VM that is configured with more than one vCPU actively uses only one of those vCPUs,
resources that could be used to perform useful work are being wasted.
The guest OS might be configured with a uniprocessor kernel (with Linux) or hardware
abstraction layer (HAL) (with Windows). For a VM to take advantage of multiple vCPUs, the
guest OS running in the VM must be able to recognize and use multiprocessor cores. Follow the
documentation provided by your OS vendor to verify for the type of kernel or HAL that is being
used by the guest OS.
The application might be pinned to a single core in the guest OS. Many modern operating systems
provide controls for restricting applications to run on only a subset of available processor cores. If
these controls were applied in the guest OS, the application might run on only vCPU0. To
determine whether this is the case, inspect the commands used to start the application or other OS-
level tunings applied to the application. The inspection of these values must be accompanied by an
understanding of the OS vendor’s documentation regarding restricting CPU resources.

Module 5: CPU Optimization457

The application might be single-threaded. Many applications, particularly older applications, are
written with only a single thread of control. These applications cannot take advantage of more than
one processor core. Running a single-threaded application on an SMP VM might lead to low guest
CPU use, instead of running a single-threaded application on vCPU0 only. Without knowledge of
the application design, determining whether an application is single-threaded might be difficult.
Observing the behavior of the application in an SMP VM might help in determining whether the
application is single-threaded.

458 Module 5: CPU Optimization

5-60 Low Guest CPU Use

To determine whether the problem is low guest CPU use, use vSphere Web Client to verify the
CPU use of the VMs on the host.

Module 5: CPU Optimization459

5-61 Introduction to Lab 8: Monitoring CPU Performance

In this lab, you compare the performance of a MySQL database running on a single-vCPU VM
with a dual-vCPU VM. You run three tests. Each test measures CPU use, ready time, idle time,
and operations per minute.

You record the data after each test:

• Case 1: On a single-vCPU VM, you run the starttest1 program. starttest1 simulates a single-
threaded application accessing the database.

• Case 2: On a dual-vCPU VM, you run the starttest1 program.

• Case 3: On a dual-vCPU VM, you run the starttest2 program. starttest2 simulates a dual-
threaded application accessing the database.

460 Module 5: CPU Optimization

5-62 Lab 16: Monitoring CPU Performance

Module 5: CPU Optimization461

5-63 Review of Lab: Monitoring CPU Performance

462 Module 5: CPU Optimization

5-64 Review of Learner Objectives

Module 5: CPU Optimization463

5-65 Key Points

464 Module 5: CPU Optimization

Module 6
Memory Optimization

Module 6: Memory Optimization 465

6-2 Importance

466 Module 6: Memory Optimization

6-3 Module Lessons

Module 6: Memory Optimization 467

6-4 Lesson 1: Memory Virtualization Concepts

468 Module 6: Memory Optimization

6-5 Learner Objectives

Module 6: Memory Optimization 469

6-6 Virtual Memory Overview

The following levels of memory exist when running a VM:

• Guest virtual memory: Memory visible to the applications running in the VM. This memory is
a continuous virtual address space presented by the guest OS to applications.

• Guest physical memory: Memory visible to the guest OS running in the VM. The guest OS
sees this memory as physical memory.

• Host physical memory: Also called machine memory, it is the physical memory that is
available to the hypervisor.

470 Module 6: Memory Optimization

6-7 Application and Guest OS Memory Management

In a nonvirtual environment, the OS owns all physical memory in the system. The hardware does
not provide interfaces for the OS to explicitly allocate or free physical memory. The OS
establishes the definitions of allocated or free physical memory.
Operating systems have different implementations to realize this abstraction. Whether a physical
page is free depends on which list the page resides in.

Module 6: Memory Optimization 471

6-8 VM Memory Management

Because a VM runs a guest OS and one or more applications, the VM memory management
properties combine both application and OS memory management properties. Like an application,
when a VM first starts, it has no preallocated host physical memory. Like an OS, the VM cannot
explicitly allocate host physical memory through standard interfaces.
The hypervisor intercepts the VM’s memory accesses and allocates host physical memory for the
VM on its first access to the memory. The hypervisor always writes zeros to the host physical
memory before assigning it to a VM.

472 Module 6: Memory Optimization

6-9 Memory Reclamation

VM memory deallocation acts like an OS, such that the guest OS frees guest physical memory by
adding memory page numbers to the guest free list.
However, the data of the freed memory might not be modified at all. As a result, when a portion of
guest physical memory is freed, the mapped host physical memory does not usually change its
state. Only the guest free page list is changed.
It is difficult for the hypervisor to know when to free host physical memory when guest physical
memory is deallocated, or freed, because the guest OS free list is not accessible to the hypervisor.
The hypervisor is completely unaware of which pages are free or allocated in the guest OS. As a
result, the hypervisor cannot reclaim host physical memory when the guest OS frees guest
physical memory.

Module 6: Memory Optimization 473

6-10 Reason for Reclaiming Memory

With memory overcommitment, ESXi ensures that the host physical memory is consumed by
active guest memory as much as possible. Typically, some VMs might be lightly loaded compared
to others, and so, for much of the time, their memory sits idle. Memory overcommitment allows
the hypervisor to use memory-reclamation techniques to take the inactive or unused host physical
memory away from the idle VMs and give it to other VMs that will actively use it.
With memory overcommitment, each VM has a smaller footprint in host physical memory,
making it possible to fit more VMs on the host while still achieving good performance for all
VMs. In the example, you can enable a host with 4 GB of physical memory to run three VMs with
2 GB of VM memory each.
This action assumes that all VMs are using the default setting for memory reservation, which is 0.
In such a case, all the VMs would power on. If all the VMs had a memory reservation of 2 GB
each, without memory overcommitment, then only one VM can be run. The hypervisor cannot
reserve host physical memory for more than one VM because each VM has overhead memory as
well.

474 Module 6: Memory Optimization

6-11 Guest OS Memory Terminology

Module 6: Memory Optimization 475

The memory used by the guest OS in the VM can be described in the following way:

• Memory size is the total amount of memory that is presented to the guest. By default, the
memory size corresponds to the memory configured when the VM is created.

• The total amount of memory (memory size) can be divided into two parts:

• Free memory: Memory that is not assigned to the guest OS or to applications.

• Allocated memory: Memory that is assigned to the guest OS or to applications.

• Allocated memory can be further subdivided into two types:

• Active memory: Memory that was recently used by applications.

• Idle memory: Memory that was not recently used by applications.

Keeping a VM’s active memory in host physical memory is vital for maximizing VM
performance.
These terms are used to describe the guest OS’s memory. The ESXi host does not know about a
guest’s free, active, or idle memory.

476 Module 6: Memory Optimization

6-12 VM Memory Reclamation Techniques

When multiple VMs are running, some VMs might have identical sets of memory content.
Opportunities exist for sharing memory across VMs and in a single VM. With transparent page
sharing (TPS), the hypervisor can reclaim redundant memory page copies. Only one read-only
copy is kept in the host physical memory. The page is shared by multiple VMs. When a VM
writes to the page, a copy of the page is created for the VM before the write operation is complete.
Ballooning, memory compression, host-level swapping, and host swap cache are the other
memory reclamation techniques.
Hypervisor swapping is used as a last resort when ballooning and TPS are insufficient to reclaim
memory. If needed, the hypervisor will swap out guest physical memory to a virtual swap (VSWP)
file.
Host swap cache is an optional memory reclamation technique that uses local flash storage to
cache a VM’s memory pages. By using local flash storage, the VM avoids the latency associated
with a storage network that would be used if it swapped memory pages to VSWP files.

Module 6: Memory Optimization 477

When severe memory pressure occurs and the hypervisor must swap memory pages to disk, the
hypervisor swaps to a host swap cache rather than to a VSWP file. When a host runs out of space
on the host cache, a VM’s cached memory is migrated to a VM’s regular VSWP file. Each host
must have its own host swap cache configured.

478 Module 6: Memory Optimization

6-13 Transparent Page Sharing (1)

Many physical memory pages on a host often store identical contents. For example, if many VMs
are running Windows 2012R2, each VM has the same executable files in its memory. TPS is a
background process that scans and hashes the contents of guest physical memory pages. Pages that
generate the same hash value are scanned, byte by byte. If the pages are truly identical, they are
single-instanced and shared between the relevant VMs.
A hash value is generated based on the candidate guest physical page content. The hash value is
used as a key to find a global hash table. In this table, each entry records a hash value and the
physical page number of a shared page. If the hash value of the candidate guest physical page
matches an existing entry, a full comparison of the page contents is performed to exclude a false
match. If the candidate guest physical page’s content is confirmed to match the content of an
existing shared host physical page, then the guest physical-to-host physical mapping of the
candidate guest physical page is changed to the shared host physical page. The redundant host
memory copy (the page pointed to by the dashed arrow) is reclaimed. This remapping is invisible
to the VM and inaccessible to the guest OS. Because of this invisibility, sensitive information
cannot be leaked from one VM to another.

Module 6: Memory Optimization 479

A standard copy-on-write technique is used to handle writes to the shared host physical pages. An
attempt to write to the shared pages generates a minor page fault. In the page fault handler, the
hypervisor transparently creates a private copy of the page for the VM and remaps the affected
guest physical page to this private copy. In this way, VMs can safely modify the shared pages
without disrupting other VMs sharing that memory. Writing to a shared page incurs overhead
compared to writing to nonshared pages because of the extra work performed in the page fault
handler.

480 Module 6: Memory Optimization

6-14 Transparent Page Sharing (2)

For information about the security risks associated with inter-VM TPS, see VMware knowledge
base article 2080735 at http://kb.vmware.com/kb/2080735.

Module 6: Memory Optimization 481

6-15 Impact of Hardware MMU on Transparent Page
Sharing

In modern x86-64 systems, the memory management unit (MMU) is integrated into the central
processing unit (CPU). MMU is a computer hardware unit having all memory references passed
through itself, primarily performing the translation of virtual memory addresses to physical
addresses. The translation lookaside buffer (TLB) caches the most recent translations of virtual-to-
physical memory addresses.
Intel EPT Hardware Assist and AMD RVI Hardware Assist are the second generation of MMU.
When using EPT or RVI, ESXi backs guest physical pages with large host physical pages. These
physical pages have a 2 MB contiguous memory region, instead of 4 KB, for regular pages for
better performance due to fewer translation lookaside misses.

482 Module 6: Memory Optimization

In such systems, ESXi does not share those large pages for the following reasons:

• The probability of finding two large pages with identical contents is low.

• The overhead of doing a bit-by-bit comparison for a 2 MB page is much larger than that for a
4 KB page.

When using EPT or RVI, the esxtop tool might show zero or few shared pages, because TPS uses
small pages (4 KB) and EPT and RVI use large pages (2 MB). An ESXi host tracks what pages
can be shared. If memory resources become overcommitted, ESXi breaks the large pages into
small pages and begins sharing memory between VMs.

Module 6: Memory Optimization 483

6-16 Impact of NUMA Architecture on Transparent Page
Sharing

484 Module 6: Memory Optimization

6-17 Transparent Page Sharing and Intra-VM Memory
Sharing

Module 6: Memory Optimization 485

6-18 Using Salting with Transparent Page Sharing (1)

The concept of salting was introduced to address security concerns related to using TPS. With
salting, VMs can share pages only when the pages are identical and the salt values for the VMs are
the same.
Intra-VM TPS is always enabled. The Mem.ShareForceSalting host configuration option is used
to force the concept of salting. Each VM can have a unique salt value set in the VM’s .vmx file
called sched.mem.pshare.salt. When salting is enforced, VMs with unique salt values
have inter-VM TPS disabled. However, any VMs with the same salt value configured have inter-
VM TPS enabled as a group.
For more information about TPS capabilities, see VMware knowledge base article 2097593 at
http://kb.vmware.com/kb/2097593.

486 Module 6: Memory Optimization

6-19 Using Salting with Transparent Page Sharing (2)

Module 6: Memory Optimization 487

6-20 Using Salting with Transparent Page Sharing (3)

When Mem.ShareForceSalting is set to 0, salting is not enforced and all VMs have inter-VM TPS
enabled in the host.

488 Module 6: Memory Optimization

6-21 Using Salting with Transparent Page Sharing (4)

When Mem.ShareForceSalting is set to 1, salting is enforced for only the VMs that have their salt
parameter configured. Because VMs do not have a salt parameter by default, most VMs have
inter-VM TPS enabled. Only the VMs with a unique salt value set have inter-VM TPS disabled.

Module 6: Memory Optimization 489

6-22 Using Salting with Transparent Page Sharing (5)

The default setting is when Mem.ShareForceSalting is set to 2. Salting is enforced and uses a
different VM parameter to determine if inter-VM TPS will be enabled. The vc.uuid parameter
is a unique value set by vCenter Server when the VM is created. Because vc.uuid is unique for
every VM, inter-VM TPS is disabled for all VMs by default. Only VMs that have the
sched.mem.pshare.salt set to the same value will have inter-VM TPS enabled. In this
case, the sched.mem.pshare.salt value overrides the vc.uuid setting and permits
different groups of VMs with inter-VM TPS enabled for only their group. If a group of VMs can
be trusted to share pages, they can be assigned a common salt value.

490 Module 6: Memory Optimization

6-23 Memory Ballooning in the Guest OS

The vmmemctl driver is loaded with VMware Tools. The driver uses a proprietary ballooning
technique that provides predictable performance that closely matches the behavior of a native
system under similar memory constraints. This technique increases or decreases memory pressure
on the guest OS, causing the guest to use its own native memory management algorithms. When
memory is tight, the guest OS determines which pages to reclaim and, if necessary, swaps them to
its own swap file, for example, to pagefile.sys in Windows.
The diagram shows the following stages of operation:
1. Under normal operation, the VMkernel does not need to reclaim memory from the VM.
Memory pressure does not exist and ballooning is not active.
2. The balloon driver artificially creates memory pressure on the VM. The guest OS responds to
the memory pressure by swapping out the optimal pages according to its analysis.
3. Memory pressure is relieved, ballooning is inactive, and the guest OS swaps pages into
memory as needed.

Module 6: Memory Optimization 491

If necessary, you can limit the amount of memory that vmmemctl reclaims by setting the
sched.mem.maxmemctl parameter for a specific VM. This option specifies the maximum
amount of memory that can be reclaimed from a VM in megabytes.

492 Module 6: Memory Optimization

6-24 Reclaiming Memory with Ballooning

Due to the VM’s isolation, the guest OS is unaware that it is running in a VM and unaware of the
states of other VMs on the same host. When the hypervisor runs multiple VMs, the total amount of
free host physical memory might become low. Because a guest OS cannot detect the host physical
memory shortage, none of the VMs free guest physical memory.
The guest OS determines whether it must page out guest physical memory to satisfy the balloon
driver’s allocation requests. If the VM has plenty of free or idle guest physical memory, inflating
the balloon does not induce guest-level paging and does not affect guest performance. However, if
the guest is already under memory pressure, the guest OS decides which guest physical pages are
to be paged to satisfy the balloon driver’s allocation requests.

Module 6: Memory Optimization 493

6-25 Memory Compression

The compression cache is located in the VM’s memory space. Memory compression outperforms
host swapping because the next access to the compressed page causes only a page decompression,
which can be significantly faster than the disk access. ESXi determines whether a page can be
compressed by verifying the compression ratio for the page. Memory compression occurs when
the page’s compression ratio is greater than 50 percent. Otherwise, the page is swapped out. Only
pages that are to be swapped out are chosen as candidates for memory compression. This
specification means that ESXi does not compress guest pages when host swapping is unnecessary.
Memory compression does not affect workload performance when host memory is not
overcommitted.
For this example, assume that ESXi must reclaim 8 KB physical memory (that is, two 4 KB pages)
from a VM. With memory compression, a swap candidate page is compressed and stored using 2
KB of space in a per-VM compression cache. Each compressed page yields 2 KB memory space
for ESXi to reclaim. To reclaim 8 KB of physical memory, four swap candidate pages must be
compressed. The page compression is much faster than the normal page swap-out operation,
which involves a disk I/O.

494 Module 6: Memory Optimization

The compression cache size starts at 0 and, by default, can increase to a maximum of 10 percent of
the VM’s memory size. You can control the cache size by using the Mem.MemZipMaxPct ESXi
advanced system setting. Because this setting is applied at the host level, it affects all VMs
running on that host.

Module 6: Memory Optimization 495

6-26 Reclaiming Memory with Host Swapping

When starting a VM, the hypervisor creates a separate swap VSWP file for the VM to support
host-level swapping. Then, if necessary, the hypervisor can directly swap out guest physical
memory to the swap file, which frees host physical memory for other VMs.
Host-level or hypervisor swapping is a guaranteed technique to reclaim a specific amount of
memory within a specific amount of time. However, host-level swapping might severely penalize
guest performance. Penalization occurs when the hypervisor has no knowledge of which guest
physical pages should be swapped out and the swapping might cause unintended interactions with
the native memory management policies in the guest OS. For example, the guest OS will never
page out its kernel pages because those pages are critical to ensure guest kernel performance.
However, the hypervisor cannot identify those guest kernel pages, so it might swap out those
physical pages.
ESXi swapping is distinct from the swapping performed by a guest OS due to memory pressure in
a VM. Guest OS level swapping might occur even when the ESXi host has ample resources.

496 Module 6: Memory Optimization

6-27 Activity: Identifying Memory Reclamation Features (1)

Module 6: Memory Optimization 497

6-28 Activity: Identifying Memory Reclamation Features (2)

498 Module 6: Memory Optimization

6-29 Sliding Scale Mem.MemMinFreePct

ESXi uses a sliding scale for determining the Mem.MemMinFreePct threshold. By using a sliding
scale, a sensible value is calculated regardless of the memory size of the ESXi host. For example,
assume that a server is configured with 128 GB of RAM. With the sliding-scale technique, the
scale allocates 245.76 MB of RAM or 6 percent of the memory, up to 4 GB. The scale then
allocates 327.68 MB of memory out of the 4 GB through 12 GB range, 327.68 MB out of the 12
GB through 28 GB range, and 1024 MB out of the remaining memory, for a total of 1925.12 MB
to calculate the Mem.MemMinFreePct value.

Module 6: Memory Optimization 499

6-30 Criteria for Reclaiming Host Memory

ESXi enables page sharing by default and reclaims host physical memory with little overhead.
In the high state, the aggregate VM guest memory use is less than the host physical memory size.
Normal TPS occurs.
The clear memory state was introduced in vSphere 6.0. In this state, large pages are broken into
smaller pages and TPS is actively called to collapse pages.
If the host free memory drops below the soft threshold, the hypervisor starts to reclaim memory by
using ballooning. Ballooning happens before free memory reaches the soft threshold because it
takes time for the balloon driver to allocate guest physical memory. Usually, the balloon driver
can reclaim memory in a timely fashion so that the host free memory stays above the soft
threshold.
If ballooning is insufficient to reclaim memory or the host free memory drops below the hard
threshold, the hypervisor starts to use memory compression and swapping. Through memory
compression, the hypervisor should be able to quickly reclaim memory and bring the host memory
state back to the soft state.

500 Module 6: Memory Optimization

If the host free memory drops below the low threshold, the hypervisor reclaims memory through
compression and swapping. The hypervisor also blocks the execution of all VMs that consume
more memory than their target memory allocations.

Module 6: Memory Optimization 501

6-31 Transitioning from One Memory State to Another (1)

In this example, the ESXi host transitions from High state to Clear state when memory usage is
greater than 125,294 MB.
This value is calculated in the following way:

• The host has 128 GB, or 131,072 MB, of RAM. Using the transition threshold for Clear state,
the value of 300% of MinFree is 3 * 1,926 MB, which equals 5,778 MB. Memory usage is
131,072 MB minus 5,778 MB, which equals 125,294 MB.

ESXi transitions from Clear state to Soft state when memory usage is greater than 129,839 MB.
This value is calculated using the transition threshold of 64% of MinFree:

• 0.64 multiplied by 1,926 MB equals 1,233 MB.

131,072 MB minus 1,233 MB equals 129,839 MB.

502 Module 6: Memory Optimization

ESXi transitions from Soft state to Hard state when memory usage is greater than 130,456 MB.
This value is calculated using the transition threshold of 32% of MinFree:

• 0.32 multiplied by 1,926 MB equals 616 MB.

131,072 MB minus 616 MB equals 130,456 MB.

ESXi transitions from Hard state to Low state when memory usage is greater than 130,764 MB.
This value is calculated using the transition threshold of 16% of MinFree:

• 0.16 multiplied by 1,926 MB equals 308 MB.

131,072 MB minus 308 MB equals 130,764 MB.

For a detailed discussion on memory reclamation, see VMware vSphere 6.5 Host Resources Deep
Dive on the Frank Denneman website at https://frankdenneman.nl/.

Module 6: Memory Optimization 503

6-32 Transitioning from One Memory State to Another (2)

In this example, the ESXi host transitions from Low state to Hard state when memory usage is less
than 130,610 MB. This value is calculated using the transition threshold of 24% of MinFree:

• 0.24 multiplied by 1,926 MB equals 462 MB.

131,072 MB minus 462 MB equals 130,610 MB.

ESXi transitions from Hard state to the Soft state when memory usage is less than 130,148 MB.
This value is calculated using the transition threshold of 48% of MinFree:

• 0.48 multiplied by 1,926 MB equals 924 MB.

131,072 MB minus 924 MB equals 130,148 MB.

ESXi transitions from Soft state to the Clear state when memory usage is less than 129,146 MB.
This value is calculated using the transition threshold of 100% of MinFree:

• 1 multiplied by 1,926 MB equals 1,926 MB.

131,072 MB minus 1,926 MB equals 129,146 MB.

504 Module 6: Memory Optimization

ESXi transitions from Clear state to the High state when memory usage is less than 123,368 MB.
This value is calculated using the transition threshold of 400% of MinFree:

• 4 multiplied by 1,926 MB equals 7,704 MB.

131,072 MB minus 7,704 MB equals 123,368 MB.

For a detailed discussion on memory reclamation, see VMware vSphere 6.5 Host Resources Deep
Dive on the Frank Denneman website at https://frankdenneman.nl/.

Module 6: Memory Optimization 505

6-33 Review of Learner Objectives

506 Module 6: Memory Optimization

6-34 Lesson 2: Monitoring Memory Activity

Module 6: Memory Optimization 507

6-35 Learner Objectives

508 Module 6: Memory Optimization

6-36 Memory Usage Metrics in the Guest OS

For guest memory usage, the values reported in vSphere Web Client might be different from the
active memory usage reported by the guest OS. The first reason for this difference is that the guest
OS generally has a better idea than the hypervisor of what memory is active. The guest OS knows
what applications are running and how it has allocated memory. The second reason is that the
method employed by the hypervisor to estimate active memory usage takes time to converge. So
the guest OS’s estimate might be more accurate than the ESXi host’s if the memory workload is
fluctuating.
For host memory usage, the host memory usage metric has no meaning inside the guest OS. It has
no meaning because the guest OS does not know that it is running in a VM or that other VMs exist
on the same physical host.

Module 6: Memory Optimization 509

6-37 About Consumed Host Memory and Active Guest
Memory

Consumed host memory is greater than active guest memory because, for physical hosts that are
not overcommitted on memory, consumed host memory represents the highest amount of memory
usage by a VM. In the past, this VM might have actively used a large amount of memory.
Because the host physical memory is not overcommitted, the hypervisor has no reason to invoke
ballooning or host-level swapping to reclaim memory. Therefore, you can find cases where the
active guest memory use is low but the amount of host physical memory assigned to it is high.
This combination is a normal situation.
Consumed host memory might be less than or equal to active guest memory because the active
guest memory of a VM does not completely reside in the host physical memory. Consumed host
memory might be less if a guest’s active memory is reclaimed by the balloon driver or if the VM
is swapped out by the hypervisor. Either case of lower consumed host memory is probably due to
high memory overcommitment.

510 Module 6: Memory Optimization

6-38 Using esxtop to Monitor Memory Usage

The following metrics are useful in monitoring memory usage:

• PMEM: The total amount of physical memory on your host in megabytes.

• VMKMEM: The memory that is managed by the VMkernel in megabytes.

• PSHARE: The saving field shows you how much memory you saved because of TPS. In the
example, two VMs are running with a memory savings of 62 MB.

• Memory state: This value can be high, clear, soft, hard, or low. This value refers to the current
state of the VMkernel’s memory. This value also indicates whether the VMkernel has enough
free memory to perform its critical operations.
If the state is high, the VMkernel has sufficient memory to perform critical tasks. However, if
the state is clear, soft, hard, or low, the VMkernel is unable to maintain adequate free
memory.
Typically, you should see only a state of high, even if you are overcommitting your memory.

Module 6: Memory Optimization 511

 ESXi employs efficient mechanisms for reserving memory for the VMkernel. However, if
your host is severely overcommitted, you might see a state other than high.

• A PCI hole is the memory reserved by the PCI devices on the host. This memory is used
solely by the PCI devices for I/O purposes, such as DMA ring buffers. This memory cannot
be used by the VMkernel. Reserving memory by PCI devices is common in all operating
systems. The amount of memory used for PCI devices is part of the other value, in the
PMEM/MB line.

512 Module 6: Memory Optimization

6-39 Host Ballooning Activity in esxtop (1)

The following metrics are useful in monitoring ballooning activity:

• MEMCTL/MB: This line displays the memory balloon statistics for the entire host.
All numbers are in megabytes:

• Curr: The total amount of physical memory reclaimed using the balloon driver
(vmmemctl).

• Target: The total amount of physical memory that ESXi wants to reclaim with the balloon
driver.

• Max: The maximum amount of physical memory that the ESXi host can reclaim with the
balloon driver.

• MCTL?: This value, which is either Y (for yes) or N (for no), indicates whether the balloon
driver is installed in the VM.

Module 6: Memory Optimization 513

• MCTLSZ: This value is reported for each VM and represents the amount of physical memory
that the balloon driver is holding for use by other VMs.

• MCTLTGT: This value is the amount of physical memory that the host wants to reclaim from
the VM through ballooning.

514 Module 6: Memory Optimization

6-40 Host Ballooning Activity in esxtop (2)

The example shows why ballooning is preferable to swapping. Multiple VMs are running, and all
the VMs are running a memory-intensive application. The MCTL? field shows that VMs named
Workload01 and Workload02 do not have the balloon driver installed. The balloon driver is
installed on all the other VMs. If the balloon driver is not installed, typically VMware Tools was
not installed on the VM.
The VMs with no balloon driver installed usually have a higher swap target (SWTGT) than the
VMs that have the balloon driver installed. VMs that have the balloon driver installed can reclaim
memory by using the balloon driver. If ESXi cannot reclaim memory through ballooning, it must
resort to other methods, such as host-level swapping.

Module 6: Memory Optimization 515

6-41 Host Memory Compression Activity in esxtop

ESXi provides a memory compression cache to improve VM performance when you use memory
overcommitment. Memory compression is enabled by default. When a host’s memory becomes
overcommitted, ESXi compresses virtual pages and stores them in memory.
Because accessing compressed memory is faster than accessing memory that is swapped to disk,
memory compression in ESXi enables you to overcommit memory without significantly hindering
performance. When a virtual page must be swapped, ESXi first tries to compress the page. Pages
that can be compressed to 2 KB or smaller are stored in the VM’s compression cache, increasing
the capacity of the host.
You can monitor the following values in esxtop:

• CACHESZ (MB): Compression memory cache size

• CACHEUSD (MB): Used compression memory cache

516 Module 6: Memory Optimization

• ZIP/s (MB/s): Compressed memory per second

• UNZIP/s (MB/s): Decompressed memory per second

Module 6: Memory Optimization 517

6-42 Monitoring Host Cache Swapping

In esxtop, LLSWR/s is the rate (in MB) at which memory is read from the host cache. LLSWW/s
is the rate (in MB) at which memory is written to the host cache.

518 Module 6: Memory Optimization

6-43 Host Swapping Activity in esxtop: Memory Screen

A useful metric in the memory screen is SWAP/MB. This metric represents total swapping for all
the VMs on the host. In the example, the value of curr is 1791 MB, which means that 1,791 MB of
swap space is currently used. The rclmtgt value (1994 in the example) is where the ESXi system
expects the reclaimed memory to be. Memory can be reclaimed by swapping or by compression.
To monitor host-level swapping activity per VM, enter uppercase V in the esxtop window to
display only the VMs.
The following metrics are useful:

• SWR/s and SWW/s: Measured in megabytes, these counters represent the rate at which the
ESXi host is swapping memory in from disk (SWR/s) and swapping memory out to disk
(SWW/s).

• SWCUR: The amount of swap space currently used by the VM.

• SWTGT: The amount of swap space that the host expects the VM to use.

Module 6: Memory Optimization 519

6-44 Host Swapping Activity in esxtop: CPU Screen

The CPU screen has a metric named %SWPWT. This metric is the best indicator of a performance
problem due to wait time experienced by the VM. This metric represents the percentage of time
that the VM is waiting for memory pages to be swapped in.
The ready time (%RDY) is low because no competition exists for the CPU resource until the
pages are swapped in.

520 Module 6: Memory Optimization

6-45 Causes of Active Host-Level Swapping

The following conditions can cause host-level swapping:

• Excessive memory overcommitment: In this condition, the level of memory overcommitment

is too high for the combination of memory capacity, VM allocations, and VM workload. In
addition, the amount of overhead memory must be considered when using memory
overcommitment.

• Memory overcommitment with memory reservations: In this condition, the total amount of
balloonable memory might be reduced by memory reservations. If a large portion of a host’s
memory is reserved for some VMs, the host might be forced to swap the memory of other
VMs. Swapping memory with other VMs might occur even though VMs are not actively
using all their reserved memory.

• Balloon drivers in VMs not running or disabled: In this condition, if the balloon driver is not
running or was deliberately disabled in some VMs, the amount of balloonable memory on the

Module 6: Memory Optimization 521

 host will be decreased. As a result, the host might be forced to swap even though idle memory
is available that might have been reclaimed through ballooning.

522 Module 6: Memory Optimization

6-46 Activity: Identifying esxtop Fields (1)

Module 6: Memory Optimization 523

6-47 Activity: Identifying esxtop Fields (2)

524 Module 6: Memory Optimization

6-48 Resolving Host-Level Swapping

In most situations, reducing memory overcommitment levels is the correct approach for
eliminating swapping on an ESXi host. However, you should consider as many factors as possible
to ensure that the reduction is adequate to eliminate swapping. If other approaches are used,
monitor the host to ensure that swapping was eliminated.

Module 6: Memory Optimization 525

6-49 Reducing Memory Overcommitment

To reduce the level of memory overcommitment, perform one or more of the following actions:

• Add physical memory to the ESXi host: Adding physical memory to the host reduces the level
of memory overcommitment and might eliminate the memory pressure that caused swapping
to occur.

• Reduce the number of VMs running on the ESXi host: One way to reduce the number of VMs
is to use vSphere vMotion to migrate VMs to hosts with available memory resources. To use
vSphere vMotion successfully, you should use the vSphere Web Client performance charts to
determine the memory usage of each VM on the host and the available memory resources on
the target hosts. You must ensure that the migrated VMs will not cause swapping to occur on
the target hosts.

526 Module 6: Memory Optimization

 If additional ESXi hosts are not available, you can reduce the number of VMs by powering
off noncritical VMs. Reducing the number of running VMs makes additional memory
resources available for critical applications. Even idle VMs consume some memory resources.

• Increase available memory resources by adding the host to a vSphere DRS cluster: Using a
vSphere DRS cluster is similar to the previous solution. However, in a vSphere DRS cluster,
load rebalancing can be performed automatically. You do not have to manually compute the
compatibility of specific VMs and hosts or account for peak usage periods.

Module 6: Memory Optimization 527

6-50 Enabling the Balloon Driver in VMs

If a VM has critical memory needs, reservations and other resource controls should be used to
ensure that those needs are satisfied. The balloon driver is enabled when VMware Tools is
installed on the VM.
However, if excessive memory overcommitment exists, ballooning only delays the onset of
swapping. Ballooning is therefore not a sufficient solution. The level of memory overcommitment
should be reduced as well.
The balloon driver should never be deliberately disabled in a VM. Disabling the balloon driver
might cause unintended performance problems, such as host-level swapping. It also makes
tracking down memory-related problems more difficult. vSphere provides other mechanisms, such
as memory reservations, for controlling the amount of memory available to a VM.

528 Module 6: Memory Optimization

6-51 Reducing VM Memory Reservation

Module 6: Memory Optimization 529

6-52 Dedicating Memory to Critical VMs

When reducing the level of memory overcommitment is impossible, configure your performance
critical VMs with sufficient memory reservations to prevent them from swapping.
However, using memory reservations only moves the swapping problem to other VMs whose
performance will be severely degraded. In addition, the swapping of other VMs might still affect
the performance of VMs with reservations, because of the added disk traffic and memory
management overhead. This approach should be used only with caution when no other options
exist.

530 Module 6: Memory Optimization

6-53 Introduction to Lab: Monitoring Memory Performance

In this lab, you perform two tests:

• Test case 1: Run starttest2 on your test VM and record baseline performance data.

• Test case 2: Start the ResourceHog01 and ResourceHog02 VMs to generate memory
contention. After the VMs start up, record performance data.

Answer questions posed in the lab and compare the baseline data with the performance data after
memory contention is generated.

Module 6: Memory Optimization 531

6-54 Lab 17: Monitoring Memory Performance

532 Module 6: Memory Optimization

6-55 Review of Lab: Monitoring Memory Performance

Module 6: Memory Optimization 533

6-56 Review of Learner Objectives

534 Module 6: Memory Optimization

6-57 Key Points

Module 6: Memory Optimization 535

536 Module 6: Memory Optimization
Module 7
Storage Optimization

Module 7: Storage Optimization 537

7-2 Importance

538 Module 7: Storage Optimization

7-3 Module Lessons

Module 7: Storage Optimization 539

7-4 Lesson 1: Storage Virtualization Concepts

540 Module 7: Storage Optimization

7-5 Learner Objectives

Module 7: Storage Optimization 541

7-6 Storage Performance Overview

ESXi enables multiple hosts to share physical storage using its optimized storage stack. Shared
storage of VMs uses VMFS, NFS, vSAN, and vSphere Virtual Volumes. Shared storage enables
vSphere vMotion, vSphere DRS, and vSphere HA.
To get the most from shared storage, you must understand the storage performance limits of a
given physical environment. Understanding limits helps ensure that you do not overcommit
resources.

542 Module 7: Storage Optimization

7-7 Storage Protocol Performance

When using Fibre Channel, Fibre Channel over Ethernet (FCoE), and hardware iSCSI, a major
part of the protocol processing is offloaded to the host bus adapter (HBA). Therefore, the cost of
each I/O is very low.
For software iSCSI and NFS, host CPUs are used for protocol processing, which increases cost.
Furthermore, the cost of NFS and software iSCSI is higher with larger block sizes, such as 64 KB.
This higher cost is because of the additional CPU cycles needed for each block for tasks, such as
check summing and blocking. Software iSCSI and NFS are more efficient at smaller blocks. Both
can deliver high throughput performance when CPU resource is not causing a bottleneck.

Module 7: Storage Optimization 543

7-8 About SAN Configuration

Storage performance depends on workload, hardware, vendor, RAID level, cache size, stripe size,
and so on. Consult VMware documentation and storage vendor documentation for more
information about configuring your storage devices.
Because each application running in your vSphere environment has different requirements, you
can improve throughput and minimal latency by selecting the appropriate RAID level and path
selection policy.
By default, active-passive storage arrays use the Most Recently Used path policy. To avoid LUN
thrashing, do not use the Fixed path policy for active-passive storage arrays.
By default, active-active storage arrays use the Fixed path policy. When using this policy, you can
maximize the use of your bandwidth to the storage array by designating preferred paths to each
LUN through different storage controllers.

544 Module 7: Storage Optimization

The Round Robin policy uses an automatic path selection by rotating through all available paths
and enabling the distribution of load across the configured paths:

• For active-passive storage arrays, only the paths to the active controller are used in the Round
Robin policy.

• For active-active storage arrays, all paths are used in the Round Robin policy.

For more information, see vSphere Storage at https://docs.vmware.com/en/VMware-

vSphere/7.0/vsphere-esxi-vcenter-server-70-storage-guide.pdf.
For more information on path selection policies, see Viewing and Managing Paths at
https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.storage.doc/GUID-
98F0851E-12F2-4FB2-A794-7643DD2E6F9B.html.

Module 7: Storage Optimization 545

7-9 About Storage Queues

The device driver queue is used for low-level interaction with the storage device. This queue
controls how many active commands can be on a LUN at the same time. This number is
effectively the concurrency of the storage stack. Set the device queue to 1 and each storage
command becomes sequential, that is, each command must complete before the next command
starts.
The kernel queue can be thought of as an overflow queue for the device driver queues. A kernel
queue includes features that optimize storage. These features include multipathing for failover and
load balancing, prioritization of storage activities based on VM and cluster shares, and
optimizations to improve efficiency for long sequential operations.
SCSI device drivers have a configurable parameter, called the LUN queue depth, that determines
how many commands can be active at a time on a given LUN. If the total number of outstanding
commands from all VMs exceeds the LUN queue depth, the excess commands are queued in the
ESXi kernel, which increases latency.
In addition to LUN queue depth, command queuing can also occur at the storage array.

546 Module 7: Storage Optimization

7-10 Network Storage: iSCSI and NFS

Applications and systems, such as data acquisition or transaction logging systems, perform best
with multiple connections to storage devices.
For iSCSI and NFS, design your network topology to prevent Ethernet bottlenecks. Bottlenecks
where multiple links are routed through fewer links can result in oversubscription and dropped
network packets. When the number of links transmitting near capacity get switched to a smaller
number of links, such oversubscription becomes possible. Recovering from dropped network
packets results in large performance degradation.
In shared configurations, using VLANs or VPNs is not a suitable solution to the problem of link
oversubscription. However, separating VLANs for NFS and iSCSI minimizes network
interference from other packet sources.
Finally, with software-initiated iSCSI and NFS, the network protocol processing takes place on the
host system and therefore requires more CPU resources than other storage options.

Module 7: Storage Optimization 547

7-11 Review of Learner Objectives

548 Module 7: Storage Optimization

7-12 Lesson 2: Enhancements to NVMe Storage Systems

Module 7: Storage Optimization 549

7-13 Learner Objectives

550 Module 7: Storage Optimization

7-14 Enhanced Support for NVMe

ESXi uses Pluggable Storage Architecture (PSA) as a VMkernel layer for managing storage
multipathing.
The PSA is an open and modular framework. It coordinates software modules that are responsible
for multipathing operations.
Modules include generic multipathing modules (MPPs) that VMware provides, Native
Multipathing Plug-In (NMP) and High-Performance Plug-In (HPP), and third-party MPPs.

Module 7: Storage Optimization 551

7-15 Supported NVMe Topologies

552 Module 7: Storage Optimization

7-16 FC-NVMe and NVMe over RDMA (RoCE v2)

Module 7: Storage Optimization 553

7-17 Support for FC-NVMe

Your ESXi host detects the device and displays it in the vSphere Client as a standard Fibre
Channel adapter (vmhba) with the storage protocol indicated as NVMe.
You do not need to configure the controller. After you install the required hardware NVMe
adapter, it automatically connects to all targets and controllers that it can reach.
You can later reconfigure the adapter and disconnect its controllers or connect other controllers
that were not available during the host boot.

554 Module 7: Storage Optimization

7-18 Requirements for FC-NVMe

For more information about NVMe concepts, see About VMware NVMe Storage chapter in the
vSphere Storage documentation at https://docs.vmware.com/en/VMware-vSphere/7.0/vsphere-
esxi-vcenter-server-70-storage-guide.pdf.

Module 7: Storage Optimization 555

7-19 Support for NVMe over RDMA (RoCE v2)

NVMe over RDMA with RoCE v2 technology provides ultra-efficient storage to optimize
performance-sensitive workloads over your existing Ethernet fabric.
You must configure both adapters to use them for storage discovery. The adapter configuration
process involves setting up VMkernel binding for an RDMA network adapter and enabling a
software NVMe over RDMA adapter.
For more information about configuring adapters for NVMe over RDMA (RoCE v2) storage, see
Configure Adapters for NVMe over RDMA (RoCE v2) Storage at
https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.storage.doc/GUID-
293BC371-B260-4CDD-9BB3-94E834851689.html.

556 Module 7: Storage Optimization

7-20 Requirements for NVMe over RDMA (RoCE v2)

Module 7: Storage Optimization 557

7-21 Configuring ESXi for NVMe over RDMA (RoCE v2) (1)

558 Module 7: Storage Optimization

7-22 Configuring ESXi for NVMe over RDMA (RoCE v2) (2)

Module 7: Storage Optimization 559

7-23 Configuring ESXi for NVMe over RDMA (RoCE v2) (3)

560 Module 7: Storage Optimization

7-24 Configuring ESXi for NVMe over RDMA (RoCE v2) (4)

The software NVMe over RDMA adapter appears on the list as a vmhba storage adapter. You can
remove the adapter if you must free the underlying RDMA network adapter for other purposes.
For more information about configuring adapters for NVMe over RDMA (RoCE v2) storage, see
Configure Adapters for NVMe over RDMA (RoCE v2) Storage at
https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.storage.doc/GUID-
293BC371-B260-4CDD-9BB3-94E834851689.html.

Module 7: Storage Optimization 561

7-25 Pluggable Storage Architecture

VMware provides generic native multipathing modules, called VMware NMP and VMware HPP.
In addition, the PSA offers a collection of VMkernel APIs that third-party developers can use.
Software developers can create their own load balancing and failover modules for a particular
storage array. These third-party multipathing modules (MPPs) can be installed on the ESXi host
and run in addition to the VMware native modules, or as their replacement.
As the diagram on the slide shows, multiple third-party MPPs can run in parallel with the VMware
NMP or HPP.
When installed, the third-party MPPs can replace the behavior of the native modules. The MPPs
can take control of the path failover and the load-balancing operations for the specified storage
devices.

562 Module 7: Storage Optimization

7-26 High-Performance Plug-In for NVMe

In vSphere 7, HPP integrates path selection and failover logic within a single module, reducing
cross-module calls.
To support multipathing, HPP uses a Path Selection Scheme (PSS) when selecting physical paths
for I/O requests. You can use vSphere ESXi Shell or the vSphere CLI to configure the HPP and
PSS.
For local NVMe devices, NMP remains the default plug-in, but you can replace it with HPP.
For more information about configuring and using HPP, see VMware High Performance Plug-In
and Path Selection Schemes at https://docs.vmware.com/en/VMware-
vSphere/7.0/com.vmware.vsphere.storage.doc/GUID-F7B60A5A-D077-4E37-8CA7-
8CB912173D24.html.

Module 7: Storage Optimization 563

7-27 Support for High-Performance Plug-In Multipathing

564 Module 7: Storage Optimization

7-28 High-Performance Plug-In Best Practices

Do not activate the HPP for HDDs or slower flash devices. The HPP does not provide any
performance benefits with devices incapable of at least 200,000 IOPS.
If you do not attach the disks to separate virtual controllers in the VM, I/O throughput might be
limited. This limitation is due to saturation of the CPU core that is responsible for processing I/O
operations on a virtual storage controller.
For information about device identifiers for NVMe devices that support only NGUID ID format,
see NVMe Devices with NGUID Device Identifiers at https://docs.vmware.com/en/VMware-
vSphere/7.0/com.vmware.vsphere.storage.doc/GUID-E38066BA-8AC1-4A5D-8E2C-
EB24FCC8E9E1.html.

Module 7: Storage Optimization 565

7-29 Review of Learner Objectives

566 Module 7: Storage Optimization

7-30 Lesson 3: Support for iSER

Module 7: Storage Optimization 567

7-31 Learner Objectives

568 Module 7: Storage Optimization

7-32 Support for iSER-Enabled vSphere Virtual Volumes
Arrays

Module 7: Storage Optimization 569

7-33 Configuring iSER

When installed on the host, the RDMA-capable adapter appears in vCenter Server as a network
adapter (vmnic).
To make the adapter functional, you must enable the VMware iSER component and connect the
iSER adapter to the RDMA-capable vmnic. You can then configure properties, such as targets and
CHAP, for the iSER adapter.
NOTE: iSER does not support NIC teaming. When configuring port binding, use only one
RDMA adapter per virtual switch.

570 Module 7: Storage Optimization

7-34 Requirements for iSER

To enable flow control, see the VMware knowledge base article 1013413 at
http://kb.vmware.com/kb/1013413.

Module 7: Storage Optimization 571

7-35 Running iSER

572 Module 7: Storage Optimization

7-36 Review of Learner Objectives

Module 7: Storage Optimization 573

7-37 Lesson 4: Monitoring Storage Activity

574 Module 7: Storage Optimization

7-38 Learner Objectives

Module 7: Storage Optimization 575

7-39 Where Storage Problems Can Occur

576 Module 7: Storage Optimization

7-40 Storage Key Performance Indicators

Several questions can be asked when looking at disk performance problems:

• Are you getting the expected I/O per second (IOPS)?

• Are you getting the expected bandwidth (read/write)?

The same questions apply to disk latency. Verify disk latency and compare it with your
expectations.
Are the latencies higher than you expected? Compare performance with the hardware
specifications of the particular storage subsystem.
Disk bandwidth and latency help determine whether storage is overloaded or slow.

Module 7: Storage Optimization 577

7-41 Monitoring Disk Throughput by Storage Adapter (1)

The esxtop window displays disk activity per disk adapter. To display the set of columns, enter f.
Ensure that the following columns are selected: A (adapter name), B (path name), C (number of
paths), E (I/O statistics), and G (overall latency statistics). If necessary, select columns to display
by entering a, b, c, e, or g.

The sum of reads per second and writes per second equals IOPS. IOPS is a common benchmark
for storage subsystems and can be measured with tools such as Iometer.
Disk throughput can be monitored using esxtop:

• READS/s: Number of disk reads per second

• WRITES/s: Number of disk writes per second

578 Module 7: Storage Optimization

If you prefer, you can monitor throughput by using the following metrics instead:

• MBREAD/s: Number of megabytes read per second

• MBWRTN/s: Number of megabytes written per second

All these metrics can be monitored per HBA (vmhba#).

CMDS/s shows the total amount of commands per second and includes IOPS (Input/Output
Operations Per Second). It also shows other SCSI commands such as SCSI reservations, locks,
vendor string requests, unit attention commands, and so on, being sent to or coming from the
device or VM being monitored.
In most cases, CMDS/s = IOPS, unless there are many metadata operations (such as SCSI
reservations).

Module 7: Storage Optimization 579

7-42 Monitoring Disk Throughput by Storage Adapter (2)

In this example, vmhba65 is entered in the esxtop window. The esxtop window displays disk/LUN
throughput information for each of the adapter’s paths.

580 Module 7: Storage Optimization

7-43 Correlating Storage Devices to Datastores (1)

Storage administrators usually keep a record (for example, in a spreadsheet) of each storage
device, its unique ID, and its use.
You can use the vSphere Client to correlate the storage device with a datastore. In the example,
the LUN identified as naa.60003ff44dc75adc92bc42d0a5cb5795 is a Microsoft iSCSI disk that
backs the OPSCALE-Datastore datastore.

Module 7: Storage Optimization 581

7-44 Correlating Storage Devices to Datastores (2)

You can also identify what ESXi hosts can access that particular LUN. In this example, the
naa.60003ff44dc75adc92bc42d0a5cb5795 LUN is accessed by the sa-esxi-01.vclass.local host.
You can view which ESXi hosts can access a particular datastore by selecting the datastore in the
Navigator pane and selecting Configure > Connectivity and Multipathing.

582 Module 7: Storage Optimization

7-45 Monitoring Path Statistics for a Specific Storage
Device (1)

Storage devices (or LUNs) are identified by a unique identifier, such as a Network Address
Authority ID (NAA ID) or a T10 identifier.
This esxtop window shows disk activity by storage device. To display the set of columns, enter f.

Ensure that the following columns are selected: A (device name), B (path, world, partition ID), F
(queue statistics), G (I/O statistics), and I (overall latency statistics). If necessary, select columns
to display by entering a, b, f, g, or i.

Module 7: Storage Optimization 583

7-46 Monitoring Path Statistics for a Specific Storage
Device (2)

In this example, naa.60003ff44dc75adc92bc42d0a5cb5795 is entered in the esxtop window.

esxtop displays path statistics for this particular device.

584 Module 7: Storage Optimization

7-47 Monitoring Individual VMDK Statistics for a VM (1)

This esxtop window shows disk activity per VM. To display the set of columns, enter f.
Ensure that the following columns are selected: B (group ID), C (VM name), D (virtual device
name), E (number of virtual disks), I (I/O statistics), J (read latency statistics), and K (write
latency statistics). If necessary, select columns to display by entering b, c, d, e, i, j, or k.

Module 7: Storage Optimization 585

7-48 Monitoring Individual VMDK Statistics for a VM (2)

In this example, the GID for the Linux01 VM (57941) is entered in the esxtop window. The esxtop
window displays disk throughput statistics for each VMDK connected to the VM.

586 Module 7: Storage Optimization

7-49 Disk Latency Metrics

Latency statistics are measured at the various layers of the ESXi storage stack.
GAVG is the round-trip latency that the guest OS sees for all I/O requests sent to the virtual
storage device.
KAVG tracks the latencies due to the ESXi VMkernel commands. The KAVG value should be
very small in comparison to the DAVG value and it should be close to zero. When much queuing
occurs in ESXi, KAVG can be as high as or higher than DAVG. If this situation occurs, verify the
queue statistics.
DAVG is the latency seen at the device driver level. It includes the round-trip time between the
HBA and the storage. DAVG is a good indicator of performance of the back-end storage. If I/O
latencies are suspected of causing performance problems, DAVG should be examined.
Compare I/O latencies with corresponding data from the storage array. If they are close, verify the
array for misconfiguration or faults. If they are not close, compare DAVG with corresponding data
from points in between the array and the ESXi host, for example, Fibre Channel switches. If this

Module 7: Storage Optimization 587

intermediate data also matches DAVG values, the storage is likely underconfigured for the
application. Adding disk spindles or changing the RAID level might help in such cases.
QAVG is the average queue latency. QAVG is part of KAVG. Response time is the sum of the
time spent in queues in the storage stack and the service time spent by each resource in servicing
the request. The largest component of the service time is the time spent retrieving data from
physical storage. If QAVG is high, consider examining the queue depths at each level in the
storage stack.

588 Module 7: Storage Optimization

7-50 Monitoring Disk Latency with esxtop

ADAPTR is the name of the host bus adapter (vmhba#), which includes SCSI, iSCSI, RAID, Fibre
Channel, and FCoE adapters.
DAVG/cmd is the average amount of time that it takes a device, which includes the HBA, the
storage array, and everything in between, to service a single I/O request (read or write). A value of
less than 10 indicates that the system is healthy. A value in the range of 11 through 20 indicates
that you must monitor the value more frequently. A value greater than 20 indicates a problem.
KAVG/cmd is the average amount of time that it takes the VMkernel to service a disk operation.
This number represents the time taken by the CPU to manage I/O. Because processors are much
faster than disks, this value should be close to zero. A value of 1 or 2 is considered high for this
metric.
GAVG/cmd is the total latency seen from the VM when performing an I/O request.
GAVG is the sum of DAVG plus KAVG.

Module 7: Storage Optimization 589

7-51 Monitoring Commands and Command Queuing

The metrics in the table provide information about your disk performance. They are often used to
further interpret the latency values that you might be observing:

• Number of active commands: This metric represents the number of I/O operations that are
currently active. This number includes operations for which the host is processing. This
metric can serve as a quick view of storage activity. If the value of this metric is close to or at
zero, the storage subsystem is not being used. If the value is a nonzero number, sustained over
time, then constant interaction with the storage subsystem is occurring.

• Number of commands queued: This metric represents the number of I/O operations that
require processing but have not yet been addressed. Commands are queued and awaiting
management by the kernel when the driver’s active command buffer is full. Occasionally, a
queue forms and results in a small, nonzero value for QUED. However, any significant
(double-digit) average of queued commands means that the storage hardware is unable to
keep up with the host’s needs.

590 Module 7: Storage Optimization

7-52 Example: Disk Latency and Queuing

This example shows monitoring the kernel latency value, KAVG/cmd. This value is being
monitored for the vmhba0 device. In the first esxtop window (enter d in the window), the kernel
latency value is 0.01 milliseconds. This value is good because it is nearly zero.
In the second esxtop window (enter u in the window), 32 active I/O operations (ACTV) and 2 I/O
operations are being queued (QUED). Some queuing is happening at the VMkernel level.
Queuing happens if I/O to the device is excessive and the LUN queue depth setting is insufficient.
The default LUN queue depth is 32. The default value depends on the HBA. QLogic HBAs have a
default queue depth of 64 in vSphere 5.x and 6.x.
If too many I/O operations (that is, more than 32) exist simultaneously, the device gets
bottlenecked to only 32 outstanding I/O operations at a time. To resolve this problem, you might
change the queue depth of the device driver.
For more information about changing the queue depth of the device driver, see About vSphere
Monitoring and Performance at https://docs.vmware.com/en/VMware-

Module 7: Storage Optimization 591

vSphere/7.0/com.vmware.vsphere.monitoring.doc/GUID-A8B06BE0-E5FC-435C-B12F-
A31618B21E2C.html.

592 Module 7: Storage Optimization

7-53 Example: Bad Disk Throughput

In the example, the first esxtop window (enter d in the window) shows the disk activity of a host.
Looking at the vmhba1 adapter, you see that very good throughput exists: 135.90 MB written per
second. The device latency per command is 7.22, which is low.
However, the second esxtop window shows low throughput (3.95 MB written per second) on
vmhba1 but very high device latency (252.04). In this configuration, the SAN cache was disabled.
The disk cache is important, especially for writes.
The use of caches applies not only to SAN storage arrays but also to RAID controllers. A RAID
controller typically has a cache with a battery backup. If the battery dies, the cache is disabled to
prevent any potential loss of data. Therefore, a problem might occur if you are experiencing very
good performance and the performance suddenly degrades. In this case, verify the battery status on
the RAID controller to see if the battery is still good.

Module 7: Storage Optimization 593

7-54 Activity: Hardware Disk Latency (1)

594 Module 7: Storage Optimization

7-55 Activity: Hardware Disk Latency (2)

Module 7: Storage Optimization 595

7-56 Overloaded Storage and Command Aborts

Severely overloaded storage can result from several problems in the underlying storage layout or
infrastructure. Overloaded storage can manifest itself in many ways, depending on the applications
running on the VMs. When storage is severely overloaded, operation timeouts can cause
commands that are already issued to disk to be terminated (or aborted).

596 Module 7: Storage Optimization

7-57 Unexpected Increase in I/O Latency on Shared
Storage

Sharing storage resources offers such advantages as ease of management, power savings, and
higher resource use. However, situations might occur when high I/O activity on the shared storage
might affect the performance of certain latency-sensitive applications:

• A higher-than-expected number of I/O-intensive applications in VMs sharing a storage device

become active at the same time. As a result, these VMs issue a very high number of I/O
requests concurrently. Increased I/O requests increases the load on the storage device, and the
response time of I/O requests also increases. The performance of VMs running critical
workloads might be affected because these workloads now must compete for shared storage
resources.

• Operations such as vSphere Storage vMotion migration or a backup operation running in a

VM use the I/O bandwidth of a shared storage device. These operations cause other VMs
sharing the device to suffer from resource starvation.

Module 7: Storage Optimization 597

Storage I/O Control can overcome these problems. It continuously monitors the aggregate
normalized I/O latency across the shared storage devices (VMFS and NFS datastores). Storage I/O
Control can detect the existence of an I/O congestion based on a preset congestion threshold. After
the condition is detected, Storage I/O Control uses the resource controls set on the VMs to control
each VM’s access to the shared datastore.
Also, vSphere DRS can automatically or manually balance datastore cluster I/O capacity and
performance. vSphere Storage DRS can perform automatic vSphere Storage vMotion migrations
of VMs. These migrations can occur when a datastore's threshold for space use or I/O response
time are exceeded.

598 Module 7: Storage Optimization

7-58 Storage Performance Best Practices

Module 7: Storage Optimization 599

7-59 Introduction to Lab: Monitoring Storage Performance

You generate various types of disk I/O and compare performance. Your VM is configured with a
system disk and two data disks. The system disk and one of the data disks are on the local
datastore. The other data disk is on the remote datastore.
You run the following scripts:

• fileserver1.sh: This script generates random reads to the local data disk.

• fileserver2.sh: This script generates random reads to the remote data disk.

• datawrite.sh: This script generates random writes to the remote data disk.

• logwrite.sh: This script generates sequential writes to the remote data disk.

Each script starts an application called aio-stress, which is a simple command-line

application that measures the performance of a disk subsystem. For more information about the
aio-stress command, see the readme.txt file on the test VM in the same directory as the
scripts.

600 Module 7: Storage Optimization

7-60 Lab 18: Monitoring Storage Performance

Module 7: Storage Optimization 601

7-61 Review of Lab: Monitoring Storage Performance

602 Module 7: Storage Optimization

7-62 Review of Learner Objectives

Module 7: Storage Optimization 603

7-63 Key Points

604 Module 7: Storage Optimization

Module 8
Network Optimization

Module 8: Network Optimization 605

8-2 Importance

606 Module 8: Network Optimization

8-3 Module Lessons

Module 8: Network Optimization 607

8-4 Lesson 1: Network Virtualization Concepts

608 Module 8: Network Optimization

8-5 Learner Objectives

Module 8: Network Optimization 609

8-6 Network I/O Virtualization Overhead

The overhead that VMs experience is due to packets traversing an extra layer of virtualization
stack. The network I/O latency due to the virtualization stack can come from several sources,
including the following most common sources:

• Emulation overhead: Certain privileged instructions and some operations to access I/O
devices that the VM executes are intercepted by the hypervisor. This activity adds some
overhead and contributes to network I/O latency.

• Packet processing: The network virtualization stack forwards a network packet from the
physical NIC to the VM, and the reverse. This activity requires some computation and
processing, such as switching decisions at the virtual switch, inserting and stripping the
VLAN tag, and copying packets if necessary. This processing adds some latency on both the
transmit and the receive paths.

• Scheduling: Packet transmission and reception involves multiple hypervisor threads and
virtual CPUs. The VMkernel scheduler has to schedule and then execute these threads, if they
are not already running, on receipt and transmission of a packet. On an idle system, this

610 Module 8: Network Optimization

 activity takes a couple of microseconds. On a busy system, if high CPU contention exists, this
activity can take tens of microseconds and, occasionally, milliseconds.

• Virtual interrupt coalescing: Similar to physical NIC interrupt coalescing, the VM does virtual
interrupt coalescing. That is, a VM might not be interrupted immediately after receiving or
transmitting a packet. Instead, the VMkernel might wait to receive or transmit more than one
packet before an interrupt is posted. Also, on the transmit path, the VM might wait until a few
packets are queued up before sending the packets down to the hypervisor. Sometimes
interrupt coalescing might have a noticeable effect on average latency.

• Halting and waking up the physical CPU: Modern hardware and operating systems put the
physical CPU in sleep mode (halt) when the CPU is idle. Waking up the CPU from halt mode
adds some latency. The amount of latency varies from CPU to CPU. Depending on the load of
the system, the CPU might go into deeper sleep states from which it takes even longer to
wake up. This overhead is generally common between nonvirtualized and virtualized
platforms.

Sometimes the side effects of this halting and waking can be worse in virtualized
environments than in physical environments. For example, a CPU might flush its cache on
entering halt mode. In this case, the halt operation might affect a virtualized system more than
a physical system because the cache footprint is larger in a virtual environment.

• Halting and waking up the virtual CPU: When the guest OS issues a halt instruction, the
VMkernel scheduler might deschedule the virtual CPU. Rescheduling the virtual CPU and
putting the virtual CPU back in the run queue can take a couple of microseconds.

Latency introduced due to the virtual or physical CPU halt function is more prominent when
the CPU is lightly loaded, that is, when the CPU halts frequently.

Module 8: Network Optimization 611

8-7 VMXNET Network Adapters

The VMXNET driver implements an idealized network interface that passes through network
traffic from the VM to the physical cards with minimal overhead.
The driver improves performance through several optimizations:

• It shares a ring buffer between the VM and the VMkernel. And it uses zero-copy which, in
turn, saves CPU cycles. Traditional networking uses a series of buffers to process incoming
network data and deliver it efficiently to users. However, higher-speed modern networks are
turning this approach into a performance bottleneck as the amount of data received from the
network often exceeds the size of the kernel buffers. Zero-copy improves performance by
having the VMs and the VMkernel share a buffer, reducing the internal copy operations
between buffers to free up CPU cycles.

• It uses transmission packet coalescing to reduce address space switching.

612 Module 8: Network Optimization

• It batches packets and issues a single interrupt, rather than issuing multiple interrupts.

• It offloads TCP checksum calculation to the network hardware instead of using the CPU
resources of the VM monitor.

Module 8: Network Optimization 613

8-8 Virtual Network Adapters

The following virtual NIC (vNIC) types are supported:

• Vlance: This vNIC is an emulated version of the AMD 79C970 PCnet32 LANCE NIC.
Vlance is an older 10 Mbps NIC, with drivers available in most 32-bit guest operating
systems. This type cannot be selected directly. You can configure it only by using the flexible
adapter type.

• VMXNET: This vNIC has no physical counterpart. VMXNET is optimized for performance
in a VM. This type cannot be selected directly. You can configure it only by using the flexible
adapter type.

• Flexible: This vNIC identifies itself as a Vlance adapter when a VM boots but it initializes
itself and functions as either a Vlance or a VMXNET adapter.

• E1000 and E1000E: The E1000 NIC is an emulated version of the Intel 82545EM Gigabit
Ethernet NIC. The E1000E is an emulated version of the Intel 82574L Gigabit Ethernet NIC.

614 Module 8: Network Optimization

• Enhanced VMXNET (VMXNET2): The enhanced VMXNET adapter is based on the
VMXNET adapter but it provides high-performance features, such as jumbo frames and
hardware offloads.

• VMXNET3: The VMXNET3 adapter is a paravirtualized NIC designed for performance. It is

not related to VMXNET or VMXNET2. The VMXNET3 adapter offers all the features
available in VMXNET2 and adds several features, like multiqueue support (called receive-
side scaling in Windows), IPv6 offloads, and MSI/MSI-X interrupt delivery.

• PVRDMA: The PVRDMA adapter is a paravirtualized NIC that supports remote direct
memory access between VMs through the OFED verbs API.

• SR-IOV passthrough: SR-IOV passthrough is a representation of a virtual function on a

physical NIC with an SR-IOV support.

For more information about network adapter types, see Network Adapter Basics at
https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vm_admin.doc/GUID-
AF9E24A8-2CFA-447B-AC83-35D563119667.html.

Module 8: Network Optimization 615

8-9 About VMXNET3 Optimization

To assist the VM in processing packets internally as quickly as possible, the VMXNET3 driver is
optimized to remove unnecessary metadata from smaller packets. It is optimized using the Copy
Tx function for small message size (≤ 256 bytes), improving processing time by up to 10 percent.
The use of pinned memory can greatly increase the VM’s ability to move packets through the OS
more efficiently. Normally, VM memory addresses must be translated to physical memory
addresses, which adds an overhead to the memory access time. Also, if memory is not reserved,
some memory pages might be put in swap memory or moved to accommodate the demands of
other VMs. By reserving and pinning the entire VM memory, you address translation as the VM’s
pages are permanently mapped to specific physical addresses. Therefore, the address translation
never needs repeating.
Memory can be pinned by configuring VMX options or, more simply, by setting the latency
sensitivity of the VM to high.
As shown, you add lines in the VM’s VMX file to pin all VM memory. These options ensure that
the VM can never use swap memory, which requires that the VM memory always be backed by

616 Module 8: Network Optimization

real physical memory. All the memory is configured to be preallocated and pinned, which results
in the appearance that all VM memory is permanently active.

Module 8: Network Optimization 617

8-10 Configuring a Single CPU Thread per vNIC

CPU scheduling is optimized for network packet processing in the ESXi.

Typically, when VM packets are being processed, a single CPU thread is used to process all
transmit traffic coming from a VM, regardless of how many vNICs the VM has. If multiple vNICs
are configured for a VM, it is advantageous to allow one CPU thread per vNIC to help speed up
the processing time. This arrangement helps boost transmit rates for the VM when more than one
vNIC is configured.
You can edit the VMX file to allow one CPU thread per device. Assume, for example, that the
VMX file is named VM1.vmx and that the VM has two NICs: Ethernet 0 and Ethernet 1. You
must add the line ethernetX.ctxPerDev = “1” for each device:
ethernet0.ctxPerDev = “1”
ethernet1.ctxPerDev = “1”

618 Module 8: Network Optimization

8-11 Configuring Multiple CPU Threads per vNIC

For VMs that require a high transmit rate and multiple simultaneous TCP streams, multiple CPU
threads can be used per vNIC. Up to eight threads can be used by the vNIC, depending on the
number of streams.
To enable this feature per VM, add the line EthernetX.ctxPerDev=“3” to the VMX file.
The value 1 allows a single thread per vNIC. The value 3 allows multiple threads per NIC, up to a
maximum of eight threads, created by the host. Using the value 3 does not signify that three
threads will be created. It only allows the host to create as many threads as it needs, up to a
maximum of eight. To enable this feature per VM, you add ethernetX.ctxPerDev=“3” to
the VMX file.
To set this feature at the host level, change the Net.NetVMTxType value to 3 in the ESXi host’s
advanced settings in the vSphere Client.

Module 8: Network Optimization 619

Acceptable values for Net.NetVMTxType are 1, 2, or 3:
1 1 transmit context (CPU thread) per vNIC
2 (Default): 1 transmit context (CPU thread) per VM
3 2 through 8 transmit contexts (CPU threads) per vNIC

Many of the CPU scheduling improvements require available CPU threads on a host. If your CPU
is already overprovisioned, implementing some of these features might make matters worse.
Adding CPU threads to process traffic flows requires that your CPU be underprovisioned to
ensure that network processing does not encounter contention on the CPUs that it is trying to use.

620 Module 8: Network Optimization

8-12 About TCP Segmentation Offload

TCP segmentation offload (TSO) improves performance for TCP network traffic coming from a
VM and for network traffic, such as vSphere vMotion traffic, sent out from the server.
TSO is used in the guest when the VMXNET2 (or later) network adapter is installed. To enable
TSO at the VM level, you must replace the existing VMXNET or flexible virtual network adapter
with a VMXNET2 (or later) adapter. This replacement might result in a change in the MAC
address of the virtual network adapter.
If TSO becomes disabled for a particular VMkernel interface, the only way to enable TSO is to
delete that VMkernel interface and recreate it with TSO enabled.
When the physical NICs provide TSO functionality, the ESXi host can use the specialized NIC
hardware to improve performance. However, performance improvements related to TSO do not
require NIC hardware support for TSO.
VMs that use TSO on an ESXi host show lower CPU utilization than VMs that lack TSO support,
when performing the same network activities.

Module 8: Network Optimization 621

For information about enabling TSO support for a VM or checking whether TSO is enabled on a
VMkernel interface, see TCP Segmentation Offload at https://docs.vmware.com/en/VMware-
vSphere/7.0/com.vmware.vsphere.networking.doc/GUID-E105A601-9331-496C-A213-
F76EA3863E31.html.
For more information about TSO, see VMware knowledge base article 2055140 at
http://kb.vmware.com/kb/2055140 and VMware knowledge base article 1009548 at
http://kb.vmware.com/kb/1009548.

622 Module 8: Network Optimization

8-13 About Jumbo Frames

For each packet, the system must perform a nontrivial amount of work to package and transmit the
packet. As Ethernet speed increases, so does the amount of work necessary which results in a
greater burden on the system.
Jumbo frames decrease the number of packets requiring packaging, compared to previously sized
packets. That decrease results in less work for network transactions, which frees up resources for
other activities.
The physical NICs at both ends, as well as all the intermediate hops, routers, and switches, must
support jumbo frames. Jumbo frames must be enabled at the virtual switch level, at the VM, and at
the VMkernel interface. ESXi hosts using jumbo frames realize a decrease in load due to network
processing. Before enabling jumbo frames, verify with your hardware vendor to ensure that your
network adapter supports jumbo frames.
For information about jumbo frames, see Jumbo Frames at https://docs.vmware.com/en/VMware-
vSphere/7.0/com.vmware.vsphere.networking.doc/GUID-53F968D9-2F91-41DA-B7B2-
48394D997F2A.html.

Module 8: Network Optimization 623

8-14 About SplitRx Mode

Multicasting is an efficient way of disseminating information and communicating over the

network.
A single sender can connect to multiple receivers and exchange information while conserving
network bandwidth. Financial stock exchanges, multimedia content delivery networks, and
commercial enterprises often use multicasting as a communication mechanism. Multiple receivers
can be enabled on a single ESXi host. Because the receivers are on the same host, the physical
network does not have to transfer multiple copies of the same packet. Packet replication is carried
out in the hypervisor instead.
SplitRx mode is an ESXi feature that provides a scalable and efficient platform for multicast
receivers.
SplitRx mode typically improves throughput and CPU efficiency for multicast traffic workloads.
SplitRx mode is enabled only if the network traffic is arriving on a physical NIC, not when the
traffic is entirely internal to the ESXi host. If the traffic is entirely internal, SplitRx mode can be
manually enabled.

624 Module 8: Network Optimization

SplitRx mode is individually configured for each vNIC. For more information about how to enable
this feature, see the ESXi Networking Considerations section at
https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/techpaper/performance/vs
phere-esxi-vcenter-server-67-performance-best-practices.pdf.

Module 8: Network Optimization 625

8-15 About SplitTx Mode

626 Module 8: Network Optimization

8-16 Activity: vSphere Networking Features (1)

Module 8: Network Optimization 627

8-17 Activity: vSphere Networking Features (2)

628 Module 8: Network Optimization

8-18 Review of Learner Objectives

Module 8: Network Optimization 629

8-19 Lesson 2: Monitoring Network I/O Activity

630 Module 8: Network Optimization

8-20 Learner Objectives

Module 8: Network Optimization 631

8-21 Monitoring for Performance in the Network I/O Stack

632 Module 8: Network Optimization

8-22 Network Capacity Metrics

To determine whether packets are being dropped, you can use the advanced performance charts in
the vSphere Client or use the esxtop command.

If received packets are being dropped, adjust the VM CPU shares. If packets are not being
dropped, check the size of the network packets, the data received rate, and the data transmitted
rate. In general, the larger the network packets, the faster the network speed. When the packet size
is large, fewer packets are transferred, which reduces the amount of CPU required to process the
data.
When network packets are small, more packets are transferred, but the network speed is slower
because more CPU is required to process the data. In some instances, large packets can result in
high latency. To rule out this problem, check network latency.
If packets are not being dropped and the data receive rate is slow, the host probably lacks the CPU
resources required to handle the load. Check the number of VMs assigned to each physical NIC. If
necessary, perform load balancing by moving VMs to different virtual switches or by adding more
NICs to the host. You can also move VMs to another host or increase the CPU resources of the
host or VMs.

Module 8: Network Optimization 633

8-23 About esxtop Network Statistics

634 Module 8: Network Optimization

8-24 About esxtop Network Output

In the esxtop window, configuration information about the objects is listed in the leftmost
columns, followed by the performance metrics.
In this example, the TestVM01 VM is connected to a distributed switch and vmnic1 is connected
to the same distributed switch.
The USED-BY column identifies network connections by physical adapter or vSphere network
object. vmnic0 is an example of a physical adapter listing. A VMkernel port, such as vmk0, is an
example of a vSphere network object.
Another example of a vSphere network object is a VM's NIC, identified as
357663:TestVM01.eth0. The value 357663 is an internal VM ID, TestVM01 is the VM name, and
eth0 identifies the network interface.

• PKTTX/s – Average number of packets transmitted per second in the sampling interval

• MbTX/s – Amount of data transmitted in Mbps

• PKTRX/s – Average number of packets received per second in the sampling interval

Module 8: Network Optimization 635

• MbRX/s – Amount of data received in Mbps

• %DRPTX – Percentage of transmit packets dropped.

• %DRPRX – Percentage of receive packets dropped

You might see Shadow of vmnic in some lines under the USED-BY column in the Network
view of ESXTOP. Shadow ports are used to transmit and receive health check data for each
uplink. You use exstop to monitor these shadow ports to debug and troubleshoot the network
health check feature.
For more information about shadow ports, see What Are the Shadow of vmnic in ESXTOP? at
https://www.virtuallyghetto.com/2013/01/what-are-shadow-of-vmnic-in-esxtop.html

636 Module 8: Network Optimization

8-25 Dropped Network Packets

Network packets might get stored (buffered) in queues at multiple points along their route from
the source to the destination. Network switches, physical NICs, device drivers, and network stacks
contain queues where packet data or headers might get buffered.
TCP/IP networks use congestion-control algorithms that limit, but do not eliminate, dropped
packets.
When a packet is dropped, the TCP/IP recovery mechanisms work to maintain in-order delivery of
packets to applications. However, these mechanisms operate at a cost to both networking
performance and CPU overhead, a penalty that becomes more severe as the physical network
speed increases.
vSphere presents vNICs, such as VMXNET or virtual E1000 devices, to the guest OS running in a
VM. For received packets, the vNIC buffers packet data coming from a virtual switch until it is
retrieved by the device driver running in the guest OS. The virtual switch contains queues for
packets sent to the vNIC.

Module 8: Network Optimization 637

If the guest OS does not retrieve packets from the vNIC quickly enough, the queues in the vNIC
device can fill up. This condition can, in turn, cause the queues in the corresponding virtual switch
port to fill up. If a virtual switch port receives a packet bound for a VM when its packet queue is
full, the port must drop the packet.

638 Module 8: Network Optimization

8-26 Dropped Packets in vNICs

When the applications and the guest OS are driving the VM to high CPU use, extended delays
might occur. These delays occur from the time the guest OS receives notification that packets are
available until those packets are retrieved from the vNIC.
Sometimes, the high CPU use might be caused by high network traffic because the processing of
network packets can place a significant demand on CPU resources.
Device drivers for networking devices often have parameters that are tunable from within the
guest OS. These parameters control such behavior as whether to use interrupts or perform polling.
Improper configuration of these parameters can cause poor network performance and dropped
packets in the networking infrastructure.
When the VM is dropping receive packets due to high CPU use, adding vCPUs might be
necessary to provide sufficient CPU resources. If the high CPU use is due to network processing,
ensure that the guest OS can use multiple CPUs when processing network traffic. See the OS
documentation for the appropriate CPU requirements.

Module 8: Network Optimization 639

Applications that have high CPU use can often be tuned to improve their use of CPU resources.
You might be able to tune the networking stack within the guest OS to improve the speed and
efficiency with which it handles network packets. For more information, eee the documentation
for the OS.
In some guest operating systems, all of the interrupts for each NIC are directed to a single
processor core. As a result, the single processor can become a bottleneck, leading to dropped
receive packets.
Adding more vNICs to these VMs enables the processing of network interrupts to be spread across
multiple processor cores.

640 Module 8: Network Optimization

8-27 Dropped Received Packets in Physical NICs

Module 8: Network Optimization 641

8-28 Dropped Transmit Packets in Physical NICs

When a VM transmits packets on a vNIC, those packets are buffered in the associated virtual
switch port until they are transmitted on the physical uplink devices.
Adding more physical uplink NICs to the virtual switch might alleviate the conditions that are
causing transmit packets to be dropped. However, traffic should be monitored to ensure that the
NIC teaming policies selected for the virtual switch lead to proper load distribution over the
available uplinks.
If the virtual switch uplinks are overloaded, moving some of the VMs to different virtual switches
can help rebalance the load.
Sometimes the bottleneck might be in the networking infrastructure, for example, in the network
switches or interswitch links. You might have to add capacity in the network to handle the load.
Reducing the network traffic generated by a VM can help alleviate bottlenecks in the networking
infrastructure. The implementation of this solution depends on the application and guest OS being
used. Techniques such as using caches for network data or tuning the network stack to use larger
packets (for example, jumbo frames) might reduce the load on the network.

642 Module 8: Network Optimization

8-29 Unexpected Increases in Data Transfer Rate

Examples of traffic flows are flows from applications running in a VM, vSphere vMotion, and
vSphere Fault Tolerance. These traffic flows can coexist and share a single link.
The total demand from all network users can exceed the total capacity of the network link. In this
case, users of the network resources might experience an unexpected impact on performance.
These instances, though infrequent, can still cause fluctuating performance behavior, which might
be frustrating.
If triggered, vSphere vMotion or vSphere Storage vMotion migrations can consume extra network
bandwidth when storage is IP/Ethernet-based. In this case, the performance of VMs that share
network resources with these traffic flows can be affected. The performance impact might be more
significant if the applications running in these VMs were latency-sensitive or business-critical.
To overcome the problems listed on the slide, use a resource-control mechanism such as Network
I/O Control. Network I/O Control allows applications to freely use shared network resources when
resources are underused. When the network is congested, Network I/O Control restricts the traffic
flows of applications according to their shares, reservations, and limits.

Module 8: Network Optimization 643

8-30 Networking Best Practices

When you install VMware Tools, a VMXNET adapter replaces the default Vlance adapter.
VMXNET adapters should be used for optimal performance in any guest OS for which they are
available.
For the best networking performance, use network adapters that support high-performance
hardware features, such as TCP checksum offload, jumbo frames, and the ability to handle high-
memory DMA.
NIC teams can provide passive failover if hardware failure or a network outage occurs. In some
configurations, NIC teaming can increase performance by distributing the traffic across those
physical network adapters.
To avoid unnecessary CPU and network overhead when two VMs reside on the same ESXi host,
connect both VMs to the same virtual switch. VMs that communicate with each other on the same
ESXi host can also use the VMware Machine Communication Interface device.

644 Module 8: Network Optimization

Network I/O Control can allocate user-defined proportions of bandwidth for specific needs and
can prevent any one resource pool from affecting others.
In a native environment, CPU use plays a significant role in network throughput. To process
higher levels of throughput, more CPU resources are needed. Because insufficient CPU resources
limit maximum throughput, monitoring the CPU use of high-throughput workloads is essential.

Module 8: Network Optimization 645

8-31 Introduction to Lab: Monitoring Network Performance
(1)

In this lab, you generate network traffic and compare the network performance of different
network adapters connected to different networks or the same network. Both test VMs are
configured with E1000 network adapters.
You use a client-side script named nptest1.sh to generate a lot of network traffic. You are
instructed to run this script on the network client VM, Linux01.
You also use a server-side script called netserver. This script runs on the network server VM,
Linux02, and receives the data sent by nptest1.sh.

646 Module 8: Network Optimization

You perform three test cases, two of which are described here:

• Test case 1: The first test VM is connected to the pg-SA-Production port group on the
distributed switch named dvs-Lab. The second test VM is connected to the pg-SA-
Management port group on the distributed switch named dvs-SA-Datacenter. Network traffic
flows between two networks.

• Test case 2: This test case is similar to test case 1, except that the network speed is
constrained.

Module 8: Network Optimization 647

8-32 Introduction to Lab: Monitoring Network Performance
(2)

For test case 3, both test VMs are connected to the pg-SA-Production port group on the dvs-Lab
distributed switch.

648 Module 8: Network Optimization

8-33 Lab 19: Monitoring Network Performance

Module 8: Network Optimization 649

8-34 Review of Lab: Monitoring Network Performance

650 Module 8: Network Optimization

8-35 Review of Learner Objectives

Module 8: Network Optimization 651

8-36 Key Points

652 Module 8: Network Optimization

Module 9
vCenter Server Performance Optimization

Module 9: vCenter Server Performance Optimization 653

9-2 Importance

654 Module 9: vCenter Server Performance Optimization

9-3 Lesson 1: vCenter Server Performance Optimization

Module 9: vCenter Server Performance Optimization 655

9-4 Learner Objectives

656 Module 9: vCenter Server Performance Optimization

9-5 vCenter Server Performance Considerations

Module 9: vCenter Server Performance Optimization 657

9-6 vCenter Server CPU Performance

Consistent high CPU usage can cause poor performance. Verify which processes, such as vpxd
and vsphere-client, use the most CPU. CPU usage that is consistently higher than 70
percent can cause a vCenter Server process to perform more queries than necessary.
The following actions might require increasing the amount of resources used by the vCenter
Server instance beyond the guidance provided in the VMware documentation:

• Adding extensions to vCenter Server

• Adding code to the vSphere Client

• Adding another extension point from where to get data

• Running more services might require more CPUs and memory.

For example, with NSX, running the vSphere ESX Agent Manager (EAM) service is known
to increase demand on resources.

658 Module 9: vCenter Server Performance Optimization

9-7 vCenter Server Memory Performance

vCenter Server and vCenter Server Appliance are critical components of your software-defined
data center (SDDC). For maximum performance and uptime, ensure that vCenter Server
Appliance is provisioned with enough resources to keep its systems running optimally.

Module 9: vCenter Server Performance Optimization 659

9-8 Monitoring CPU and Memory Use with the VAMI

To access the virtual appliance management interface (VAMI), open a web browser and navigate
to https://vCenter_Server_Name:5480. Click Monitor in the left pane. The CPU &
Memory tab displays by default.

660 Module 9: vCenter Server Performance Optimization

9-9 Using vimtop to Monitor vCenter Server Appliance
Resources

To access vimtop , enter vimtop at the Bash shell command prompt.

To see the vimtop help menu, enter h.

To quit vimtop, enter q.

Module 9: vCenter Server Performance Optimization 661

9-10 vimtop Commands and Columns

To add a metric (or column) to the vimtop display, perform the following steps:

1. Enter c to display the column selection pane.

2. Use the up and down arrow keys to highlight a metric in the column selection panel.
3. Press the space bar to select the metric.
A tilde (~) appears to the left of the metric name.

4. Press the Esc key to exit the column selection panel.

662 Module 9: vCenter Server Performance Optimization

9-11 Monitoring CPU Use with vimtop

You can monitor aggregate CPU use from the overview pane or examine CPU use of individual
process in the task pane.
Some columns that display include:

• Name: User-friendly name of the process (service).

• %CPU: Current CPU use (in percentage) for this process. If the value is 100 percent, the
process is using an entire core.

• MHZ: Current CPU use (in megahertz) for this process.

Module 9: vCenter Server Performance Optimization 663

9-12 Activity: CPU and Memory Use (1)

664 Module 9: vCenter Server Performance Optimization

9-13 Activity: CPU and Memory Use (2)

Module 9: vCenter Server Performance Optimization 665

9-14 Monitoring Memory Use with vimtop

Some columns that display include:

• %MEM: Physical memory use (in percentage) for this process.

• MEM: Physical or resident memory (in MB) used by this process.

• VIRT: Total virtual memory size (in MB). This value represents the complete working set,
which includes resident and swapped memory.

666 Module 9: vCenter Server Performance Optimization

9-15 Viewing Service Memory Allocation with cloudvm-ram-
size

Module 9: vCenter Server Performance Optimization 667

9-16 Viewing Service Heap Size

668 Module 9: vCenter Server Performance Optimization

9-17 Changing Service Heap Size

If you do not want to take the downtime involved with rebooting vCenter Server after adding
memory, increase the heap size of each individual service and restart that service. The heap is
automatically resized.

Module 9: vCenter Server Performance Optimization 669

9-18 vCenter Server Network Performance

670 Module 9: vCenter Server Performance Optimization

9-19 Monitoring Network Activity with the VAMI

Module 9: vCenter Server Performance Optimization 671

9-20 Monitoring Network Activity with vimtop

You can monitor the following columns for network activity:

• DROPPED: Number of dropped packets through this network interface because the system
ran out of buffers during the last measurement cycle.

• ERRS: Total number of transmit faults and receive faults on this interface.

672 Module 9: vCenter Server Performance Optimization

9-21 vCenter Server Database Performance (1)

CPU usage at 100 percent for long periods is not normal. When you see consistently high CPU
usage, verify that nothing basic is malfunctioning.

If you have a large inventory, you might have many statistics. If you do not have a good enough
disk subsystem, sufficient memory, or sufficient CPU, then rollups or Top_N calculations might
be slow.
A large inventory does not only mean a high number of hosts and VMs, it might mean many
datastores, networks, or devices per VM. Some queries require many joins or full scans on tables,
which is not ideal but does occur. In such cases, the database statistics might need to be
recomputed or reindexed to make the query engine operate more efficiently.

Module 9: vCenter Server Performance Optimization 673

9-22 vCenter Server Database Performance (2)

You can verify the vpxd log for diagnosis. You might see the [VdbStatement] Execution
elapsed time: … and SQL execution took too long statements in the log. You
should expect to see a few of these events (often for Top_N, statistics, and events), each of which
typically lasts between three and four seconds. If these events are extremely frequent and each
lasts 10 or more seconds, this behavior often indicates problems in your database. Verify the
resources (I/O, CPU, and memory) and that the database is correctly provisioned.

674 Module 9: vCenter Server Performance Optimization

9-23 Effects of Changing Statistics Level on Database
Traffic

The tables show what happens when different statistics levels are used.
The configuration of two VMs and a host is shown in the first table. The configuration of your
VMs and hosts is critical to how many statistics are generated. In the example, the VM
configurations are the same except for the number of virtual disks.
The second table shows what happens when you increase the statistics level. Of the four statistics
levels, 1 is the least detailed and 4 is the most detailed. The specific numbers are not as important
as the significant increase in statistics from level 1 to level 2. Level 1 reports on aggregated
statistics, such as aggregated CPU, aggregated disk latencies, and so on. Level 2 reports more
detailed statistics, such as per-NIC and per-datastore statistics.

In the second table, you can see that VM1 has far more statistics than VM2, which is because
VM1 has many more disks than VM2. Several datastore and network counters (such as read/write

Module 9: vCenter Server Performance Optimization 675

requests per second and packets sent/received) are introduced at level 2. As a result, the larger the
number of disks and networks in your configuration, the more statistics that are generated.

In your environment, ensure that you size the infrastructure and database optimally to
accommodate the load of a level.

676 Module 9: vCenter Server Performance Optimization

9-24 vCenter Server Appliance Disk Use (1)

The substantial source of disk I/O tends to be the database. The main disk bandwidth consumers
are the following partitions:

• /storage/db is the location of the vCenter Server database.

• /storage/dblog

• /storage/seat

When using vCenter Server Appliance, ensure that these partitions are on high-speed storage and
have sufficient space.
If you must increase the disk space for vCenter Server Appliance, see VMware knowledge base
article 2145603 at http://kb.vmware.com/kb/2145603.

Module 9: vCenter Server Performance Optimization 677

9-25 vCenter Server Appliance Disk Use (2)

678 Module 9: vCenter Server Performance Optimization

9-26 vCenter Server Appliance Disk Use (3)

Module 9: vCenter Server Performance Optimization 679

9-27 Monitoring Disk Use with the VAMI

680 Module 9: vCenter Server Performance Optimization

9-28 Monitoring Disk Usage with the df Command

The df command displays file system use information, such as total size, used space, and
available space. The -h option shows sizes in powers of 1024, for example, in megabytes,
gigabytes, and so on.

Module 9: vCenter Server Performance Optimization 681

9-29 Monitoring Disk Activity with vimtop

The following columns show:

• READS: Number of reads issued to this partition and completed successfully during the last
measurement interval.

• WRITES: Number of writes issued to this partition and completed successfully during the last
measurement interval.

682 Module 9: vCenter Server Performance Optimization

9-30 Monitoring Database Activity with the VAMI

Module 9: vCenter Server Performance Optimization 683

9-31 Activity: Network Performance (1)

684 Module 9: vCenter Server Performance Optimization

9-32 Activity: Network Performance (2)

Module 9: vCenter Server Performance Optimization 685

9-33 Activity: vCenter Server Tools (1)

686 Module 9: vCenter Server Performance Optimization

9-34 Activity: vCenter Server Tools (2)

Module 9: vCenter Server Performance Optimization 687

9-35 Review of Learner Objectives

688 Module 9: vCenter Server Performance Optimization

9-36 Key Points

Module 9: vCenter Server Performance Optimization 689

690 Module 9: vCenter Server Performance Optimization
Module 10
Appendix: vSphere Auto Deploy

Module 10: Appendix: vSphere Auto Deploy 691

10-2 Importance

692 Module 10: Appendix: vSphere Auto Deploy

10-3 Appendix Lessons

Module 10: Appendix: vSphere Auto Deploy 693

10-4 Lesson 1: vSphere ESXi Image Builder

694 Module 10: Appendix: vSphere Auto Deploy

10-5 Learner Objectives

Module 10: Appendix: vSphere Auto Deploy 695

10-6 ESXi Images

An ESXi image contains all the software necessary to run on an ESXi host and includes the
following components:

• The base ESXi software, also called the core hypervisor

• Specific hardware drivers

• Common Information Model (CIM) providers

• Specific applications or plug-in components

An ESXi image can be installed on a local or a SAN-based boot device, or it can be delivered at
boot time using vSphere Auto Deploy.

696 Module 10: Appendix: vSphere Auto Deploy

10-7 vSphere Installation Bundles

An ESXi image includes one or more vSphere installation bundles (VIBs). A VIB is an ESXi
software package. VMware and its partners package solutions, drivers, CIM providers, and
applications that extend the ESXi platform in VIBs.
An ESXi image should always contain one base VIB. Other VIBs can be added to include
additional drivers, CIM providers, updates, patches, and applications.

Module 10: Appendix: vSphere Auto Deploy 697

10-8 ESXi Image Deployment

VMware provides standard ESXi images, which are available on the VMware website. ESXi
images can also be provided by VMware partners.
The challenge that administrators face when using the standard ESXi image that VMware provides
is that the standard image is sometimes limited in functionality. For example, the standard ESXi
image might not contain all the drivers or CIM providers for a specific set of hardware, or the
standard image might not contain vendor-specific plug-in components.
To create an ESXi image that contains custom components, use the vSphere Client.

698 Module 10: Appendix: vSphere Auto Deploy

10-9 vSphere ESXi Image Builder Overview

vSphere ESXi Image Builder is a utility for customizing ESXi images. You access vSphere ESXi
Image Builder through the vSphere Client. You use vSphere ESXi Image Builder to create and
manage VIBs, image profiles, software depots, and software channels.
For more information about vSphere ESXi Image Builder and its prerequisite software, see
vSphere ESXi Image Builder Installation and Usage at
https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.esxi.install.doc/GUID-62B15826-
B529-4519-B57A-98DFD0CC5522.html.
For more information about PowerCLI, see the documentation page at
https://www.vmware.com/support/developer/PowerCLI.

Module 10: Appendix: vSphere Auto Deploy 699

10-10 vSphere ESXi Image Builder Architecture

VIBs are software packages that consist of packaged solutions, drivers, CIM providers, and
applications that extend the ESXi platform. VIBs are available in software depots.
An image profile always includes a base VIB and typically includes other VIBs. You use the
vSphere Client to examine and define an image profile.
The software depot is a hierarchy of files and folders that can be available through an HTTP URL
(online depot) or a ZIP file (offline depot or bundle). VMware has depots that you can use. The
offline bundle is available on the Downloads page on the VMware website. Go to the Downloads
page for VMware vSphere Hypervisor (ESXi) 6.7.
VMware partners also make software depots available. Companies with large VMware
installations might also create internal depots to provision ESXi hosts with vSphere Auto Deploy
or to export an ISO image for ESXi installation.

700 Module 10: Appendix: vSphere Auto Deploy

10-11 Building ESXi Images

Module 10: Appendix: vSphere Auto Deploy 701

10-12 Step 1: Starting the Image Builder Service

702 Module 10: Appendix: vSphere Auto Deploy

10-13 Step 2: Connecting to a Software Depot

Before you can create or customize an ESXi image, vSphere ESXi Image Builder must be able to
access one or more software depots.
Use the vSphere Client to add an ESXi software depot or offline depot ZIP file to vCenter Server.
You can then create image profiles and generate ISO files from the image profiles and VIBs in the
depots.
After adding the software depot to vSphere ESXi Image Builder, verify that you can view the
software packages that are found in the depot.

Module 10: Appendix: vSphere Auto Deploy 703

10-14 Step 3: Creating an Image Profile

Cloning a published profile is the easiest way to create a custom image profile. Cloning a profile is
useful when you want to remove a few VIBs from a profile. Cloning is also useful when you want
to use hosts from different vendors and use the same basic profile, with the addition of vendor-
specific VIBs. VMware partners or customers with large installations might consider creating a
profile.

704 Module 10: Appendix: vSphere Auto Deploy

10-15 Step 4: Generating the New ESXi Image

After creating an image profile, you can generate an ESXi image. You can export an image profile
as an ISO image or ZIP file. An ISO image can be used to boot a host to install ESXi. A ZIP file
can be used by vSphere Update Manager for remediating ESXi hosts. The exported image profile
can also be used with vSphere Auto Deploy to boot ESXi hosts.

Module 10: Appendix: vSphere Auto Deploy 705

10-16 Review of Learner Objectives

706 Module 10: Appendix: vSphere Auto Deploy

10-17 Lesson 2: Overview of vSphere Auto Deploy

Module 10: Appendix: vSphere Auto Deploy 707

10-18 Learner Objectives

708 Module 10: Appendix: vSphere Auto Deploy

10-19 About vSphere Auto Deploy

With vSphere Auto Deploy, the ESXi image is streamed across the network to the host and loaded
directly into memory. All changes made to the state of the host are stored in memory and
synchronized to vCenter Server. When the host is shut down, the memory-based state of the host
is lost but can be streamed into memory again from vCenter Server when the host is powered back
on.
vSphere Auto Deploy does not store the ESXi state on the host disk. vCenter Server stores and
manages ESXi updates and patching through an image profile and, optionally, the host
configuration through a host profile.
With vSphere Auto Deploy, many hosts can be rapidly deployed. vSphere Auto Deploy simplifies
ESXi host management by eliminating the necessity to maintain a separate boot image for each
host. A standard ESXi image can be shared across many hosts.
Because the host image is decoupled from the physical server, the host can be recovered without
having to recover the hardware or restore from a backup. vSphere Auto Deploy can eliminate the
need for a dedicated boot device, freeing the boot device for other uses.

Module 10: Appendix: vSphere Auto Deploy 709

For more information about vSphere Auto Deploy, see Using vSphere Auto Deploy to Reprovision
Hosts
at https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.esxi.upgrade.doc/GUID-
E2A5AB39-3EA7-4D26-8F17-3AB4BCC1EBC4.html.

710 Module 10: Appendix: vSphere Auto Deploy

10-20 vSphere Auto Deploy Modes

Module 10: Appendix: vSphere Auto Deploy 711

10-21 vSphere Auto Deploy Stateless Mode

With stateless mode, the ESXi host boots with networking connectivity to vCenter Server through
the built-in standard switch. If the host profile specifies distributed switch membership, vCenter
Server joins the ESXi host to the distributed switch.
Using the stateless caching or the stateful installation mode greatly lessens the dependency on
other systems for booting ESXi hosts. Hosts can be rebooted even when the vSphere Auto Deploy
service, vCenter Server, or the Dynamic Host Configuration Protocol (DHCP) and Trivial File
Transfer Protocol (TFTP) servers are unavailable.

712 Module 10: Appendix: vSphere Auto Deploy

10-22 vSphere Auto Deploy Mode: Stateless Caching

vSphere Auto Deploy stateless caching PXE boots the ESXi host and loads the image in memory
which is similar to the process for loading an ESXi host using stateless mode. However, when the
host profile is applied to the ESXi host, the image running in memory is copied to a boot device.
The saved image acts as a backup in case the PXE infrastructure, vCenter Server, or vSphere Auto
Deploy service are unavailable. If the host must reboot and it cannot contact the DHCP or TFTP
server, or vSphere Auto Deploy service, the network boot times out and the host reboots using the
cached disk image.
Stateless caching might help ensure availability of an ESXi host because the host can boot if an
outage occurs that affects the DHCP, TFTP, or vCenter Server, or the vSphere Auto Deploy
service. However, stateless caching does not guarantee that the image is current or that the vCenter
Server system is available after booting. The primary benefit of stateless caching is that you can
boot the host to help troubleshoot and resolve problems that prevent a successful PXE boot.
Unlike stateless ESXi hosts, stateless caching requires that a dedicated boot device is assigned to
the host.

Module 10: Appendix: vSphere Auto Deploy 713

10-23 vSphere Auto Deploy Mode: Stateful Installation

Setting up stateful installation is similar to configuring stateless caching. But instead of attempting
to use the vSphere Auto Deploy service on vCenter Server to boot the host, the host does a one-
time PXE boot to install ESXi. After the image is cached to disk, the host boots from the disk
image on all subsequent reboots.

714 Module 10: Appendix: vSphere Auto Deploy

10-24 Locations for Configuration and State Information

Without the use of vSphere Auto Deploy, the ESXi host’s image (binaries and VIBs),
configuration, state, and log files are stored on the boot device.

With stateless vSphere Auto Deploy, a boot device no longer holds all the host’s information. The
host information that is stored includes the following items:

• Image state: Executable software to run on the ESXi host. The information is part of the
image profile, which can be created and customized with the vSphere Client or vSphere ESXi
Image Builder.

• Configuration state: Configurable settings that determine how the host is configured.
Examples include virtual switch settings, boot parameters, and driver settings. Host profiles
are created using the host profile user interface in the vSphere Client.

Module 10: Appendix: vSphere Auto Deploy 715

• Running state: Settings that apply while the ESXi host is up and running. This state also
includes the location of the VM in the inventory and the VM autostart information. The
vCenter Server instance manages state information.

• Event recording: Information found in log files and core dumps. This information can be
managed by vCenter Server, using services such as vSphere ESXi Dump Collector and
vSphere Syslog Collector. vSphere Syslog Collector uses Linux rsyslogd.

716 Module 10: Appendix: vSphere Auto Deploy

10-25 vSphere Auto Deploy Architecture

The vSphere Auto Deploy infrastructure consists of the following components:

• vSphere Auto Deploy service: Serves images and host profiles to ESXi hosts. The service,
which runs on the vCenter Server, is at the heart of the vSphere Auto Deploy infrastructure.

• vSphere Auto Deploy rules engine: Tells the vSphere Auto Deploy service which image and
which host profiles to serve to a host. You use the vSphere Web Client to define the rules that
assign image profiles and host profiles to hosts.

• Image profiles: Define the set of VIBs with which to boot ESXi hosts. VMware and its
partners make image profiles and VIBs available in public software depots. Use the vSphere
Client to examine the depot and the vSphere Auto Deploy rules engine to specify which
image profile to assign to a host. You can create a custom image profile based on the public
image profiles and VIBs in the depot and apply that image profile to the host.

• Host profiles: Templates that define an ESXi host’s configuration, such as networking or
storage setup. You can save the host profile for an individual host and reuse this host profile

Module 10: Appendix: vSphere Auto Deploy 717

 to reprovision that host. You can save the host profile of a template host and use that profile
for other hosts.

• Host customizations: Stores host-specific information. A host customization must be created

for each host.

718 Module 10: Appendix: vSphere Auto Deploy

10-26 Rules Engine Basics

Module 10: Appendix: vSphere Auto Deploy 719

The rules engine includes the following information:

• Rules: Rules can assign image profiles and host profiles to a set of hosts or specify the
inventory location (folder or cluster) of a host on the target vCenter Server system. A rule can
identify target hosts by boot MAC address, SMBIOS asset tag, BIOS UIID, or fixed DHCP IP
address. In most cases, rules apply to multiple hosts. You use the vSphere Client to create
rules. After you create a rule, you must add it to a rule set. After you add a rule to a rule set,
you cannot edit it.

• Active rule set: When a newly started host contacts the vSphere Auto Deploy service with a
request for an image, the server checks the active rule set for matching rules. The image
profile, host profile, and vCenter Server inventory location that are mapped by matching rules
are used to boot the host. If more than one item is mapped by the rules, the vSphere Auto
Deploy service uses the first item in the rule set.

• Working rule set: You can use the working rule set to test changes to rules before making
them active. For example, you can use the vSphere Client to test compliance with the working
rule set.

720 Module 10: Appendix: vSphere Auto Deploy

10-27 PXE Boot Infrastructure Setup

A DHCP server and a TFTP server must be configured. The DHCP server assigns IP addresses to
each autodeployed host on startup and points the host to a TFTP server to download the gPXE
configuration files. The ESXi hosts can either use the infrastructure’s existing DHCP and TFTP
servers or new DHCP and TFTP servers can be created for use with vSphere Auto Deploy. Any
DHCP server that supports the next-server and filename options can be used.

Module 10: Appendix: vSphere Auto Deploy 721

10-28 Initial Boot of an Autodeployed ESXi Host (1)

When an autodeployed host boots for the first time, the following events occur:
1. When the ESXi host is powered on, the ESXi host starts a PXE boot sequence.
2. The DHCP server assigns an IP address to the ESXi host and instructs the host to contact the
TFTP server.
3. The ESXi host downloads the gPXE image file and the gPXE configuration file from the
TFTP server.

722 Module 10: Appendix: vSphere Auto Deploy

10-29 Initial Boot of an Autodeployed ESXi Host (2)

The gPXE configuration file instructs the host to make an HTTP boot request to the vSphere Auto
Deploy service.

Module 10: Appendix: vSphere Auto Deploy 723

10-30 Initial Boot of an Autodeployed ESXi Host (3)

The vSphere Auto Deploy service queries the rules engine for the following information about the
host:

• The image profile to use

• The host profile to use

• Which cluster the host belongs to, if any

724 Module 10: Appendix: vSphere Auto Deploy

10-31 Initial Boot of an Autodeployed ESXi Host (4)

The vSphere Auto Deploy service streams VIBs specified in the image profile to the host.
The host boots based on the image profile and the host profile received from vSphere Auto
Deploy.
vSphere Auto Deploy also assigns the host to the vCenter Server instance with which it is
registered.

Module 10: Appendix: vSphere Auto Deploy 725

10-32 Initial Boot of an Autodeployed ESXi Host (5)

If a rule specifies a target folder or a cluster on the vCenter Server instance, the host is added to
that location. If no rule exists, vSphere Auto Deploy adds the host to the first data center.
If a host profile is used without a host customization and the profile requires specific information
from the user, the host is placed in maintenance mode. The host remains in maintenance mode
until the user reapplies the host profile and answers any outstanding questions.
If the host is part of a fully automated vSphere DRS cluster, the cluster rebalances itself based on
the new host, moving VMs onto the new host.
To make subsequent boots quicker, the vSphere Auto Deploy service stores the ESXi image as
well as the following information:

• The image profile to use

• The host profile to use

• The location of the host in the vCenter Server inventory

726 Module 10: Appendix: vSphere Auto Deploy

10-33 Subsequent Boot of a Stateless ESXi Host (1)

When an autodeployed ESXi host is rebooted, a slightly different sequence of events occurs:
1. The ESXi host goes through the PXE boot sequence, as it does in the initial boot sequence.
2. The DHCP server assigns an IP address to the ESXi host and instructs the host to contact the
TFTP server.
3. The host downloads the gPXE image file and the gPXE configuration file from the TFTP
server.

Module 10: Appendix: vSphere Auto Deploy 727

10-34 Subsequent Boot of a Stateless ESXi Host (2)

As in the initial boot sequence, the gPXE configuration file instructs the host to make an HTTP
boot request to the vSphere Auto Deploy service.

728 Module 10: Appendix: vSphere Auto Deploy

10-35 Subsequent Boot of a Stateless ESXi Host (3)

The subsequent boot sequence differs from the initial boot sequence.
When an ESXi host is booted for the first time, vSphere Auto Deploy queries the rules engine for
information about the host. The information about the host’s image profile, host profile, and
cluster is stored on the vSphere Auto Deploy service.
On subsequent reboots, the vSphere Auto Deploy service uses the saved information instead of
using the rules engine to determine this information. Using the saved information saves time
during subsequent boots because the host does not have to be checked against the rules in the
active rule set. vSphere Auto Deploy checks the host against the active rule set only once and that
is during the initial boot.
The vSphere Auto Deploy service delivers the image profile and host profile to the host.

Module 10: Appendix: vSphere Auto Deploy 729

10-36 Subsequent Boot of a Stateless ESXi Host (4)

The ESXi host is placed in its assigned cluster on the vCenter Server instance.

730 Module 10: Appendix: vSphere Auto Deploy

10-37 Activity: vSphere Auto Deploy Modes (1)

Module 10: Appendix: vSphere Auto Deploy 731

10-38 Activity: vSphere Auto Deploy Modes (2)

732 Module 10: Appendix: vSphere Auto Deploy

10-39 Running Custom Scripts on Stateless Hosts

The scripts run in alphabetical order after the initial ESXi boot workflow of the host.
For more information about how to add a script bundle, see Working with Script Bundles at
https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.esxi.install.doc/GUID-
485DCF6A-4674-47E4-AD44-
A1432774D792.html?hWord=N4IghgNiBcIM4GMBOBLADgFwAQCMCuAdgCYQCmIAvkA.

Module 10: Appendix: vSphere Auto Deploy 733

10-40 Booting Many Stateless Hosts

When a stateless host boots, the host requests an image from the vSphere Auto Deploy service.
Under normal circumstances, the vSphere Auto Deploy service can serve an image to the host.
However, if the network is saturated because hundreds of stateless hosts are booting at the same
time, the vSphere Auto Deploy service might take a long time to respond to a request.
Proxy servers (off-the-shelf proxy servers) help lessen the load on the vSphere Auto Deploy
service that is on vCenter Server. If proxy servers front the vSphere Auto Deploy service, a list of
proxy servers is sent to the stateless host when, in an extreme situation, vSphere Auto Deploy
cannot serve the image. The stateless host tries to boot from the first proxy server on the list. If
this proxy server is available, the image is served to the host. But if the first proxy server is
overloaded or unavailable, the stateless host requests an image from the second proxy server in the
list.
For information about how to Register a Caching Proxy Server Address with vSphere Auto Deploy
see https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.esxi.install.doc/GUID-
836883D3-24C8-46A1-BD66-0F4756A64E04.html.

734 Module 10: Appendix: vSphere Auto Deploy

10-41 Stateless Caching Host Profile Configuration

Stateless caching is configured using host profiles. When configuring stateless caching, you can
choose to save the image to a local boot device or to a USB disk. You can also leave the VMFS
datastore on the boot device.

Module 10: Appendix: vSphere Auto Deploy 735

10-42 Subsequent Boot of a Stateless Caching Host

vSphere Auto Deploy caches the image when you apply the host profile if Enable stateless
caching on the host is selected in the System Cache Configuration host profile. When you reboot
the host, it continues to use the vSphere Auto Deploy infrastructure to retrieve its image. If the
vSphere Auto Deploy service is not available, the host uses the cached image.
If the ESXi hosts that run your VMs lose connectivity to the vSphere Auto Deploy service or to
the vCenter Server system, some limitations apply the next time that you reboot the host:

• If the vCenter Server system is available but the vSphere Auto Deploy service is unavailable,
hosts do not connect to the vCenter Server system automatically. You can manually connect
the hosts to the vCenter Server system or wait until the vSphere Auto Deploy service is
available again.

• If the vCenter Server system is unavailable (which implies that the vSphere Auto Deploy
service is unavailable), you can connect to each ESXi host by using VMware Host Client and
add VMs to each host.

736 Module 10: Appendix: vSphere Auto Deploy

• If the vCenter Server system is not available, vSphere DRS does not work. The vSphere Auto
Deploy service cannot add hosts to vCenter Server. You can connect to each ESXi host by
using VMware Host Client and add VMs to each host.

• If you change your setup while connectivity is lost, the changes are lost when the connection
to the vSphere Auto Deploy service is restored.

Module 10: Appendix: vSphere Auto Deploy 737

10-43 Stateful Installation Host Profile Configuration

You configure stateful installation by using host profiles. When configuring stateful installation,
you can choose to save the image to a local boot device or to a USB device. You can also leave the
VMFS datastore on the boot device.

738 Module 10: Appendix: vSphere Auto Deploy

10-44 Subsequent Boot of a Stateful Installation Host

With stateful installation, the vSphere Auto Deploy service is used to provision new ESXi hosts.
The first time that the host boots, it uses the PXE host, the DHCP server, and the vSphere Auto
Deploy service, like a stateless host does. All subsequent reboots use the image that is saved to the
local disk.
After the image is written to the host’s local storage, the image is used for all future reboots. The
initial boot of a system is the equivalent of an ESXi host installation to a local disk. Because no
attempt is made to update the image, the image can become stale over time. Manual processes
must be implemented to manage configuration updates and patches, losing most of the key
benefits of vSphere Auto Deploy.

Module 10: Appendix: vSphere Auto Deploy 739

10-45 Stateful Installation Using vSphere Lifecycle Manager

vSphere Lifecycle Manager introduces a new architecture to coordinate the validation and
remediation of vSphere clusters. It consults the Hardware Compatibility Lists to find the latest
releases and the hardware that is compatible with each release.
Depots that are compatible with vSphere Update Manager are used to publish ESXi releases.
These depots can be hosted online or be provided as local files. You can use VMware predefined
depots, or you can add custom depots to access base images, add-ons, and components.
The vSphere Lifecycle Manager components that are in ESXi hosts and vCenter Server Appliance
instances are hosted in the vSphere Update Manager vAPI endpoint. During compliance check and
remediation, ESXi Health Perspectives queries ESXi health so that vSphere Lifecycle Manager
can safely update the hosts.

740 Module 10: Appendix: vSphere Auto Deploy

10-46 Review of Learner Objectives

Module 10: Appendix: vSphere Auto Deploy 741

10-47 Lesson 3: Configuring vSphere Auto Deploy

742 Module 10: Appendix: vSphere Auto Deploy

10-48 Learner Objectives

Module 10: Appendix: vSphere Auto Deploy 743

10-49 Configuring a vSphere Auto Deploy Environment

You need a DHCP server and a TFTP server to PXE boot the ESXi installer. To PXE boot the
ESXi installer, the DHCP server must send the address of the TFTP server and the filename of the
initial boot loader to the ESXi host.

744 Module 10: Appendix: vSphere Auto Deploy

10-50 Step 1: Preparing the DHCP Server

You must set up the DHCP server to serve each target ESXi host with an iPXE binary. In this
example, the DHCP server is included with Windows Server 2012.
The iPXE binary file undionly.kpxe.vmw-hardwired is used to boot the ESXi hosts.

See VMware ESXi Installation and Setup at https://docs.vmware.com/en/VMware-

vSphere/6.7/vsphere-esxi-67-installation-setup-guide.pdf.

Module 10: Appendix: vSphere Auto Deploy 745

10-51 Step 2: Starting the vSphere Auto Deploy Service

Before you can use vSphere Auto Deploy, you must configure the vSphere Auto Deploy service
startup type in the vCenter Server system that you plan to use for managing the hosts that you
provision.

746 Module 10: Appendix: vSphere Auto Deploy

10-52 Step 3: Preparing the TFTP Server

After you prepare the DHCP server, you must start the Auto Deploy service on vCenter Server and
configure the TFTP server. You must download a TFTP boot ZIP file from your vSphere Auto
Deploy service. The TFTP server serves the boot images that vSphere Auto Deploy provides.
When you click Download TFTP Boot Zip, the deploy-tftp.zip file is downloaded to the
system on which the vSphere Client is running. You unzip this file and copy its contents to the
TFTP server’s root directory. The TFTP root directory is defined by the TFTP_Root parameter
on the TFTP server.
For more information about how to configure the TFTP environment for vSphere Auto Deploy
provisioning, see VMware ESXi Installation and Setup at
ttps://docs.vmware.com/en/VMwarevSphere/6.7/vsphere-esxi-67-installation-setup-guide.pdf.

Module 10: Appendix: vSphere Auto Deploy 747

10-53 Step 4: Creating Deployment Rules

Rules can assign image profiles and host profiles to a set of hosts, or specify the location (folder or
cluster) of a host on the target vCenter Server system. A rule can identify target hosts by boot
MAC address, SMBIOS information, BIOS UUID, vendor, model, or fixed DHCP IP address.
Usually, rules apply to multiple hosts.
After you create a rule, you must add it to a rule set. The only supported rule sets are active and
working.
A rule can belong to both sets (the default) or to only the working rule set. After you add a rule to
a rule set, you can no longer change the rule. Instead, you copy the rule and replace items or
patterns in the copy. If you are managing vSphere Auto Deploy with the vSphere Client, you can
edit a rule if it is in the inactive state.

748 Module 10: Appendix: vSphere Auto Deploy

10-54 Step 5: Activating Deployment Rules

After you create a vSphere Auto Deploy rule, the rule is in the inactive state. You must activate
the rule for it to take effect. You can use the Activate and Reorder wizard to activate, deactivate,
and change the order of the rules.

Module 10: Appendix: vSphere Auto Deploy 749

10-55 Managing the vSphere Auto Deploy Environment

You can change a rule set, for example, to require a host to boot from a different image profile.
You can also require a host to use a different host profile. Unprovisioned hosts that you boot are
automatically provisioned according to these modified rules. For all other hosts, vSphere Auto
Deploy applies the new rules only when you test their rule compliance and perform remediation.
If vCenter Server becomes unavailable, stateless hosts that are already autodeployed remain up
and running. However, you cannot boot or reboot hosts. Installing vCenter Server in a VM and
placing the VM in a vSphere HA cluster is recommended for keeping the VM available.
For information about how to Test and Repair Rule Compliance see
https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.esxi.upgrade.doc/GUID-
4915B724-500E-4FB3-BAC2-0EA46CFBD7EE.html.

750 Module 10: Appendix: vSphere Auto Deploy

10-56 Using vSphere Auto Deploy with vSphere Update
Manager

vSphere Update Manager can be used to upgrade hosts that boot using vSphere Auto Deploy.
Image profiles must be patched and modified using the vSphere Client or vSphere ESXi Image
Builder.
Only live-install patches can be remediated with vSphere Update Manager. Any patch that
requires a reboot cannot be installed on a PXE host. The live-install patches can be from VMware
or from a third party.

Module 10: Appendix: vSphere Auto Deploy 751

10-57 Review of Learner Objectives

752 Module 10: Appendix: vSphere Auto Deploy

10-58 Key Points

Module 10: Appendix: vSphere Auto Deploy 753

754 Module 10: Appendix: vSphere Auto Deploy
Module 11
Appendix: vSphere Security

Module 11: Appendix: vSphere Security 755

11-2 Importance

756 Module 11: Appendix: vSphere Security

11-3 Appendix Lessons

Module 11: Appendix: vSphere Security 757

11-4 Lesson 1: Configuring ESXi Host Access and
Authentication

758 Module 11: Appendix: vSphere Security

11-5 Learner Objectives

Module 11: Appendix: vSphere Security 759

11-6 About ESXi Host Security Properties

An ESXi host is protected with a firewall. You can open ports for incoming and outgoing traffic,
as needed, but should restrict access to services and ports. Using the ESXi lockdown mode and
limiting access to vSphere ESXi Shell can further contribute to a more secure environment.

760 Module 11: Appendix: vSphere Security

11-7 Configuring the ESXi Firewall

Firewalls control access to devices in their perimeter by closing all communication pathways,
except for the pathways that the administrator explicitly or implicitly designates as authorized. The
pathways, or ports, that administrators open in the firewall allow traffic between devices on
different sides of the firewall.
With the ESXi firewall engine, rule sets define port rules for each service. For remote hosts, you
can define the IP addresses or range of IP addresses that are allowed to access each service.
You can configure the ESXi firewall by using the vSphere Client or vSphere CLI. You configure
firewall management at the command line by using the esxcli network firewall
namespace.

Module 11: Appendix: vSphere Security 761

11-8 Configuring ESXi Services

762 Module 11: Appendix: vSphere Security

11-9 About Lockdown Mode (1)

Module 11: Appendix: vSphere Security 763

11-10 About Lockdown Mode (2)

You can enable lockdown mode:

• When adding the host to the vCenter Server

• By using the Direct Console User Interface (DCUI)

• By using a host's security profile in the vSphere Client

In lockdown mode, all host operations must be performed through vCenter Server. The host
cannot be accessed directly through the VMware Host Client.

764 Module 11: Appendix: vSphere Security

11-11 Normal Lockdown Mode Behavior

In normal lockdown mode, users listed in the DCUI. Access advanced system setting can log in to
the Direct Console User Interface (DCUI). By being on this list, users have emergency access to
the DCUI if the connection to vCenter Server is lost. These users do not require administrative
privileges on the host. Nonadministrative DCUI.Access users can only disable or enable
lockdown.

Module 11: Appendix: vSphere Security 765

11-12 Strict Lockdown Mode Behavior

If vCenter Server is unavailable, the vSphere Client is also unavailable. Therefore, hosts that are in
strict lockdown mode are not manageable.

766 Module 11: Appendix: vSphere Security

11-13 Integrating ESXi with Active Directory

Although day-to-day vSphere management operations are done while logged in to vCenter Server
using the vSphere Client, the user sometimes requires direct access to the ESXi host, for example,
when accessing local log files.
vCenter Single Sign-On is the recommended way to manage user access to hosts. The Active
Directory (AD) domain to which users belong is added to vCenter Single Sign-On as an identity
source. You can still define and manage local users host-by-host and configure them using the
vSphere Client. You can use this approach either in place of or in addition to the vCenter Single
Sign-On and AD integration.
Whenever asked to provide credentials (for example, when using VMware Host Client to log in
directly to the ESXi host), you can enter the user name and password of a user in the domain to
which the host is joined. The advantage of this user account management model is that you can
use AD to manage your ESXi host user accounts. This model is easier and more secure than trying
to manage accounts independently per host.

Module 11: Appendix: vSphere Security 767

The only user that is defined on the ESXi host is root. The root password is not mapped to an AD
account. The initial root password is typically set during ESXi installation, but it can be changed
later through VMware Host Client, vSphere ESXi Shell, or the DCUI.
You can create other domain groups if you only want to grant administrative rights to a subset of
hosts. You give that domain group administrator rights on the specific hosts and add the domain
users to the group.

768 Module 11: Appendix: vSphere Security

11-14 VMware vSphere Authentication Proxy Service

You can use vSphere Authentication Proxy to join the host to an AD domain, instead of adding the
host directly to the AD domain. A chain of trust exists between vSphere Authentication Proxy and
the host, which allows vSphere Authentication Proxy to join the host to the AD domain.
vSphere Authentication Proxy is especially useful when used with vSphere Auto Deploy. You can
set up a reference host that points to vSphere Authentication Proxy and set up a rule that applies
the reference host profile to any ESXi host provisioned with vSphere Auto Deploy. Even if you
use vSphere Authentication Proxy in an environment that uses certificates, provisioned by
VMware CA or third-party certificates, the process works seamlessly if you follow the instructions
for using custom certificates with vSphere Auto Deploy.
The vSphere Authentication Proxy service is available on each vCenter Server system. By default,
the service is not running. If you want to use vSphere Authentication Proxy in your environment,
you can start the service from the virtual appliance management interface (VAMI) or from the
command line.

Module 11: Appendix: vSphere Security 769

11-15 Review of Learner Objectives

770 Module 11: Appendix: vSphere Security

11-16 Lesson 2: Securing vSphere

Module 11: Appendix: vSphere Security 771

11-17 Learner Objectives

772 Module 11: Appendix: vSphere Security

11-18 About Securing vSphere

VMware security advisories provide information about security vulnerabilities that are reported in
VMware products. You can sign up to receive new and updated advisories by email on the
VMware Security Advisories webpage at http://www.vmware.com/security/advisories.html.

Module 11: Appendix: vSphere Security 773

11-19 About the vSphere Security Configuration Guide

774 Module 11: Appendix: vSphere Security

11-20 Controlling Access to vCenter Server Systems

Starting with vSphere 6.0, the local administrator no longer has full administrative rights to
vCenter Server by default. Instead, you assign the vCenter Server Administrator role to one or
more named vCenter Server administrator accounts.
Assign the Browse Datastore privilege only to users or groups who really need the privilege. Users
with the privilege can view, upload, or download files on datastores associated with the vSphere
deployment through the vSphere Client.
Verify vSphere Client certificates. Without certificate verification, the user might be subject to
man-in-the-middle attacks.
By default, vCenter Server automatically changes the vpxuser password every 30 days. You can
change this frequency in the vSphere Client.
Ensure that vSphere management traffic is on a restricted network.
For the complete list of vCenter Server best practices, see vSphere Security at
https://docs.vmware.com/en/VMware-vSphere/7.0/vsphere-esxi-vcenter-server-70-security-
guide.pdf.

Module 11: Appendix: vSphere Security 775

11-21 vCenter Server Access Control

Users who are directly logged in to the vCenter Server instance can cause harm, either
intentionally or unintentionally, by altering settings and modifying processes. Those users also
have potential access to vCenter Server credentials, such as the SSL certificate.
Consider auditing login events regularly to ensure that only authorized users are accessing vCenter
Server.
Use the VAMI to control access to SSH, the DCUI, the console command line, and the Bash shell
on a vCenter Server Appliance instance.

776 Module 11: Appendix: vSphere Security

11-22 Securing vCenter Server Appliance

Follow security best practices for vCenter Server Appliance instances:

• Ensure that all systems use the same relative time source. Synchronized systems are essential
for certificate validation. Network Time Protocol (NTP) also makes it easier to track an
intruder in log files.

• Restrict access to components that are required to communicate with the vCenter Server
Appliance instance. Blocking access from unnecessary systems reduces the potential for
attacks on the operating system.

VMware makes patches available on a monthly basis. These patches might be related to third-
party products in the platform, core product functionality, or both. The patches can only be applied
between major releases of vCenter Server Appliance. For example, patches for the initial release
of vCenter Server Appliance 6.7 are not applicable to vCenter Server Appliance 6.7 Update 1
because patches that are previously made available are included with the Update 1 release.

Module 11: Appendix: vSphere Security 777

For more information about how to patch vCenter Server Appliance, see vCenter Server Upgrade
at https://docs.vmware.com/en/VMware-vSphere/7.0/vsphere-vcenter-server-70-upgrade-
guide.pdf.

778 Module 11: Appendix: vSphere Security

11-23 Securing vSphere Management Networks

Create and install the vCenter Server system on a management network whenever possible. Avoid
putting the vCenter Server system on production or storage networks, or on a network with access
to the Internet.
vCenter Server systems require network connectivity with the following systems:

• ESXi hosts

• Other vCenter Server systems in the same vCenter Single Sign-On domain

• Infrastructure services such as DNS, AD, and NTP

• Other systems that run components that are essential to the functionality of the vCenter Server
system

Module 11: Appendix: vSphere Security 779

For the best protection of your hosts, ensure that physical switch ports are configured with the
Spanning Tree Protocol disabled and that the nonnegotiate option is configured for trunk links
between external physical switches and virtual switches in Virtual Switch Tagging mode.

780 Module 11: Appendix: vSphere Security

11-24 Securing ESXi Hosts

Create at least one named user account and assign it full administrative privileges. This user
account can be either a local account or a domain account that belongs to the ESX Admins domain
group. Use this account instead of the root account. Set a complex password for the root account
and limit its use, but do not remove the root user.
You can configure ESXi hosts to use AD to manage users. Using AD alleviates maintaining local
user accounts on each ESXi host.
ESXi includes a firewall that is enabled by default. You can manage firewall settings by using
security profiles. Many essential security settings for ESXi hosts can be customized using security
profiles. Using security profiles is especially useful for single host management.
You can use SSH to restrict, control, and secure access to an ESXi host. SSH access and vSphere
ESXi Shell access are disabled on ESXi hosts by default.
Lockdown mode is disabled by default. When you use lockdown mode, operations on ESXi hosts
must be performed through vCenter Server.

Module 11: Appendix: vSphere Security 781

You manage certificates from the vSphere Client. ESXi hosts can use VMware CA, a custom
certificate authority, or thumbprint mode. By default, a certificate signed by VMware CA is
provisioned when you boot an ESXi host from installation media.
For the complete list of ESXi hypervisor and ESXi host security best practices, see vSphere
Security at https://docs.vmware.com/en/VMware-vSphere/7.0/vsphere-esxi-vcenter-server-70-
security-guide.pdf.

782 Module 11: Appendix: vSphere Security

11-25 VM Protection Overview

Using templates to deploy VMs removes the risk of misconfiguration. All VMs are created with a
known baseline level of security.

Module 11: Appendix: vSphere Security 783

11-26 Review of Learner Objectives

784 Module 11: Appendix: vSphere Security

11-27 Lesson 3: Security Standards and Protocols

Module 11: Appendix: vSphere Security 785

11-28 Learner Objectives

786 Module 11: Appendix: vSphere Security

11-29 Securing vCenter Server with TLS 1.2

For a list of VMware products that support disabling TLS 1.0 and TLS 1.1, see VMware
knowledge base article 2145796 at http://kb.vmware.com/kb/2145796.
For more information about the TLS Configurator utility, see Managing TLS Protocol
Configuration with the TLS Configurator Utility at https://docs.vmware.com/en/VMware-
vSphere/7.0/com.vmware.vsphere.security.doc/GUID-82028A21-8AB5-4E2E-90B8-
A01D1FAD77B1.html.

Module 11: Appendix: vSphere Security 787

11-30 Activity: ESXi Secure Boot (1)

788 Module 11: Appendix: vSphere Security

11-31 Activity: ESXi Secure Boot (2)

Module 11: Appendix: vSphere Security 789

11-32 vSphere Compliance with FIPS 140-2 (1)

The Federal Information Processing Standards (FIPS) 140-2 standard specifies and validates the
cryptographic and operational requirements for the modules within security systems that protect
sensitive information. These modules use National Institute of Standards and Technology (NIST)
approved security functions such as cryptographic algorithms, key sizes, key management, and
authentication techniques.
For more information about the FIPS security certification and validation for vSphere, see
VMware Certifications at https://www.vmware.com/support/support-resources/certifications.html.

790 Module 11: Appendix: vSphere Security

11-33 vSphere Compliance with FIPS 140-2 (2)

FIPS 140-2 mode is not compatible with third-party tools that do not enable cryptographic
modules.

Module 11: Appendix: vSphere Security 791

11-34 FIPS 140-2 Architecture

The vmkcrypto module contains the encryption algorithms used by ESXi.

792 Module 11: Appendix: vSphere Security

11-35 Enabling and Disabling FIPS 140-2 Mode on ESXi
Hosts

With the esxcli system security fips140 command, you can view and modify the
FIPS-140 settings for ssh and rhttpproxy.
To enable FIPS-140 on ssh and rhttpproxy, run set –-enable true.

To disable FIPS-140, run set --enable false.

Module 11: Appendix: vSphere Security 793

11-36 Enabling and Disabling FIPS 140-2 Mode on vCenter
Server Appliance

With the fipsctl command, you can view and modify the FIPS-140 settings for the rhttpproxy,
sshd, and vmca services on vCenter Server Appliance.
To enable FIPS-140 on rhttpproxy, ssh, and vmca, run enable –-service
service_name.

To disable FIPS-140, run disable --service service_name.

794 Module 11: Appendix: vSphere Security

11-37 Activity: Security Technologies (1)

Module 11: Appendix: vSphere Security 795

11-38 Activity: Security Technologies (2)

796 Module 11: Appendix: vSphere Security

11-39 Review of Learner Objectives

Module 11: Appendix: vSphere Security 797

11-40 Lesson 4: Securing VMs

798 Module 11: Appendix: vSphere Security

11-41 Learner Objectives

Module 11: Appendix: vSphere Security 799

11-42 VM Protection with UEFI Secure Boot

For certain VM hardware versions and operating systems, you enable UEFI Secure Boot exactly
as you enable it for a physical machine.
In an operating system that supports UEFI Secure Boot, each piece of boot software is signed,
including the bootloader, the operating system kernel, and operating system drivers. The VM’s
default configuration includes several code-signing certificates:

• A Microsoft certificate that is used only for booting Windows

• A Microsoft certificate that is used for third-party code and signed by Microsoft, such as
Linux bootloaders

• A VMware certificate that is used only for booting ESXi inside a VM

The VM’s default configuration includes one certificate for authenticating requests to modify the
UEFI Secure Boot configuration, including the UEFI Secure Boot revocation list from inside the
VM, which is a Microsoft key encryption key (KEK) certificate. In most cases, replacing the
existing certificates is not necessary.

800 Module 11: Appendix: vSphere Security

For a list of operating systems that support UEFI Secure Boot, see the VMware Compatibility
Guide at http://www.vmware.com/resources/compatibility.

Module 11: Appendix: vSphere Security 801

11-43 Using vTPM in a VM (1)

Compromising the guest operating system usually compromises its secrets, but enabling a vTPM
in the guest reduces this risk.
You can add a vTPM device to either a new VM or an existing VM.

802 Module 11: Appendix: vSphere Security

11-44 Using vTPM in a VM (2)

vTPM can be enabled on a VM by editing the VM’s settings with the vSphere Client.
vTPM can be removed from a VM but the VM must be powered off first. Before you remove
vTPM, disable any applications that use vTPM. If you do not disable these applications, the VM
might fail to boot when powered on.

Module 11: Appendix: vSphere Security 803

11-45 Prerequisites for Adding vTPM

vTPM does not require a physical TPM 2.0 chip to be present on the ESXi host. However, if you
want to perform host attestation, an external entity, such as a TPM 2.0 physical chip, is required.

804 Module 11: Appendix: vSphere Security

11-46 Virtualization-Based Security for Windows Guest
Operating Systems

Microsoft virtualization-based security (VBS), a feature of Windows 10 and Windows Server

2016 and later operating systems, uses hardware and software virtualization to enhance system
security by creating an isolated, hypervisor-restricted, specialized subsystem.

With VBS, you can use the following Windows security features to harden your system and
isolate key system and user secrets so that they are not compromised:

• Credential Guard: Aims to isolate and harden key system and user secrets against compromise

• Device Guard: Provides a set of features that work together to prevent and eliminate malware
from running on a Windows system

• Configurable Code Integrity: Ensures that only trusted code runs from the boot loader
onwards

Module 11: Appendix: vSphere Security 805

For more information about virtualization-based security, see the Microsoft documentation.

806 Module 11: Appendix: vSphere Security

11-47 VBS Requirements and Limitations

Module 11: Appendix: vSphere Security 807

11-48 Enabling VBS

808 Module 11: Appendix: vSphere Security

11-49 About SGX

Trusted computing refers to various technologies and proposals for resolving computer security
problems. The problems are resolved through hardware enhancements and associated software
modifications. Several major hardware manufacturers and software vendors, collectively known as
the Trusted Computing Group (TCG), cooperate in this venture and create specific plans.
For more information about the Trusted Computing Group, see https://trustedcomputinggroup.org.
Data inside an application can be compromised by rogue software or a bug in the operating system
or in the hypervisor.
Programmers can use Intel SGX to create private memory regions called enclaves. Data in
enclaves is protected from unauthorized access and modification by rogue software running at
higher privilege levels. Intel SGX provides this protection without disrupting the ability of
legitimate software to schedule and manage the use of platform resources.
vSGX is implemented as part of the vSphere core virtualization stack. With vSGX, applications
can run on VMs to create their own enclaves.

Module 11: Appendix: vSphere Security 809

11-50 vSGX Use Cases

Many third-party applications can use vSGX.

810 Module 11: Appendix: vSphere Security

11-51 SGX Terminology

The Enclave Page Cache (EPC) is a static portion of physical RAM. This portion is allocated at
boot time by the BIOS. EPC stores running enclaves. It is a limited resource, typically not more
than 93 MB.
Launch control configuration is an architectural component from which the platform can control
which enclaves can be launched.

Module 11: Appendix: vSphere Security 811

11-52 Intel SGX Architecture

Applications frequently work with private information such as passwords, encryption keys, bank
account numbers, and so on. This private data should be accessed only by the designated recipient.
The job of the operating system is to protect such private data from other applications and users.
Operating systems and applications often employ safeguards to protect this private data. Despite
these protections, most computer systems are still vulnerable.
Malware with administrative privileges can access all operating system resources including all
applications running on that system. Malware can target an application's protected data to extract
encryption keys and secret data directly from memory.
With Intel SGX, applications can create enclaves, which protect the confidentiality and integrity of
applications from privileged malware.
Privileged malware is malware that gets privileges by manipulating the underlying system.

812 Module 11: Appendix: vSphere Security

11-53 vSGX Requirements

In theory, all versions of Linux are supported. In practice, a special kernel module published by
Intel must be loaded. This special kernel module is available for only a few Linux distributions.
For more information about this module, see Intel SGX SDK at https://01.org/intel-softwareguard-
extensions and Intel SGX SDK download at https://01.org/intel-software-guard-
extensions/downloads.

Module 11: Appendix: vSphere Security 813

11-54 vSGX Restrictions

Cold migrations can be performed.

814 Module 11: Appendix: vSphere Security

11-55 Enabling and Disabling vSGX for a VM

vSGX can be enabled or disabled using the Edit Settings menu for the VM.
The VM must be powered off to enable or disable the feature.

Module 11: Appendix: vSphere Security 815

11-56 Review of Learner Objectives

816 Module 11: Appendix: vSphere Security

11-57 Lesson 5: VM Encryption

Module 11: Appendix: vSphere Security 817

11-58 Learner Objectives

818 Module 11: Appendix: vSphere Security

11-59 About VM Encryption

Introduced in vSphere 6.5, VM encryption protects not only the data on disk but also the swap file
and any guest-specific information that might be contained in the VMX or NVRAM files. VM
encryption also provides a multilayer key protection mechanism that makes breaking the security
of the VM almost impossible.
VM encryption makes it simple to configure encryption by using the orchestration capabilities of
vCenter Server storage policies. By using storage policies, you can apply encryption to any type of
VM, regardless of the guest operating system or the storage on which it resides.
By using external, enterprise-grade key servers, exposure to risk is limited because vSphere does
not need to manage keys or keep key data on disk anywhere in the vSphere environment. Using
the industry-standard Key Management Interoperability Protocol (KMIP), vSphere can be
integrated with various third-party key management server (KMS) vendors and products.
You can limit access to cryptographic operations by enforcing roles and permissions for users.
Only users with the required permissions can perform cryptographic tasks such as encrypting or
decrypting a VM.

Module 11: Appendix: vSphere Security 819

11-60 Overview of VM Encryption

The enhancements in vSphere 7 improve the general functionality of VM encryption. They are
based on competitive analysis and feedback from VMware customers and partners.
To mitigate the risk of an administrator accessing data without authorization, VMs can be
encrypted, and access to the encryption and decryption functionality can be provided to a subset of
administrators. You can provide this type of encryption by using the inbuilt cryptographic
privileges in vSphere.

820 Module 11: Appendix: vSphere Security

11-61 Advantages of VM Encryption

Module 11: Appendix: vSphere Security 821

11-62 Business Use Case: Securing VMs

The digital key used to encrypt a VM is secured in an extra layer of encryption, much like taking
the key needed to open a safe and placing it in another safe.
You allow regular vSphere administrators to continue with their daily activities without change,
but you grant cryptographic privileges to a subset of these administrators.
Access to VM files does not mean that storage administrators or vSphere administrators can walk
away with the VM (VMDK) file and read its contents offsite.

822 Module 11: Appendix: vSphere Security

11-63 VM Encryption Interoperability (1)

vSphere VM encryption has limitations in the devices and features that it can interoperate with in
vSphere 6.5 and later releases:

• Full clones are supported. The clone inherits the parent encryption state, including keys. You
can re-encrypt a full clone to use new keys or decrypt the full clone.

• Linked clones are supported, and the clone inherits the parent encryption state including keys.
You cannot decrypt the linked clone or re-encrypt a linked clone with different keys.

• Starting with vSphere 6.7, you can resume from a suspended state of an encrypted VM or
revert to a memory snapshot of an encrypted VM. You can migrate an encrypted VM with
memory snapshot and suspended state between ESXi hosts.

Module 11: Appendix: vSphere Security 823

• You cannot perform the following tasks on an encrypted VM:
– For most VM encryption operations, the VM must be powered off. You can clone an
encrypted VM, and you can perform a shallow recrypt while the VM is powered on.
– You cannot encrypt a VM that has existing snapshots. Consolidate all existing snapshots
before you perform the encryption.

• You can use vSphere VM encryption with IPv6 mode or in mixed mode. You can configure
the KMS with IPv6 addresses. Both vCenter Server and the KMS can be configured with only
IPv6 addresses.

For more information about VM encryption interoperability, see

https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.security.doc/GUID-
C0AF1F3A-67B4-41A6-A933-7E52A3603D9D.html.

824 Module 11: Appendix: vSphere Security

11-64 VM Encryption Interoperability (2)

Encrypted migration with vSphere vMotion of an unencrypted VM is supported.

You cannot send output from an encrypted VM to a serial port or parallel port. Even if the
configuration appears to succeed, output is sent to a file.
Certain types of VMDK configurations are not supported with vSphere VM encryption:

• Flash Read Cache.

• A named virtual disk unassociated with a VM, also called First Class Disk.

• RDM (Raw Device Mapping).

• Multiwriter or shared disks (MSCS, WSFC, or Oracle RAC). If a virtual disk is encrypted,
and you select Multi-writer on the VM's Edit Settings page, the OK button is disabled.

Module 11: Appendix: vSphere Security 825

For more information, see Virtual Machine Encryption Interoperability at
https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.security.doc/GUID-
C0AF1F3A-67B4-41A6-A933-7E52A3603D9D.html.

826 Module 11: Appendix: vSphere Security

11-65 VM Encryption Architecture

Module 11: Appendix: vSphere Security 827

11-66 VM Encryption Prerequisites

To encrypt VMs, you must create a storage policy with the encryption filter enabled.
You create a common rule in the storage policy that defines the encryption properties. The default
encryption properties are appropriate in most cases. You create a custom policy only if you want
to combine encryption with other features, such as caching or replication.
You must set the Allow I/O filters before encryption value to True to enable the use of a
replication filter before the encryption filter. The replication filter sees plain text before it is
encrypted.

828 Module 11: Appendix: vSphere Security

11-67 Encrypting New VMs

To encrypt a new VM, perform the following steps:

1. In the New Virtual Machine wizard, select the Encrypt this virtual machine check box to
show only the encryption policies.
2. Select an encryption policy from the VM Storage Policy drop-down menu.

Module 11: Appendix: vSphere Security 829

11-68 Encrypting Existing VMs

For the data of an existing VM, a duplicate disk is created with the encryption I/O filter applied.
Data is then copied from the unencrypted disk to the new encrypted disk, applying the encryption
during the copy process. After all the data is copied, the new encrypted VMDK file is attached to
the VM, and the old unencrypted disk is deleted.
The VM must be powered off so that no mirror driver is employed.

830 Module 11: Appendix: vSphere Security

11-69 Unlocking Encrypted VMs (1)

Module 11: Appendix: vSphere Security 831

11-70 Unlocking Encrypted VMs (2)

832 Module 11: Appendix: vSphere Security

11-71 About the KMS

For vSphere, the KMS must be a server that communicates using the KMIP protocol. vCenter
Server implements a KMIP client that can issue KMIP-formatted commands to request keys using
specific identifiers. A KMIP server can return the key values to vCenter Server using a secure
channel.
Having a KMS means that all the management tasks relating to keys can be centrally managed on
a single server or server cluster. Multiple vCenter Server instances can access the same KMS
cluster and can be granted access to the same keys. Having a centrally managed key store reduces
the attack surface for your key management. Instead of keys existing in multiple locations, which
you must secure, a single or limited number of servers are secured, which is easier and safer.
KMIP proxy servers can provide the key management capability of the KMS to a remote location.
With KMIP proxy servers, you can keep your critical key server safe in your central data center
while still allowing remote offices to use its services without compromising security.
Finally, using the KMIP requires only an IP-capable network, which is readily available in most
data centers.

Module 11: Appendix: vSphere Security 833

11-72 KMIP Client Certificate

Different KMS vendors have varying requirements for the format of the client certificate that is
used by vCenter Server.
Some vendors allow the use of the client’s own certificate, such as a self-signed or CA-signed
certificate. Other vendors require that the certificate used by the client be signed by the KMS
server.
You can use the vSphere Client to provide certificate details based on the security requirements of
the different types of implementations.

834 Module 11: Appendix: vSphere Security

11-73 Key Management Clusters

To protect against the potential failure of a KMS (also called a KMIP server), KMIP clusters are
formed. These clusters provide some failover protection. vCenter Server keeps a list of key
management servers in the cluster. If the first KMS on the list is not available in the cluster,
vCenter Server tries to communicate with the next KMS on the list, and so on.
To form a KMIP cluster, the KMIP servers must be from a single vendor. See the vendor
documentation for steps on how to create a KMIP cluster.
After the keys are replicated, a client can request the key from any of the servers participating in
the cluster. So if one server goes down, another server can provide the key.
Although multiple KMIP clusters can be added to vCenter Server, one cluster must be identified
as the default cluster. The default cluster is the cluster from which vCenter Server requests new
keys. Keys can be retrieved from nondefault clusters if the cluster is identified. Clusters of KMIP
servers are identified using a user-defined friendly name, populated in the vSphere Client.

Module 11: Appendix: vSphere Security 835

11-74 Summary of Keys

836 Module 11: Appendix: vSphere Security

11-75 Role of vCenter Server in VM Encryption

vCenter Server stores the KMS login information, and it is the only vSphere component that can
communicate with the KMS and retrieve keys. An ESXi host itself cannot request keys from the
KMS. vCenter Server performs key management tasks on behalf of the ESXi hosts.
vCenter Server provides a mechanism whereby keys can be identified by a unique identifier,
which can be stored in a VM’s VMX file or VMDK file. vCenter Server can use these identifiers to
request keys from the KMS server when required and then push these keys to the ESXi hosts.
vCenter Server provides the framework for managing permissions for cryptographic tasks. You
can restrict access to critical operations to a subset of your administrators, while ensuring that your
other administrators can continue with their daily tasks.
vCenter Server manages storage policies that define whether a VM is encrypted.
vCenter Server records events for auditing purposes. Event information includes the identity of the
user who initiated the event.

Module 11: Appendix: vSphere Security 837

11-76 KMS and vCenter Server Communication

After trust is established between the KMS (KMIP server) and the vCenter Server (KMIP client),
vCenter Server stores the KMS credential information. The VMware Endpoint Certificate Store
(VECS) stores the SSL private key information.

838 Module 11: Appendix: vSphere Security

11-77 Making an ESXi Host Cryptographically Safe

Users with the required privileges can perform cryptographic operations. These operations include
creating encrypted VMs, encrypting existing VMs, and decrypting encrypted VMs.
When host encryption mode is enabled on one host in a cluster, it is also enabled for all other hosts
in the cluster. Host encryption mode cannot be disabled while hosts in the cluster are managing
encrypted VMs.

Module 11: Appendix: vSphere Security 839

11-78 VM Encryption Process Flow (1)

When a cryptographic operation occurs, vCenter Server must determine whether a VM key must
be pushed to the ESXi hosts.
vCenter Server retrieves the key encryption key (KEK) from the KMS, but the KEK is never
stored anywhere on vCenter Server, not even in its memory. The KEK remains in vCenter Server
memory only while the key is in transit to the ESXi host. vCenter Server passes the KEK to the
ESXi host that is performing the cryptographic operation.
If the host belongs to a cluster, the cluster is deemed the security boundary. In this case, vCenter
Server gives the KEK to all hosts in the cluster so that any cluster operations, such as vSphere
vMotion migrations and vSphere HA operations, can occur.

840 Module 11: Appendix: vSphere Security

11-79 VM Encryption Process Flow (2)

If an ESXi host reboots, a VM’s KEK is no longer in the host’s memory. vCenter Server requests
the KEK with the corresponding key ID from the KMS and makes it available to the ESXi host
when the host is running. The ESXi host can then decrypt the VM’s DEK, as needed.
Because KEKs are stored in the host’s memory and might be discovered if a core dump is
analyzed, vSphere encrypts all core dumps to provide an extra level of security.

Module 11: Appendix: vSphere Security 841

11-80 Activity: VM Keys (1)

842 Module 11: Appendix: vSphere Security

11-81 Activity: VM Keys (2)

Module 11: Appendix: vSphere Security 843

11-82 Managing VM Encryption

844 Module 11: Appendix: vSphere Security

11-83 vCenter Server Role: No Cryptography Administrator

A predefined role called No Cryptography Administrator is provided. You can also create custom
roles that include cryptographic privileges.
You do not want a user with the No Cryptography Administrator role to add hosts to the vCenter
Server inventory because this task might require the installation of a host key, making the task a
cryptographic operation.

Module 11: Appendix: vSphere Security 845

11-84 Activity: Cryptographic Privileges (1)

846 Module 11: Appendix: vSphere Security

11-85 Activity: Cryptographic Privileges (2)

Module 11: Appendix: vSphere Security 847

11-86 Pushing Keys to Hosts

vSphere DRS can be enabled or not. Only vSphere HA is checked.

848 Module 11: Appendix: vSphere Security

11-87 Cross vCenter Migration and Cloning of an Encrypted
VM

During the migration operation, you cannot change keys and, therefore, recrypting the VM during
the migration is not supported. Also, you cannot change the storage policy during migration, and,
therefore, decrypting the VM during the migration is not supported.

Module 11: Appendix: vSphere Security 849

11-88 Architecture for Cross vCenter Server Migrations and
Clones (1)

850 Module 11: Appendix: vSphere Security

11-89 Architecture for Cross vCenter Server Migrations and
Clones (2)

VM keys must reside in the shared KMS cluster for VM migrations and cloning to occur between
vCenter Server instances.

Module 11: Appendix: vSphere Security 851

11-90 Architecture for Cross vCenter Server Migrations and
Clones (3)

852 Module 11: Appendix: vSphere Security

11-91 Performing a Shallow Recrypt of VMs with Snapshots

A deep recryption replaces all keys in a VM with new keys, from top to bottom. This operation
requires rewriting all encrypted data in the VM.
A shallow recryption replaces only the top-level managed keys in a VM, namely, the
VMDiskKey. Only a small amount of encrypted data must be rewritten.

Module 11: Appendix: vSphere Security 853

11-92 VM Encryption Operations When Cloning a VM

A shallow rekey (or recrypt) changes the KEK associated with the VM. The data encryption key
(DEK) is wrapped with the new KEK.
Deep rekey (or recrypt) changes the DEK and recrypts all data using the new DEK. This function
requires temporary disk space and is slower than a shallow rekey because all data must be
rewritten.

854 Module 11: Appendix: vSphere Security

11-93 Cloning an Encrypted VM with Instant Clone
Technology

You cannot change the storage policy or change encryption keys during the instant clone
operation.

Module 11: Appendix: vSphere Security 855

11-94 Cloning a VM: Encrypting or Decrypting the
Destination VM

Decrypting a VM can be a security risk. Therefore, as a best practice, restrict most users from
having the privilege to decrypt VMs.

856 Module 11: Appendix: vSphere Security

11-95 System Requirements When Cloning a VM

Module 11: Appendix: vSphere Security 857

11-96 Backing Up Encrypted VMs (1)

A transport mode defines the method that is used to transfer data to the backup server.
For more information about transport modes, see Virtual Disk Development Kit (VDDK) at
https://code.vmware.com/web/sdk/7.0/vddk.

858 Module 11: Appendix: vSphere Security

11-97 Backing Up Encrypted VMs (2)

If the VMis encrypted, SAN mode is immediately ruled out as an available transport mode. The
backup products that rely on this mode cannot back up encrypted VMs. The blocks are encrypted,
and without vSphere assistance, the backup server cannot use the data.
If the backup proxy is a physical machine, hot-add mode is ruled out, and NBD or NBD-SSL
mode is used. You rely on the ESXi host to open the disk, decrypt the data, and send the backup
data in plain text to the backup server.
If the user with which you register the backup appliance to vCenter Server does not have
cryptographic privileges, the disk cannot be decrypted to send the data to the backup server.
For any of the transport modes, the backup data is stored on the backup server in plain text. In this
way, a VM can be restored as a plain-text machine. Privileges to restore VMs should be granted
only to trusted individuals, most likely users with cryptographic privileges, so that the VM can be
immediately re-encrypted after it is restored.

Module 11: Appendix: vSphere Security 859

11-98 About Encrypted Core Dumps

A core dump is useful when you must debug system failures, especially purple screen failures. If
the VMkernel fails, this failure usually results in a purple screen. It is the most common type of
core dump analyzed by VMware technical support.
If any ESXi hosts in a cluster are running encrypted VMs and an ESXi host fails, the host creates
an encrypted core dump that is stored in the /var/core folder on the ESXi host. Core dumps
that are included in the vm-support package are also encrypted.
The core dump is encrypted using the host key.

860 Module 11: Appendix: vSphere Security

11-99 Core Dump Encryption

Normally, the host key should be available only on the host or cluster where the original host
resides. Therefore, an encrypted core dump is not useful to a VMware technical support person
who is trying to help you debug a system failure.
The vm-support command collects core dumps as is. That is, the host key is required to read
them.
Digital envelope refers to an encryption methodology that uses two keys. vSphere encrypts and
decrypts core dumps using an internal one-time key and the ESXi host key.

Module 11: Appendix: vSphere Security 861

11-100 Providing Passwords for Encrypted Core Dumps

By providing a password, the core dumps are reencrypted with a password-protected envelope.
During the support log bundle collection, a new envelope file is created with a new incident key,
and the envelope is protected by a password.
Using a secure mechanism, you provide the password to VMware technical support, or appropriate
support staff, who can view the core dump contents by using the password.
You can set a password for encrypted core dumps when you export the system logs.

862 Module 11: Appendix: vSphere Security

11-101 VM Encryption Events and Alarms

Module 11: Appendix: vSphere Security 863

11-102 Warning: Encrypted Core Dump Exists (1)

As with vSphere 6.x, you can decrypt or recrypt an encrypted core dump on your ESXi host by
using the crypto-util CLI.

864 Module 11: Appendix: vSphere Security

11-103 Warning: Encrypted Core Dump Exists (2)

Module 11: Appendix: vSphere Security 865

11-104 Critical Event: Insufficient Disk Space

866 Module 11: Appendix: vSphere Security

11-105 Critical Event: Key Generation Failed

Search for Key creation failed in the /var/log/vmware/vpxd/vpxd.log file.

Verify that the KMS is powered on and operating properly.
Verify that the KMS and vCenter Server can communicate with each other.

Module 11: Appendix: vSphere Security 867

11-106 Critical Event: KMS Unreachable

868 Module 11: Appendix: vSphere Security

11-107 About Encrypted vSphere vMotion

Encrypted vSphere vMotion secures communication on the vMotion network so that sensitive data
cannot be discovered during memory migration.
vCenter Server generates a one-use (nonce) encryption key and includes this key in a
communication (or migration specification) to both the source and the destination ESXi hosts.
This key is used to encrypt and decrypt the data sent over the vMotion network.
In a vSphere vMotion migration across vCenter Server instances, the local vCenter Server instance
generates the encryption key and includes the key in a communication over an SSL-secured
channel to the remote vCenter Server instance. The two vCenter Server instances pass the required
keys to their local ESXi hosts, and these hosts communicate directly using these keys.
For encrypted VMs, the VM’s memory is encrypted over the vMotion network. The disks are
already encrypted and transmitted over the vMotion network as is (encrypted).
For nonencrypted VMs, the VM’s memory is encrypted over the vMotion network, but the disks
are transmitted as is (nonencrypted).

Module 11: Appendix: vSphere Security 869

For VMs that are encrypted, migration with vSphere vMotion always uses encrypted vSphere
vMotion. You cannot turn off encrypted vSphere vMotion for encrypted VMs.

870 Module 11: Appendix: vSphere Security

11-108 Configuring Encrypted vSphere vMotion

Module 11: Appendix: vSphere Security 871

11-109 Encrypted vSphere vMotion Requirements

872 Module 11: Appendix: vSphere Security

11-110 Review of Learner Objectives

Module 11: Appendix: vSphere Security 873

11-111 Key Points

874 Module 11: Appendix: vSphere Security