Vmware Vsphere 5 Clustering Technical Deepdive

Frank Denneman download

https://ebookbell.com/product/vmware-vsphere-5-clustering-
technical-deepdive-frank-denneman-2373264

Explore and download more ebooks at ebookbell.com

Here are some recommended products that we believe you will be
interested in. You can click the link to download.

Vmware Vsphere 55 Essentials Kunal Kumar Christian Stankowic

https://ebookbell.com/product/vmware-vsphere-55-essentials-kunal-
kumar-christian-stankowic-50194502

Vmware Vsphere 55 Cookbook A Taskoriented Guide With Over 150

Practical Recipes To Install Configure And Manage Vmware Vsphere
Components Abhilash G B

https://ebookbell.com/product/vmware-vsphere-55-cookbook-a-
taskoriented-guide-with-over-150-practical-recipes-to-install-
configure-and-manage-vmware-vsphere-components-abhilash-g-b-5470116

Vmware Vsphere 51 Cookbook Abhilash Gb

https://ebookbell.com/product/vmware-vsphere-51-cookbook-abhilash-
gb-5497718

Vmware Vsphere 5 Administration Instant Reference 2nd Edition

Christopher Kusek

https://ebookbell.com/product/vmware-vsphere-5-administration-instant-
reference-2nd-edition-christopher-kusek-47135906
Mastering Vmware Vsphere 5 Lowe Scott

https://ebookbell.com/product/mastering-vmware-vsphere-5-lowe-
scott-22091670

Mastering Vmware Vsphere 5 Scott Lowe Lowe Scott

https://ebookbell.com/product/mastering-vmware-vsphere-5-scott-lowe-
lowe-scott-32483206

Vmware Vsphere 65 Cookbook 3rd Edition Abhilash G B Cedric Rajendran

https://ebookbell.com/product/vmware-vsphere-65-cookbook-3rd-edition-
abhilash-g-b-cedric-rajendran-7011478

Vmware Vsphere 65 Host Resources Deep Dive Frank Denneman Niels

Hagoort

https://ebookbell.com/product/vmware-vsphere-65-host-resources-deep-
dive-frank-denneman-niels-hagoort-6761314

Mastering Vmware Vsphere 65 Leverage The Power Of Vsphere For

Effective Virtualization Administration Management And Monitoring Of
Data Centers 1st Edition Andrea Mauro

https://ebookbell.com/product/mastering-vmware-vsphere-65-leverage-
the-power-of-vsphere-for-effective-virtualization-administration-
management-and-monitoring-of-data-centers-1st-edition-andrea-
mauro-7351708
VMware vSphere 5.0
Clustering
Technical Deepdive
VMware vSphere 5.0 Clustering Technical Deepdive

Copyright © 2011 by Duncan Epping and Frank Denneman.

All rights reserved. No part of this book shall be reproduced, stored in a retrieval system, or transmitted by any means, electronic, mechanical, or
otherwise, without written permission from the publisher. No patent liability is assumed with respect to the use of the information contained herein.
Although every precaution has been taken in the preparation of this book, the publisher and authors assume no responsibility for errors or
omissions. Neither is any liability assumed for damages resulting from the use of the information contained herein.

All terms mentioned in this book that are known to be trademarks or service marks have been appropriately capitalized.

Use of a term in this book should not be regarded as affecting the validity of any trademark or service mark.

Version: 1.0
About the Authors

Duncan Epping is a Principal Architect working for VMware as part of Technical Marketing. Duncan specializes in vSphere HA, Storage DRS,
Storage I/O Control and vSphere Architecture. Duncan was among the first VMware Certified Design Experts (VCDX 007). Duncan is the owner of
Yellow-Bricks.com, one of the leading VMware/virtualization blogs worldwide (Voted number 1 virtualization blog for the 4th consecutive time on
vsphere-land.com.) and lead-author of the "vSphere Quick Start Guide" and co-author of "Foundation for Cloud Computing with VMware vSphere
4", “Cloud Computing with VMware vCloud Director” and “VMware vSphere 4.1 HA and DRS technical deepdive”. He can be followed on twitter at
http://twitter.com/DuncanYB.

Frank Denneman is a Consulting Architect working for VMware as part of the Professional Services Organization. Frank works primarily with
large Enterprise customers and Service Providers. He is focused on designing large vSphere Infrastructures and specializes in Resource
Management, vSphere DRS and storage. Frank is among the first VMware Certified Design Experts (VCDX 029). Frank is the owner of
FrankDenneman.nl which has recently been voted number 6 worldwide on vsphere-land.com and co-author of “VMware vSphere 4.1 HA and DRS
technical deepdive”. He can be followed on twitter at http://twitter.com/FrankDenneman.
Table of Contents
Acknowledgements
Foreword
Introduction to vSphere High Availability
vSphere 5.0
What’s New?
What is Required for HA to Work?
Prerequisites
Firewall Requirements
Configuring VMware High Availability
Components of High Availability
HOSTD Agent
vCenter
Fundamental Concepts
Master Agent
Slaves
Files for both Slave and Master
Heartbeating
Isolated versus Partitioned
Virtual Machine Protection
Restarting Virtual Machines
Restart Priority and Order
Restart Retries
Failed Host
Isolation Response and Detection
Selecting an Additional Isolation Address
Failure Detection Time
Restarting Virtual Machines
Corner Case Scenario: Split-Brain
Adding Resiliency to HA (Network Redundancy)
Link State Tracking
Admission Control
Admission Control Policy
Admission Control Mechanisms
Unbalanced Configurations and Impact on Slot Calculation
Percentage of Cluster Resources Reserved
Failover Hosts
Decision Making Time
Host Failures Cluster Tolerates
Percentage as Cluster Resources Reserved
Specify Failover Hosts
Recommendations
Selecting the Right Percentage
VM and Application Monitoring
Why Do You Need VM/Application Monitoring?
VM Monitoring Implementation Details
Screenshots
Application Monitoring
Advanced Options
Summarizing
Introduction to vSphere DRS
Cluster Level Resource Management
Requirements
DRS Cluster Settings
Automation Level
vCenter sizing
 vCenter sizing
DRS Cluster configuration
vMotion and EVC
vMotion
Enhanced vMotion Capability
DRS Resource Entitlement
Resource Scheduler Architecture
Dynamic Entitlement
Resource Allocation Settings
Shares
Limits
Tying It All Together
Resource Pools and Controls
Root Resource Pool
Resource Pools
Resource Pool Resource Allocation Settings
Shares
The Resource Pool Priority-Pie Paradox
From Resource Pool Setting to Host-Local Resource Allocation
Resource Pool-Level Reservation
Memory Overhead Reservation
Expandable Reservation
Reservations are not Limits.
Limits
Expandable Reservation and Limits
Calculating DRS Recommendations
When is DRS Invoked?
Recommendation Calculation
Imbalance Calculation
GetBestMove
Cost-Benefit and Risk Analysis Criteria
The Biggest Bang for the Buck
Calculating the Migration Recommendation Priority Level
Guidance
Guiding DRS Recommendations
Virtual Machine Size and Initial Placement
MaxMovesPerHost
Rules
Impact of Rules on Organization
Virtual Machine Automation Level
Introduction to DPM
Enable DPM
Calculating DPM Recommendations
Evaluating Resource Utilization
Power-Off recommendations
Power-On recommendations
Recommendation Classifications
Guiding DPM Recommendations
DPM WOL Magic Packet
Baseboard Management Controller
Protocol Selection Order
DPM Scheduled Tasks
Summarizing
Introduction to vSphere Storage DRS
Requirements
Datastore Clusters
Creating a Datastore Cluster
 Creating a Datastore Cluster
Datastore Cluster Creation and Considerations
Tools assisting Storage DRS
Datastore Cluster Settings
Automation Level
SDRS Runtime Rules
Advanced Options
Select Hosts and Clusters
Ready to Complete
Storage vMotion and SIOC
Storage vMotion
Storage I/O Control
Queuing Internals
Local Disk Scheduler
Datastore-Wide Disk Scheduler
Latency Threshold Recommendations
Injector
Calculating SDRS Recommendations
When is Storage DRS Invoked?
Initial Placement
Recommendation Calculation
Load balancing
Space Load Balancing
IO load balancing
Load Imbalance Recommendations
Guiding SDRS Recommendations
SDRS Recommendation
SDRS Virtual Machine Automation Level
Partially Connected Versus Fully Connected Datastore Clusters
Affinity and Anti-Affinity rules
Impact on Storage vMotion Efficiency
SDRS Scheduling
Maintenance Mode
Summarizing
Integration
HA and Stateless ESXi
HA and SDRS
Storage vMotion and HA
HA and DRS
Summarizing
Acknowledgements
The authors of this book work for VMware. The opinions expressed here are the authors’ personal opinions. Content published was
not approved in advance by VMware and does not necessarily reflect the views and opinions of VMware. This is the authors’ book,
not a VMware book.

First of all we would like to thank our VMware management team (Charu Chaubal, Kaushik Banerjee, Neil O’Donoghue and Bogomil
Balkansky) for supporting us on this and other projects.

A special thanks goes out to our technical reviewers and editors: Doug Baer, Keith Farkas and Elisha Ziskind(HA Engineering), Anne Holler, Irfan
Ahmad and Rajesekar Shanmugam(DRS and SDRS Engineering), Puneet Zaroo (VMkernel scheduling), Ali Mashtizadeh and Gabriel Tarasuk-
Levin (vMotion and Storage vMotion Engineering), Doug Fawley and Divya Ranganathan (EVC Engineering). Thanks for keeping us honest and
contributing to this book.

A very special thanks to our families for supporting this project. Without your support we could not have done this.

We would like to dedicate this book to everyone who made VMware to what it is today. You have not only changed our lives but also revolutionized
a complete industry.

Duncan Epping and Frank Denneman

Foreword
What are you reading? It’s a question that may come up as you’re reading this book in a public place like a coffee shop or airport. How do you
answer?

If you’re reading this book then you’re part of an elite group that “gets” virtualization. Chances are the person asking you the question has no idea
how to spell VMware, let alone how powerful server and desktop virtualization have become in the datacenter over the past several years.

My first introduction to virtualization was way back in 1999 with VMware Workstation. I was a sales engineer at the time and in order to effectively
demonstrate our products I had to carry 2 laptops (yes, TWO 1999 era laptops) PLUS a projector PLUS a 4 port hub and Ethernet cables. Imagine
how relieved my back and shoulders where when I discovered VMware Workstation and I could finally get rid of that 2nd laptop and associated
cabling. It was awesome to say the least!

Here we are, over a decade after the introduction of VMware Workstation. The topic of this book is HA & DRS, who would have thought of such
advanced concepts way back in 1999? Back then I was just happy to lighten my load…now look at where we are. Thanks to virtualization and
VMware we now have capabilities in the datacenter that have never existed before (sorry mainframe folks, I mean never before possible on x86).

Admit it, we’re geeks!

You may never see this book on the New York Times Bestseller list but that’s OK! Virtualization and VMware are very narrow topics, HA & DRS
even more narrow. You’re reading this because of your thirst for knowledge and I have to admit, Duncan and Frank deliver that knowledge like no
others before them.

Both Duncan and Frank have taken time out of their busy work schedules and dedicated themselves to not only blog about VMware virtualization,
but also write and co-author several books, including this one. It takes not only dedication, but also passion for the technology to continue to
produce great content.

The first edition, “VMware vSphere 4.1 HA and DRS technical deepdive” has been extremely popular, so popular in fact, that this book is all about
HA and DRS in vSphere 5.0. Not only the traditional DRS everyone has known for years but also Storage DRS. Storage DRS is something brand
new with vSphere 5.0 and it’s great to have this deep technical reference as it’s released even though it’s only a small part of VMware vSphere.
Use the information in the following pages not only to learn about this exciting new technology, but also to implement the best practices in your own
environment.

Finally, be sure to reach out to both Duncan and Frank either via their blogs, twitter or in person and thank them for getting what’s in their heads
onto the pages of this book. If you’ve ever tried writing (or blogging) then you know it takes time, dedication and passion.

Doug Hazelman
Senior Director, Product Strategy
Veeam Software

P.S. In case you didn’t know, Duncan keeps an album of book pictures that people have sent him on his blog’s Facebook page. So go ahead and
get creative and take a picture of this book and share it with him, just don’t get too creative… http://vee.am/HADRS
Part I

vSphere High Availability

Chapter 1
Introduction to vSphere High Availability

Availability has traditionally been one of the most important aspects when providing services. When providing services on a shared platform like
VMware vSphere, the impact of downtime exponentially grows and as such VMware engineered a feature called VMware vSphere High
Availability. VMware vSphere High Availability, hereafter simply referred to as HA, provides a simple and cost effective solution to increase
availability for any application running in a virtual machine regardless of its operating system. It is configured using a couple of simple steps through
vCenter Server (vCenter) and as such provides a uniform and simple interface. HA enables you to create a cluster out of multiple ESXi or ESX
servers. We will use ESXi in this book when referring to either, as ESXi is the standard going forward. This will enable you to protect virtual
machines and hence their workloads. In the event of a failure of one of the hosts in the cluster, impacted virtual machines are automatically restarted
on other ESXi hosts within that same VMware vSphere Cluster (cluster).

Figure 1: High Availability in action

On top of that, in the case of a Guest OS level failure, HA can restart the failed Guest OS. This feature is called VM Monitoring, but is sometimes
also referred to as VM-HA. This might sound fairly complex but again can be implemented with a single click.

Figure 2: OS Level HA just a single click

Unlike many other clustering solutions, HA is a fairly simple solution to implement and literally enabled within 5 clicks. On top of that, HA is widely
adopted and used in all situations. However, HA is not a 1:1 replacement for solutions like Microsoft Clustering Services (MSCS). The main
difference between MSCS and HA being that MSCS was designed to protect stateful cluster-aware applications while HA was designed to protect
any virtual machine regardless of the type of application within.

In the case of HA, a failover incurs downtime as the virtual machine is literally restarted on one of the remaining nodes in the cluster. Whereas
MSCS transitions the service to one of the remaining nodes in the cluster when a failure occurs. In contrary to what many believe, MSCS does not
guarantee that there is no downtime during a transition. On top of that, your application needs to be cluster-aware and stateful in order to get the
most out of this mechanism, which limits the number of workloads that could really benefit from this type of clustering.

One might ask why would you want to use HA when a virtual machine is restarted and service is temporarily lost. The answer is simple; not all virtual
machines (or services) need 99.999% uptime. For many services the type of availability HA provides is more than sufficient. On top of that, many
applications were never designed to run on top of an MSCS cluster. This means that there is no guarantee of availability or data consistency if an
application is clustered with MSCS but is not cluster-aware.

In addition, MSCS clustering can be complex and requires special skills and training. One example is managing patches and updates/upgrades in
a MSCS environment; this could even lead to more downtime if not operated correctly and definitely complicates operational procedures. HA
however reduces complexity, costs (associated with downtime and MSCS), resource overhead and unplanned downtime for minimal additional
costs. It is important to note that HA, contrary to MSCS, does not require any changes to the guest as HA is provided on the hypervisor level. Also,
VM Monitoring does not require any additional software or OS modifications except for VMware Tools, which should be installed anyway as a best
practice. In case even higher availability is required, VMware also provides a level of application awareness through Application Monitoring, which
has been leveraged by partners like Symantec to enable application level resiliency and could be used by in-house development teams to increase
resiliency for their application.

HA has proven itself over and over again and is widely adopted within the industry; if you are not using it today, hopefully you will be convinced after
reading this section of the book.
vSphere 5.0
Before we dive into the main constructs of HA and describe all the choices one has to make when configuring HA, we will first briefly touch on
what’s new in vSphere 5.0 and describe the basic requirements and steps needed to enable HA. The focus of this book is vSphere 5.0 and the
enhancements made to increase stability of your virtualized infrastructure. We will also emphasize the changes made which removed the historical
constraints. We will, however, still discuss features and concepts that have been around since vCenter 2.x to ensure that reading this book provides
understanding of every aspect of HA.
What’s New?
Those who have used HA in the past and have already played around with it in vSphere 5.0 might wonder what has changed. Looking at the
vSphere 5.0 Client, changes might not be obvious except for the fact that configuring HA takes substantially less time and some new concepts like
datastore heartbeats have been introduced.

Do not assume that this is it; underneath the covers HA has been completely redesigned and developed from the ground up. This is the reason
enabling or reconfiguring HA literally takes seconds today instead of minutes with previous versions.

With the redesign of the HA stack, some very welcome changes have been introduced to compliment the already extensive capabilities of HA.
Some of the key components of HA have changed completely and new functionality has been added. We have listed some of these changes below
for your convenience and we will discuss them in more detail in the appropriate chapters.

New HA Agent - Fault Domain Manager (FDM) is the name of the agent. HA has been rewritten from the ground up and FDM replaces the
legacy AAM agent.
No dependency on DNS – HA has been written to use IP only to avoid any dependency on DNS.
Primary node concept – The primary/secondary node mechanism has been completely removed to lift all limitations (maximum of 5 primary
nodes with vSphere 4.1 and before) associated with it.
Supports management network partitions – Capable of having multiple “master nodes” when multiple network partitions exist.
Enhanced isolation validation - Avoids false positives when the complete management network has failed.
Datastore heartbeating – This additional level of heartbeating reduces chances of false positives by using the storage layer to validate the
state of the host and to avoid unnecessary downtime when there’s a management network interruption.
Enhanced Admission Control Policies:
The Host Failures based admission control allows for more than 4 hosts to be specified (31 is the maximum).
The Percentage based admission control policy allows you to specify percentages for both CPU and memory separately.
The Failover Host based admission control policy allows you to specify multiple designated failover hosts.
What is Required for HA to Work?
Each feature or product has very specific requirements and HA is no different. Knowing the requirements of HA is part of the basics we have to
cover before diving into some of the more complex concepts. For those who are completely new to HA, we will also show you how to configure it.
Prerequisites
Before enabling HA it is highly recommend validating that the environment meets all the prerequisites. We have also included recommendations
from an infrastructure perspective that will enhance resiliency.

Requirements:

Minimum of two ESXi hosts

Minimum of 3GB memory per host to install ESXi and enable HA
VMware vCenter Server
Shared Storage for virtual machines
Pingable gateway or other reliable address

Recommendation:

Redundant Management Network (not a requirement, but highly recommended)

Multiple shared datastores
Firewall Requirements
The following table contains the ports that are used by HA for communication. If your environment contains firewalls external to the host, ensure
these ports are opened for HA to function correctly. HA will open the required ports on the ESX or ESXi firewall. Please note that this is the first
substantial difference as HA as of vSphere 5 only uses a single port compared to multiple ports pre-vSphere 5.0 and ESXi 5.0 is enhanced with a
firewall.

Table 1: High Availability port settings

Configuring VMware High Availability
HA can be configured with the default settings within a couple of clicks. The following steps will show you how to create a cluster and enable HA,
including VM Monitoring, as we feel this is the bare minimum configuration.

Each of the settings and the design decisions associated with these steps will be described in more depth in the following chapters.

1. Select the Hosts & Clusters view.

2. Right-click the Datacenter in the Inventory tree and click New Cluster.
3. Give the new cluster an appropriate name. We recommend at a minimum including the location of the cluster and a sequence number ie.
ams-hadrs-001.
4. In the Cluster Features section of the page, select Turn On VMware HA and click Next.
5. Ensure Host Monitoring Status and Admission Control is enabled and click Next.
6. Leave Cluster Default Settings for what they are and click Next.
7. Enable VM Monitoring Status by selecting “VM and Application Monitoring” and click Next.
8. Leave VMware EVC set to the default and click Next.
9. Leave the Swapfile Policy set to default and click Next.
10. Click Finish to complete the creation of the cluster.

Figure 3: Ready to complete the New Cluster Wizard

When the HA cluster has been created, the ESXi hosts can be added to the cluster simply by dragging them into the cluster, if they were already
added to vCenter, or by right clicking the cluster and selecting “Add Host”.

When an ESXi host is added to the newly-created cluster, the HA agent will be loaded and configured. Once this has completed, HA will enable
protection of the workloads running on this ESXi host.

As we have clearly demonstrated, HA is a simple clustering solution that will allow you to protect virtual machines against host failure and operating
system failure in literally minutes. Understanding the architecture of HA will enable you to reach that extra 9 when it comes to availability. The
following chapters will discuss the architecture and fundamental concepts of HA. We will also discuss all decision-making moments to ensure you
will configure HA in such a way that it meets the requirements of your or your customer’s environment.
Chapter 2
Components of High Availability

Now that we know what the pre-requisites are and how to configure HA the next steps will be describing which components form HA. Keep in mind
that this is still a “high level” overview. There is more under the cover that we will explain in following chapters. The following diagram depicts a two-
host cluster and shows the key HA components.

Figure 4: Components of High Availability

As you can clearly see, there are three major components that form the foundation for HA as of vSphere 5.0:

FDM
hostd
vCenter

The first and probably the most important component that forms HA is FDM (Fault Domain Manager). This is the HA agent, and has replaced what
was once known as AAM (Legato’s Automated Availability Manager).

The FDM Agent is responsible for many tasks such as communicating host resource information, virtual machine states and HA properties to other
hosts in the cluster. FDM also handles heartbeat mechanisms, virtual machine placement, virtual machine restarts, logging and much more. We are
not going to discuss all of this in-depth separately as we feel that this will complicate things too much.

FDM, in our opinion, is one of the most important agents on an ESXi host, when HA is enabled, of course, and we are assuming this is the case.
The engineers recognized this importance and added an extra level of resiliency to HA. Contrary to AAM, FDM uses a single-process agent.
However, FDM spawns a watchdog process. In the unlikely event of an agent failure, the watchdog functionality will pick up on this and restart the
agent to ensure HA functionality remains without anyone ever noticing it failed. The agent is also resilient to network interruptions and “all path
down” (APD) conditions. Inter-host communication automatically uses another communication path (if the host is configured with redundant
management networks) in the case of a network failure.

As of vSphere 5.0, HA is no longer dependent on DNS as it works with IP addresses only. This is one of the major improvements that FDM
brought. This also means that the character limit that HA imposed on the FQDN has been lifted. (Pre-vSphere 5.0, FQDNs were limited to 27
characters.) This does not mean that ESXi hosts need to be registered with their IP addresses in vCenter; it is still a best practice to register ESXi
hosts by FQDN in vCenter. Although HA does not depend on DNS anymore, remember that many other services do. On top of that, monitoring and
troubleshooting will be much easier when hosts are correctly registered within vCenter and have a valid FQDN.
 Basic design principle
Although HA is not dependent on DNS anymore, it is still recommended to register the hosts with their FQDN.

Another major change that FDM brings is logging. Some of you might have never realized this and some of you might have discovered it the hard
way: prior to vSphere 5.0, the HA log files were not sent to syslog.

vSphere 5.0 brings a standardized logging mechanism where a single log file has been created for all operational log messages; it is called
fdm.log. This log file is stored under /var/log/ as depicted in Figure 5.

Figure 5: HA log file

Basic design principle

Ensure syslog is correctly configured and log files are offloaded to a safe location to offer the possibility of performing a root cause
analysis in case disaster strikes.
HOSTD Agent
One of the most crucial agents on a host is hostd. This agent is responsible for many of the tasks we take for granted like powering on VMs. FDM
talks directly to hostd and vCenter, so it is not dependent on vpxa, like in previous releases. This is, of course, to avoid any unnecessary overhead
and dependencies, making HA more reliable than ever before and enabling HA to respond faster to power-on requests. That ultimately results in
higher VM uptime.

When, for whatever reason, hostd is unavailable or not yet running after a restart, the host will not participate in any FDM-related processes. FDM
relies on hostd for information about the virtual machines that are registered to the host, and manages the virtual machines using hostd APIs. In
short, FDM is dependent on hostd and if hostd is not operational, FDM halts all functions and waits for hostd to become operational.
vCenter
That brings us to our final component, the vCenter Server. vCenter is the core of every vSphere Cluster and is responsible for many tasks these
days. For our purposes, the following are the most important and the ones we will discuss in more detail:

Deploying and configuring HA Agents

Communication of cluster configuration changes
Protection of virtual machines

vCenter is responsible for pushing out the FDM agent to the ESXi hosts when applicable. Prior to vSphere 5, the push of these agents would be
done in a serial fashion. With vSphere 5.0, this is done in parallel to allow for faster deployment and configuration of multiple hosts in a cluster.
vCenter is also responsible for communicating configuration changes in the cluster to the host which is elected as the master. We will discuss this
concept of master and slaves in the following chapter. Examples of configuration changes are modification or addition of an advanced setting or
the introduction of a new host into the cluster.

As of vSphere 5.0, HA also leverages vCenter to retrieve information about the status of virtual machines and, of course, vCenter is used to display
the protection status (Figure 6) of virtual machines. (What “virtual machine protection” actually means will be discussed in Chapter 3. On top of that,
vCenter is responsible for the protection and unprotection of virtual machines. This not only applies to user initiated power-offs or power-ons of
virtual machines, but also in the case where an ESXi host is disconnected from vCenter at which point vCenter will request the master HA agent to
unprotect the affected virtual machines.

Figure 6: Virtual machine protection state

Although HA is configured by vCenter and exchanges virtual machine state information with HA, vCenter is not involved when HA responds to
failure. It is comforting to know that in case of a host failure containing the virtualized vCenter Server, HA takes care of the failure and restarts the
vCenter Server on another host, including all other configured virtual machines from that failed host.

There is a corner case scenario with regards to vCenter failure: if the ESXi hosts are so called “stateless hosts” and Distributed vSwitches are
used for the management network, virtual machine restarts will not be attempted until vCenter is restarted. For stateless environments, vCenter and
Auto Deploy availability is key as the ESXi hosts literally depend on them.

If vCenter is unavailable, it will not be possible to make changes to the configuration of the cluster. vCenter is the source of truth for the set of virtual
machines that are protected, the cluster configuration, the virtual machine-to-host compatibility information, and the host membership. So, while HA,
by design, will respond to failures without vCenter, HA relies on vCenter to be available to configure or monitor the cluster.
When a virtual vCenter server has been implemented, we recommend setting the correct HA restart priorities for it. Although vCenter Server is not
required to restart, there are multiple components that rely on vCenter and, as such, a speedy recovery is desired. When configuring your vCenter
virtual machine with a high priority for restarts, remember to include all services on which your vCenter server depends for a successful restart:
DNS, MS AD and MS SQL (or any other database server you are using).
Basic design principle
In stateless environments, ensure vCenter and Auto Deploy are highly available as recovery time of your virtual machines might be
dependent on them.

Understand the impact of virtualizing vCenter. Ensure it has high priority for restarts and ensure that services which vCenter Server depends on are
available: DNS, AD and database.
Chapter 3
Fundamental Concepts

Now that you know about the components of HA, it is time to start talking about some of the fundamental concepts of HA clusters:

Master / Slave Nodes

Heartbeating
Isolated vs Network partitioned
Virtual Machine Protection

Everyone who has implemented vSphere knows that multiple hosts can be configured into a cluster. A cluster can best be seen as a collection of
resources. These resources can be carved up with the use of VMware Distributed Resource Scheduler (DRS) into separate pools of resources or
used to increase availability by enabling HA.

With vSphere 5.0, a lot has changed when it comes to HA. For example, an HA cluster used to consist of two types of nodes. A node could either
be a primary or a secondary node. This concept was introduced due to the dependency on AAM and allowed scaling up to 32 hosts per cluster.
FDM has changed this game completely and removed the whole concept of primary and secondary nodes. (For more details about the legacy
(AAM) node mechanism we would like to refer you to the vSphere 4.1 HA and DRS Technical Deepdive.)

One of the most crucial parts of an HA design used to be the design considerations around HA primary nodes and the maximum of 5 per cluster.
These nodes became the core of every HA implementation and literally were responsible for restarting VMs. Without at least one primary node
surviving the failure, a restart of VMs was not possible. This led to some limitations from an architectural perspective and was also one of the
drivers for VMware to rewrite vSphere HA.

The vSphere 5.0 architecture introduces the concept of master and slave HA agents. Except during network partitions, which are discussed later,
there is only one master HA agent in a cluster. Any agent can serve as a master, and all others are considered its slaves. A master agent is in
charge of monitoring the health of virtual machines for which it is responsible and restarting any that fail. The slaves are responsible for forwarding
information to the master agents and restarting any virtual machines at the direction of the master. Another thing that has changed is that the HA
agent, regardless of its role as master or slave, implements the VM/App monitoring feature; with AAM, this feature was part of VPXA.
Master Agent
As stated, one of the primary tasks of the master is to keep track of the state of the virtual machines it is responsible for and to take action when
appropriate. It is important to realize that a virtual machine can only be the responsibility of a single master. We will discuss the scenario where
multiple masters can exist in a single cluster in one of the following sections, but for now let’s talk about a cluster with a single master. A master will
claim responsibility for a virtual machine by taking “ownership” of the datastore on which the virtual machine’s configuration file is stored.

Basic design principle

To maximize the chance of restarting virtual machines after a failure we recommend masking datastores on a cluster basis. Although
sharing of datastores across clusters will work, it will increase complexity from an administrative perspective.

That is not all, of course. The HA master is also responsible for exchanging state information with vCenter. This means that it will not only receive
but also send information to vCenter when required. The HA master is also the host that initiates the restart of virtual machines when a host has
failed. You may immediately want to ask what happens when the master is the one that fails, or, more generically, which of the hosts can become
the master and when is it elected?

Election
A master is elected by a set of HA agents whenever the agents are not in network contact with a master. A master election thus occurs when HA is
first enabled on a cluster and when the host on which the master is running:

fails,
becomes network partitioned or isolated,
is disconnected from VC,
is put into maintenance or standby mode,
or when HA is reconfigured on the host.

The HA master election takes approximately 15 seconds and is conducted using UDP. While HA won’t react to failures during the election, once a
master is elected, failures detected before and during the election will be handled. The election process is simple but robust. The host that is
participating in the election with the greatest number of connected datastores will be elected master. If two or more hosts have the same number of
datastores connected, the one with the highest Managed Object Id will be chosen. This however is done lexically; meaning that 99 beats 100 as 9
is larger than 1. For each host, the HA State of the host will be shown on the Summary tab. This includes the role as depicted in Figure 7 where the
host is a master host.

After a master is elected, each slave that has management network connectivity with it will setup a single secure, encrypted, TCP connection to the
master. This secure connection is SSL-based. One thing to stress here though is that slaves do not communicate with each other after the master
has been elected unless a re-election of the master needs to take place.

Figure 7: Master Agent

As stated earlier, when a master is elected it will try to acquire ownership of all of the datastores it can directly access or access by proxying
requests to one of the slaves connected to it using the management network. It does this by locking a file called “protectedlist” that is stored on the
datastores in an existing cluster. The master will also attempt to take ownership of any datastores it discovers along the way, and it will periodically
retry any it could not take ownership of previously.

The naming format and location of this file is as follows:

/<root of datastore>/.vSphere-HA/<cluster-specific-directory>/protectedlist

For those wondering how “<cluster-specific-directory>” is constructed:

<uuid of VC>-<number part of the MoID of the cluster>-<random 8 char string>-<name of the host running VC>

The master uses this protectedlist file to store the inventory. It keeps track of which virtual machines are protected by HA. Calling it an inventory
might be slightly overstating: it is a list of protected virtual machines. The master distributes this inventory across all datastores in use by the virtual
machines in the cluster. Figure 8 shows an example of this file on one of the datastores.

Figure 8: Protectedlist file

Now that we know the master locks a file on the datastore and that this file stores inventory details, what happens when the master is isolated or
fails? If the master fails, the answer is simple: the lock will expire and the new master will relock the file if the datastore is accessible to it.

In the case of isolation, this scenario is slightly different, although the result is similar. The master will release the lock it has on the file on the
datastore to ensure that when a new master is elected it can claim the responsibility for the virtual machines on these datastores by locking the file
on the datastore. If, by any chance, a master should fail right at the moment that it became isolated, the restart of the virtual machines will be
delayed until a new master has been elected. In a scenario like this, accuracy and the fact that virtual machines are restarted is more important than
a short delay.

Let’s assume for a second that your master has just failed. What will happen and how do the slaves know that the master has failed? vSphere 5.0
uses a network heartbeat mechanism. If the slaves have received no network heartbeats from the master, the slaves will try to elect a new master.
This new master will read the required information and will restart the virtual machines within 10 seconds. There is more to this process but we will
discuss that in Chapter 4.

Restarting virtual machines is not the only responsibility of the master. It is also responsible for monitoring the state of the slave hosts. If a slave fails
or becomes isolated, the master will determine which virtual machines must be restarted. When virtual machines need to be restarted, the master
is also responsible for determining the placement of those virtual machines. It uses a placement engine that will try to distribute the virtual machines
to be restarted evenly across all available hosts.

All of these responsibilities are really important, but without a mechanism to detect a slave has failed, the master would be useless. Just like the
slaves receive heartbeats from the master, the master receives heartbeats from the slaves so it knows they are alive.
Slaves
A slave has substantially fewer responsibilities than a master: a slave monitors the state of the virtual machines it is running and informs the master
about any changes to this state.

The slave also monitors the health of the master by monitoring heartbeats. If the master becomes unavailable, the slaves initiate and participate in
the election process. Last but not least, the slaves send heartbeats to the master so that the master can detect outages.

Figure 9: Slave Agent

Files for both Slave and Master
Both the master and slave use files not only to store state, but also as a communication mechanism. We’ve already seen the protectedlist file
(Figure 8) used by the master to store the list of protected virtual machines. We will now discuss the files that are created by both the master and
the slaves. Remote files are files stored on a shared datastore and local files are files that are stored in a location only directly accessible to that
host.

Remote Files
The set of powered on virtual machines is stored in a per-host “poweron” file. (See Figure 8 for an example of these files.) It should be noted that,
because a master also hosts virtual machines, it also creates a “poweron” file.

The naming scheme for this file is as follows:

host-<number>-poweron

Tracking virtual machine power-on state is not the only thing the “poweron” file is used for. This file is also used by the slaves to inform the master
that it is isolated: the top line of the file will either contain a 0 or a 1. A 0 means not-isolated and a 1 means isolated. The master will inform vCenter
about the isolation of the host.

Local Files
As mentioned before, when HA is configured on a host, the host will store specific information about its cluster locally.

Figure 10: Locally stored files

Each host, including the master, will store data locally. The data that is locally stored is important state information. Namely, the VM-to-host
compatibility matrix, cluster configuration, and host membership list. This information is persisted locally on each host. Updates to this information
is sent to the master by vCenter and propagated by the master to the slaves. Although we expect that most of you will never touch these files – and
we highly recommend against modifying them – we do want to explain how they are used:

clusterconfig
This file is not human-readable. It contains the configuration details of the cluster.
compatlist
This file is not human-readable. It contains the actual compatibility info matrix for every HA protected virtual machine and lists all the hosts with
which it is compatible.
fdm.cfg
This file contains the configuration settings around logging. For instance, the level of logging and syslog details are stored in here.
hostlist
A list of hosts participating in the cluster, including hostname, IP addresses, MAC addresses and heartbeat datastores.
Heartbeating
We mentioned it a couple of times already in this chapter, and it is an important mechanism that deserves its own section: heartbeating.
Heartbeating is the mechanism used by HA to validate whether a host is alive. With the introduction of vSphere 5.0, not only did the heartbeating
mechanism slightly change, an additional heartbeating mechanism was introduced. Let’s discuss traditional network heartbeating first.

Network Heartbeating
vSphere 5.0 introduces some changes to the well-known heartbeat mechanism. As vSphere 5.0 doesn’t use the concept of primary and secondary
nodes, there is no reason for 100s of heartbeat combinations. As of vSphere 5.0, each slave will send a heartbeat to its master and the master
sends a heartbeat to each of the slaves. These heartbeats are sent by default every second.

When a slave isn’t receiving any heartbeats from the master, it will try to determine whether it is Isolated– we will discuss “states” in more detail
later on in this chapter.

Basic design principle

Network heartbeating is key for determining the state of a host. Ensure the management network is highly resilient to enable proper
state determination.

Datastore Heartbeating
Those familiar with HA prior to vSphere 5.0 hopefully know that virtual machine restarts were always attempted, even if only the heartbeat network
was isolated and the virtual machines were still running on the host. As you can imagine, this added an unnecessary level of stress to the host. This
has been mitigated by the introduction of the datastore heartbeating mechanism. Datastore heartbeating adds a new level of resiliency and
prevents unnecessary restart attempts from occurring.

Datastore heartbeating enables a master to more correctly determine the state of a host that is not reachable via the management network. The
new datastore heartbeat mechanism is only used in case the master has lost network connectivity with the slaves. The datastore heartbeat
mechanism is then used to validate whether a host has failed or is merely isolated/network partitioned. Isolation will be validated through the
“poweron” file which, as mentioned earlier, will be updated by the host when it is isolated. Without the “poweron” file, there is no way for the master
to validate isolation. Let that be clear! Based on the results of checks of both files, the master will determine the appropriate action to take. If the
master determines that a host has failed (no datastore heartbeats), the master will restart the failed host’s virtual machines. If the master
determines that the slave is Isolated or Partitioned, it will only take action when it is appropriate to take action. With that meaning that the master
will only initiate restarts when virtual machines are down or powered down / shut down by a triggered isolation response, but we will discuss this in
more detail in Chapter 4.

By default, HA picks 2 heartbeat datastores – it will select datastores that are available on all hosts, or as many as possible. Although it is possible
to configure an advanced setting (das.heartbeatDsPerHost) to allow for more datastores for datastore heartbeating we do not recommend
configuring this option as the default should be sufficient for every scenario.
The selection process gives preference to VMFS datastores over NFS ones, and seeks to choose datastores that are backed by different storage
arrays or NFS servers. If desired, you can also select the heartbeat datastores yourself. We, however, recommend letting vCenter deal with this
operational “burden” as vCenter uses a selection algorithm to select heartbeat datastores that are presented to all hosts. This however is not a
guarantee that vCenter can select datastores which are connected to all hosts.

Figure 11: Selecting the heartbeat datastores

The question now arises: what, exactly, is this datastore heartbeating and which datastore is used for this heartbeating? Let’s answer which
datastore is used for datastore heartbeating first as we can simply show that with a screenshot, Figure 12. vSphere 5.0 has brought some new
capabilities to the “Cluster Status” feature on the Cluster’s Summary tab. This now shows you which datastores are being used for heartbeating
and which hosts are using which specific datastore(s). In addition, it displays how many virtual machines are protected and how many hosts are
connected to the master.

Figure 12: Validating the heartbeat datastores

How does this heartbeating mechanism work? HA leverages an existing VMFS file system mechanism. The mechanism uses a so called
“heartbeat region” which is updated as long as the file exists. On VMFS datastores, HA will simply check whether the heartbeat region has been
updated. In order to update a datastore heartbeat region, a host needs to have at least one open file on the volume. HA ensures there is at least
one file open on this volume by creating a file specifically for datastore heartbeating. In other words, a per-host file is created on the designated
heartbeating datastores, as shown in Figure 13. The naming scheme for this file is as follows:
host-<number>-hb

Figure 13: Heartbeat file

On NFS datastores, each host will write to its heartbeat file once every 5 seconds, ensuring that the master will be able to check host state. The
master will simply validate this by checking the time-stamp of the file.

Realize that in the case of a converged network environment, the effectiveness of datastore heartbeating will vary depending on the type of failure.
For instance, a NIC failure could impact both network and datastore heartbeating. If, for whatever reason, the datastore or NFS share becomes
unavailable or is removed from the cluster, HA will detect this and select a new datastore or NFS share to use for the heartbeating mechanism.

Basic design principle

Datastore heartbeating adds a new level of resiliency but is not the be-all end-all. In converged networking environments, the use of
datastore heartbeating adds little value due to the fact that a NIC failure may result in both the network and storage becoming unavailable.
Isolated versus Partitioned
We’ve already briefly touched on it and it is time to have a closer look. As of vSphere 5.0 HA, a new cluster node state called Partitioned exists.
What is this exactly and when is a host Partitioned rather than Isolated? Before we will explain this we want to point out that there is the state as
reported by the master and the state as observed by an administrator and the characteristics these have.

First of all, a host is considered to be either Isolated or Partitioned when it loses network access to a master but has not failed. To help explain the
difference, we‘ve listed both states and the associated criteria below:

Isolated
Is not receiving heartbeats from the master
Is not receiving any election traffic
Cannot ping the isolation address
Partitioned
Is not receiving heartbeats from the master
Is receiving election traffic
(at some point a new master will be elected at which the state will be reported to vCenter)

In the case of an Isolation, a host is separated from the master and the virtual machines running on it might be restarted, depending on the selected
isolation response and the availability of a master. It could occur that multiple hosts are fully isolated at the same time.

When multiple hosts are isolated but can still communicate amongst each other over the management networks, we call this a network partition.
When a network partition exists, a master election process will be issued so that a host failure or network isolation within this partition will result in
appropriate action on the impacted virtual machine(s). Figure 14 shows possible ways in which an Isolation or a Partition can occur.

Figure 14: Isolated versus Partitioned

If a cluster is partitioned in multiple segments, each partition will elect its own master, meaning that if you have 4 partitions your cluster will have 4
masters. When the network partition is corrected, any of the four masters will take over the role and be responsible for the cluster again. It should be
noted that a master could claim responsibility for a virtual machine that lives in a different partition. If this occurs and the virtual machine happens to
fail, the master will be notified through the datastore communication mechanism.
This still leaves the question open how the master determines if a host has Failed, is Partitioned or has become Isolated. This is where the new
datastore heartbeat mechanism comes in to play.

When the master stops receiving network heartbeats from a slave, it will check for host “liveness” for the next 15 seconds. Before the host is
declared failed, the master will validate if it has actually failed or not by doing additional liveness checks. First, the master will validate if the host is
still heartbeating to the datastore. Second, the master will ping the management IP-address of the host. If both are negative, the host will be
declared Failed. This doesn’t necessarily mean the host has PSOD’ed; it could be the network is unavailable, including the storage network, which
would make this host Isolated from an administrator’s perspective but Failed from an HA perspective. As you can imagine, however, there are a
various combinations possible. The following table depicts these combinations including the “state”.

Table 2: Host states

HA will trigger an action based on the state of the host. When the host is marked as Failed, a restart of the virtual machines will be initiated. When
the host is marked as Isolated, the master might initiate the restarts. As mentioned earlier, this is a substantial change compared to HA prior to
vSphere 5.0 when restarts were always initiated, regardless of the state of the virtual machines or hosts. The one thing to keep in mind when it
comes to isolation response is that a virtual machine will only be shut down or powered off when the isolated host knows there is a master out there
that has taken ownership for the virtual machine or when the isolated host loses access to the home datastore of the virtual machine.

For example, if a host is isolated and runs two virtual machines, stored on separate datastores, the host will validate if it can access each of the
home datastores of those virtual machines. If it can, the host will validate whether a master owns these datastores. If no master owns the
datastores, the isolation response will not be triggered and restarts will not be initiated. If the host does not have access to the datastore, for
instance, during an “All Paths Down” condition, HA will trigger the isolation response to ensure the “original” virtual machine is powered down and
will be safely restarted. This to avoid so-called “split-brain” scenarios.

To reiterate, as this is a major change compared to all previous versions of HA, the remaining hosts in the cluster will only be requested to restart
virtual machines when the master has detected that either the host has failed or has become isolated and the isolation response was triggered. If
the term isolation response is not clear yet, don’t worry as we will discuss it in more depth in Chapter 4.
Virtual Machine Protection
The way virtual machines are protected has changed substantially in vSphere 5.0. Prior to vSphere 5.0, virtual machine protection was handled by
vpxd which notified AAM through a vpxa module called vmap. With vSphere 5.0, virtual machine protection happens on several layers but is
ultimately the responsibility of vCenter. We have explained this briefly but want to expand on it a bit more to make sure everyone understands the
dependency on vCenter when it comes to protecting virtual machines. We do want to stress that this only applies to protecting virtual machines;
virtual machine restarts in no way require vCenter to be available at the time.

When the state of a virtual machine changes, vCenter will direct the master to enable or disable HA protection for that virtual machine. Protection,
however, is only guaranteed when the master has committed the change of state to disk. The reason for this, of course, is that a failure of the
master would result in the loss of any state changes that exist only in memory. As pointed out earlier, this state is distributed across the datastores
and stored in the “protectedlist” file.

When the power state change of a virtual machine has been committed to disk, the master will inform vCenter Server so that the change in status is
visible both for the user in vCenter and for other processes like monitoring tools.

To clarify the process, we have created a workflow diagram (Figure 15) of the protection of a virtual machine from the point it is powered on through
vCenter:

Figure 15: VM protection workflow

But what about “unprotection?” When a virtual machine is powered off, it must be removed from the protectedlist. We have documented this
workflow in Figure 16.

Figure 16: Unprotection workflow

Chapter 4
Restarting Virtual Machines

In the previous chapter, we have described most of the lower level fundamental concepts of HA. We have shown you that multiple new mechanisms
have been introduced to increase resiliency and reliability of HA. Reliability of HA in this case mostly refers to restarting virtual machines, as that
remains HA’s primary task.

HA will respond when the state of a host has changed, or, better said, when the state of one or more virtual machines has changed. There are
multiple scenarios in which HA will attempt to restart a virtual machine of which we have listed the most common below:

Failed host
Isolated host
Failed guest Operating System

Depending on the type of failure, but also depending on the role of the host, the process will differ slightly. Changing the process results in slightly
different recovery timelines. There are many different scenarios and there is no point in covering all of them, so we will try to describe the most
common scenario and include timelines where possible.

Before we dive into the different failure scenarios, we want to emphasize a couple of very substantial changes compared to vSphere pre-5.0 with
regards to restart priority and retries. These apply to every situation we will describe.
Restart Priority and Order
Prior to vSphere 5.0, HA would take the priority of the virtual machine into account when a restart of multiple virtual machines was required. This, by
itself, has not changed; HA will still take the configured priority of the virtual machine in to account. However, with vSphere 5.0, new types of virtual
machines have been introduced: Agent Virtual Machines. These virtual machines typically offer a “service” to other virtual machines and, as such,
take precedence during the restart procedure as the “regular” virtual machines may rely on them. A good example of an agent virtual machine is a
vShield Endpoint virtual machine which offers anti-virus services. These agent virtual machines are considered top priority virtual machines.

Prioritization is done by each host and not globally. Each host that has been requested to initiate restart attempts will attempt to restart all top
priority virtual machines before attempting to start any other virtual machines. If the restart of a top priority virtual machine fails, it will be retried after
a delay. In the meantime, however, HA will continue powering on the remaining virtual machines. Keep in mind that some virtual machines might be
dependent on the agent virtual machines. You should document which virtual machines are dependent on which agent virtual machines and
document the process to start up these services in the right order in the case the automatic restart of an agent virtual machine fails.

Basic design principle

Virtual machines can be dependent on the availability of agent virtual machines or other virtual machines. Although HA will do its best to
ensure all virtual machines are started in the correct order, this is not guaranteed. Document the proper recovery process.

Besides agent virtual machines, HA also prioritizes FT secondary machines. We have listed the full order in which virtual machines will be restarted
below:

Agent virtual machines

FT secondary virtual machines
Virtual Machines configured with a restart priority of high,
Virtual Machines configured with a medium restart priority
Virtual Machines configured with a low restart priority

Now that we have briefly touched on it, we would also like to address “restart retries” and parallelization of restarts as that more or less dictates how
long it could take before all virtual machines of a failed or isolated host are restarted.
Restart Retries
The number of retries is configurable as of vCenter 2.5 U4 with the advanced option “das.maxvmrestartcount”. The default value is 5. Prior to
vCenter 2.5 U4, HA would keep retrying forever which could lead to serious problems. This scenario is described in KB article 1009625 where
multiple virtual machines would be registered on multiple hosts simultaneously, leading to a confusing and an inconsistent state.
(http://kb.vmware.com/kb/1009625)

Note:
Prior to vSphere 5.0 “das.maxvmrestartcount” did not include the initial restart. Meaning that the total amount of restarts was 6. As of vSphere
5.0 the initial restart is included in the value.

HA will try to start the virtual machine on one of your hosts in the affected cluster; if this is unsuccessful on that host, the restart count will be
increased by 1. Before we go into the exact timeline, let it be clear that T0 is the point at which the master initiates the first restart attempt. This by
itself could be 30 seconds after the virtual machine has failed. The elapsed time between the failure of the virtual machine and the restart, though,
will depend on the scenario of the failure, which we will discuss in this chapter.

As said, prior to vSphere 5, the actual number of restart attempts was 6, as it excluded the initial attempt. With vSphere 5.0 the default is 5. There
are specific times associated with each of these attempts. The following bullet list will clarify this concept. The ‘m’ stands for “minutes” in this list.

T0 – Initial Restart
T2m – Restart retry 1
T6m – Restart retry 2
T14m – Restart retry 3
T30m – Restart retry 4

Figure 17: High Availability restart timeline

As shown above and clearly depicted in Figure 17, a successful power-on attempt could take up to ~30 minutes in the case where multiple power-
on attempts are unsuccessful. This is, however, not exact science. For instance, there is a 2-minute waiting period between the initial restart and
the first restart retry. HA will start the 2-minute wait as soon as it has detected that the initial attempt has failed. So, in reality, T2 could be T2 and 8
seconds. Another important fact that we want emphasize is that if a different master claims responsibility for the virtual machine during this
sequence, the sequence will restart.

Let’s give an example to clarify the scenario in which a master fails during a restart sequence:

Cluster: 4 Host (esxi01, esxi02, esxi03, esxi04)

Master: esxi01

The host “esxi02” is running a single virtual machine called “vm01” and it fails. The master, esxi01, will try to restart it but the attempt
fails. It will try restarting “vm01” up to 5 times but, unfortunately, on the 4 th try, the master also fails. An election occurs and “esxi03” becomes
the new master. It will now initiate the restart of “vm01”, and if that restart would fail it will retry it up to 4 times again for a total including the
initial restart of 5.

Be aware, though, that a successful restart might never occur if the restart count is reached and all five restart attempts (the default value) were
unsuccessful.

When it comes to restarts, one thing that is very important to realize is that HA will not issue more than 32 concurrent power-on tasks on a given
host. To make that more clear, let’s use the example of a two host cluster: if a host fails which contained 33 virtual machines and all of these had the
same restart priority, 32 power on attempts would be initiated. The 33 rd power on attempt will only be initiated when one of those 32 attempts has
completed regardless of success or failure of one of those attempts.

Now, here comes the gotcha. If there are 32 low-priority virtual machines to be powered on and a single high-priority virtual machine, the power on
attempt for the low-priority virtual machines will not be issued until the power on attempt for the high priority virtual machine has completed. Let it be
absolutely clear that HA does not wait to restart the low-priority virtual machines until the high-priority virtual machines are started, it waits for the
issued power on attempt to be reported as “completed”. In theory, this means that if the power on attempt fails, the low-priority virtual machines
could be powered on before the high priority virtual machine.

Basic design principle

Configuring restart priority of a virtual machine is not a guarantee that virtual machines will actually be restarted in this order. Ensure
proper operational procedures are in place for restarting services or virtual machines in the appropriate order in the event of a failure.

Now that we know how virtual machine restart priority and restart retries are handled, it is time to look at the different scenarios.

Failed host
Failure of a master
Failure of a slave
Isolated host and response
Failed Host
Prior to vSphere 5.0, the restart of virtual machines from a failed host was straightforward. With the introduction of master/slave hosts and heartbeat
datastores in vSphere 5.0, the restart procedure has also changed, and with it the associated timelines. There is a clear distinction between the
failure of a master versus the failure of a slave. We want to emphasize this because the time it takes before a restart attempt is initiated differs
between these two scenarios. Let’s start with the most common failure, that of a host failing, but note that failures generally occur infrequently. In
most environments, hardware failures are very uncommon to begin with. Just in case it happens, it doesn’t hurt to understand the process and its
associated timelines.

The Failure of a Slave

This is a fairly complex scenario compared to how HA handled host failures prior to vSphere 5.0. Part of this complexity comes from the
introduction of a new heartbeat mechanism. Actually, there are two different scenarios: one where heartbeat datastores are configured and one
where heartbeat datastores are not configured. Keeping in mind that this is an actual failure of the host, the timeline is as follows:

T0 – Slave failure
T3s – Master begins monitoring datastore heartbeats for 15 seconds
T10s – The host is declared unreachable and the master will ping the management network of the failed host. This is a continuous ping for 5
seconds
T15s – If no heartbeat datastores are configured, the host will be declared dead
T18s – If heartbeat datastores are configured, the host will be declared dead

The master monitors the network heartbeats of a slave. When the slave fails, these heartbeats will no longer be received by the master. We have
defined this as T0. After 3 seconds (T3s), the master will start monitoring for datastore heartbeats and it will do this for 15 seconds. On the 10 th
second (T10s), when no network or datastore heartbeats have been detected, the host will be declared as “unreachable”. The master will also start
pinging the management network of the failed host at the 10th second and it will do so for 5 seconds. If no heartbeat datastores were configured,
the host will be declared “dead” at the 15th second (T15s) and VM restarts will be initiated by the master. If heartbeat datastores have been
configured, the host will be declared dead at the 18th second (T18s) and restarts will be initiated. We realize that this can be confusing and hope
the timeline depicted in Figure 18 makes it easier to digest.

Figure 18: Restart timeline slave failure

That leaves us with the question of what happens in the case of the failure of a master.

The Failure of a Master

In the case of a master failure, the process and the associated timeline are slightly different. The reason being that there needs to be a master
before any restart can be initiated. This means that an election will need to take place amongst the slaves. The timeline is as follows:

T0 – Master failure.
T10s – Master election process initiated.
T25s – New master elected and reads the protectedlist.
T35s – New master initiates restarts for all virtual machines on the protectedlist which are not running.

Slaves receive network heartbeats from their master. If the master fails, lets define this as T0, the slaves detect this when the network heartbeats
cease to be received. As every cluster needs a master, the slaves will initiate an election at T10s. The election process takes 15s to complete,
which brings us to T25s. At T25s, the new master reads the protectedlist. This list contains all the virtual machines which are protected by HA. At
T35s, the master initiates the restart of all virtual machines that are protected but not currently running. The timeline depicted in Figure 19 hopefully
clarifies the process.
Figure 19: Restart timeline master failure
Besides the failure of a host, there is another reason for restarting virtual machines: an isolation event.
Isolation Response and Detection
Before we will discuss the timeline and the process around the restart of virtual machines after an isolation event, we will discuss Isolation
Response and Isolation Detection. One of the first decisions that will need to be made when configuring HA is the “Isolation Response”.

Isolation Response
The Isolation Response refers to the action that HA takes for its virtual machines when the host has lost its connection with the network and the
remaining nodes in the cluster. This does not necessarily mean that the whole network is down; it could just be the management network ports of
this specific host. Today there are three isolation responses: “Power off”, “Leave powered on” and “Shut down”. This isolation response answers
the question, “what should a host do with the virtual machines it manages when it detects that it is isolated from the network?” Let’s discuss these
three options more in-depth:

Power off – When isolation occurs, all virtual machines are powered off. It is a hard stop, or to put it bluntly, the “virtual” power cable of the
virtual machine will be pulled out
Shut down – When isolation occurs, all virtual machines running on the host will be shut down using a guest-initiated shutdown through
VMware Tools. If this is not successful within 5 minutes, a “power off” will be executed. This time out value can be adjusted by setting the
advanced option das.isolationShutdownTimeout. If VMware Tools is not installed, a “power off” will be initiated immediately
Leave powered on – When isolation occurs on the host, the state of the virtual machines remains unchanged

This setting can be changed on the cluster settings under virtual machine options (Figure 20).

Figure 20: Cluster default settings

The default setting for the isolation response has changed multiple times over the last couple of years and this has caused some confusion.

Up to ESXi3.5 U2 / vCenter 2.5 U2 the default isolation response was “Power off”
With ESXi3.5 U3 / vCenter 2.5 U3 this was changed to “Leave powered on”
With vSphere 4.0 it was changed to “Shut down”
With vSphere 5.0 it has been changed to “Leave powered on”

Keep in mind that these changes are only applicable to newly created clusters. When creating a new cluster, it may be required to change the
default isolation response based on the configuration of existing clusters and/or your customer’s requirements, constraints and expectations. When
upgrading an existing cluster, it might be wise to apply the latest default values. You might wonder why the default has changed once again. There
was a lot of feedback from customers that “Leave powered on” was the desired default value.

Basic design principle

Before upgrading an environment to later versions, ensure you validate the best practices and default settings. Document them,
including justification, to ensure all people involved understand your reasons.

The question remains, which setting should be used? The obvious answer applies here; it depends. We prefer “Leave powered on” because it
Other documents randomly have
different content
 "Still," said the elder man, "don't throw me over altogether. If I
can help you in any way, little or big, let me know. If I can give you
any hints, any advice, anything at all, I want to do it. And if you
happen upon what seems to be a promising clue, come and talk it
over with me. Oh, don't be afraid! I'll leave it to you to work out. I
shan't spoil your game."
"Ah, now that's very good of you," said Ste. Marie. "Only you
make me seem more than ever an ungrateful fool. Thanks, I will
come to you with my troubles if I may. I have a foolish idea that I
want to follow out a little first, but doubtless I shall be running to
you soon for information."
The elder man's eyes sharpened again with keen interest.
"An idea!" he said quickly. "You have an idea? What—may I ask
what sort of an idea?"
"Oh it's nothing," declared Ste. Marie. "You have already
laughed at it. I just want to find that man O'Hara, that's all. I've a
feeling that I should learn something from him."
"Ah!" said Captain Stewart slowly. "Yes, the man O'Hara. There's
nothing in that, I'm afraid. I've made inquiries about O'Hara. It
seems he left Paris six months ago, saying he was off for America.
An old friend of his told me that. So you must have been mistaken
when you thought you saw him in the Champs Elysées, and he
couldn't very well have had anything to do with poor Arthur. I'm
afraid that idea is hardly worth following up."
"Perhaps not," said Ste. Marie. "I seem to start badly, don't I?
Ah well, I'll have to come to you all the sooner, then."
"You'll be welcome," promised Captain Stewart. "Good-bye to
you! Good day, Hartley. Come and see me both of you. You know
where I live."
He took his leave then, and Hartley, standing beside the
window, watched him turn down the street, and at the corner get
into one of the fiacres there and drive away.
Ste. Marie laughed aloud.
"There's the second time," said he, "that I've had him about
O'Hara. If he is as careless as that about everything, I don't wonder
he hasn't found Arthur Benham. O'Hara disappeared from Paris
(publicly, that is) at about the time young Benham disappeared. As a
matter of fact he remains, or at least for a time remained in the city
without letting his friends know, because I made no mistake about
seeing him in the Champs Elysées. All that looks to me suspicious
enough to be worth investigation.
"Of course," he admitted doubtfully—"of course I'm no
detective, but that's how it looks to me."
"I don't believe Stewart is any detective either," said Richard
Hartley. "He's altogether too cock-sure. That sort of man would
rather die than admit he is wrong about anything. He's a good old
chap though, isn't he? I liked him to-day better than ever before. I
thought he was rather pathetic when he went on about his age."
"He has a good heart," said Ste. Marie. "Very few men under
the circumstances would come here and be as decent as he was.
Most men would have thought I was a presumptuous ass and would
have behaved accordingly."
Ste. Marie took a turn about the room and his face began to
light up with its new excitement and exaltation.
"And to-morrow," he cried, "to-morrow we begin! To-morrow we
set out into the world and the Adventure is on foot. God send it
success!" He laughed across at the other man, but it was a laugh of
eagerness not of mirth.
"I feel," said he, "like Jason. I feel as if we were to set sail to-
morrow for Colchis and the Golden Fleece."
"Ye—es," said the other man a little drily. "Yes, perhaps. I don't
want to seem critical, but isn't your figure somewhat ill chosen?"
"'Ill chosen'?" cried Ste. Marie. "What d'you mean? Why ill
chosen?"
"I was thinking of Medea," said Richard Hartley.

CHAPTER VIII
STE. MARIE MEETS WITH A MISADVENTURE AND DREAMS A
DREAM

So on the next day these two rode forth upon their quest, and no
quest was ever undertaken with a stouter courage or with a grimmer
determination to succeed. To put it fancifully they burnt their tower
behind them, for to one of them at least—to him who led—there was
no going back.
But after all they set forth under a cloud, and Ste. Marie took a
heavy heart with him. On the evening before an odd and painful
incident had befallen, a singularly unfortunate incident.
It chanced that neither of the two men had a dinner
engagement that evening, and so, after their old habit, they dined
together. There was some wrangling over where they should go,
Hartley insisting upon Armenonville or the Madrid in the Bois, Ste.
Marie objecting that these would be full of tourists so late in June,
and urging the claims of some quiet place in the Quarter, where they
could talk instead of listening perforce to loud music. In the end, for
no particular reason, they compromised on the little Spanish
restaurant in the Rue Helder. They went there about eight o'clock,
without dressing; for it is a very quiet place which the world does
not visit, and they had a sopa de yerbas, and some langostinos,
which are shrimps, and a heavenly arroz with fowl in it, and many
tender succulent strips of red pepper. They had a salad made out of
a little of everything that grows green, with the true Spanish oil,
which has a tang and a bouquet unappreciated by the philistine; and
then they had a strange pastry and some cheese and green
almonds. And to make them glad they drank a bottle of old red
Valdepeñas, and afterwards a glass each of a special Manzanilla,
upon which the restaurant very justly prides itself. It was a simple
dinner and a little stodgy for that time of the year, but the two men
were hungry, and sat at table, almost alone in the upper room, for a
long time, saying how good everything was, and from time to time
despatching the saturnine waiter, a Madrileno, for more peppers.
When at last they came out into the narrow street and thence to the
thronged Boulevard des Italiens, it was nearly eleven o'clock. They
stood for a little time in the shelter of a kiosk, looking down the
boulevard to where the Place de l'Opéra opened wide, and the lights
of the Café de la Paix shone garish in the night, and Ste. Marie said
—
"There's a street fête in Montmartre. We might drive home that
way."
 "An excellent idea," said the other man. "The fact that
Montmartre lies in an opposite direction from home makes the plan
all the better. And after that we might drive home through the Bois.
That's much farther in the wrong direction. Lead on!"
So they sprang into a waiting fiacre, and were dragged up the
steep stone-paved hill to the heights where La Bohême still reigns,
though the glory of Moulin Rouge has departed, and the trail of
tourist is over all. They found Montmartre very much en fête. In the
Place Blanche were two of the enormous and brilliantly lighted
merry-go-rounds which only Paris knows—one furnished with stolid
cattle, theatrical-looking horses, and Russian sleighs, the other with
the ever-popular galloping pigs. When these dreadful machines were
in rotation mechanical organs concealed somewhere in their bowels
emitted hideous brays and shrieks, which mingled with the shrieks of
the ladies mounted upon the galloping pigs, and together insulted a
peaceful sky.
The square was filled with that extremely heterogeneous throng
which the Parisian street fête gathers together, but it was, for the
most part, a well-dressed throng, largely recruited from the
boulevards, and it was quite determined to have a very good time in
the cheerful harmless Latin fashion. The two men got down from
their fiacre and elbowed a way through the good-natured crowd to a
place near the more popular of the merry-go-rounds. The machine
was in rotation. Its garish lights shone and glittered, its hidden
mechanical organ blared a German waltz tune, the huge pink-
varnished pigs galloped gravely up and down as the platform upon
which they were mounted whirled round and round. A little group of
American trippers, sight-seeing, with a guide, stood near by, and one
of the group, a pretty girl with red hair, demanded plaintively of the
friend upon whose arm she hung: "Do you think mamma would be
shocked if we took a ride? Wouldn't I love to!"
Hartley turned laughing from this distressed maiden to Ste.
Marie. He was wondering with mild amusement why anybody should
wish to do such a foolish thing, but Ste. Marie's eyes were fixed
upon the galloping pigs and the eyes shone with a wistful
excitement. To tell the truth it was impossible for him to look on at
any form of active amusement without thirsting to join it. A joyous
and care-free lady in a blue hat, who was mounted astride upon one
of the pigs, hurled a paper serpentine at him, and shrieked with
delight when it knocked his hat off.
"That's the second time she has hit me with one of those
things," he said, groping about his feet for the hat. "Here, stop that
boy with the basket!" A vendor of the little rolls of paper ribbon was
shouting his wares through the crowd. Ste. Marie filled his pockets
with the things, and when the lady with the blue hat came round on
the next turn, lassoed her neatly about the neck and held the end of
the ribbon till it broke. Then he caught a fat gentleman, who was
holding himself on by his steed's neck, in the ear, and the red-haired
American girl laughed aloud.
"When the thing stops," said Ste. Marie, "I'm going to take a
ride, just one ride. I haven't ridden a pig for many years." Hartley
jeered at him, calling him an infant, but Ste. Marie bought more
serpentines, and when the platform came to a stop clambered up to
it, and mounted the only unoccupied pig he could find. His friend still
scoffed at him and called him names, but Ste. Marie tucked his long
legs round the pig's neck and smiled back, and presently the
machine began again to revolve.
At the end of the first revolution Hartley gave a shout of delight,
for he saw that the lady with the blue hat had left her mount and
was making her way along the platform towards where Ste. Marie
sat hurling serpentines in the face of the world. By the next time
round she had come to where he was, mounted astride behind him,
and was holding herself with one very shapely arm round his neck,
while with the other she rifled his pockets for ammunition. Ste. Marie
grinned, and the public, loud in its acclaims, began to pelt the two
with serpentines until they were hung with many-coloured ribbons
like a Christmas-tree. Even Richard Hartley was so far moved out of
the self-consciousness with which his race is cursed as to buy a
handful of the common missiles, and the lady in the blue hat
returned his attention with skill and despatch.
But as the machine began to slacken its pace, and the hideous
wail and blare of the concealed organ died mercifully down, Hartley
saw that his friend's manner had all at once altered, that he sat
leaning forward away from the enthusiastic lady with the blue hat,
and that the paper serpentines had dropped from his hands. Hartley
thought that the rapid motion must have made him a little giddy, but
presently, before the merry-go-round had quite stopped, he saw the
man leap down and hurry towards him through the crowd. Ste.
Marie's face was grave and pale. He caught Hartley's arm in his hand
and turned him round, crying in a low voice—
"Come out of this as quickly as you can! No, in the other
direction. I want to get away at once."
 "What's the matter?" Hartley demanded. "Lady in the blue hat
too friendly? Well, if you're going to play this kind of game, you
might as well play it."
"Helen Benham was down there in the crowd," said Ste. Marie.
"On the opposite side from you. She was with a party of people who
got out of two motor-cars, to look on. They were in evening things,
so they had come from dinner somewhere, I suppose. She saw me."
"The devil!" said Hartley under his breath. Then he gave a shout
of laughter, demanding—
"Well, what of it? You weren't committing any crime, were you?
There's no harm in riding a silly pig in a silly merry-go-round.
Everybody does it in these fête things." But even as he spoke he
knew how extremely unfortunate the meeting was, and the laughter
went out of his voice.
"I'm afraid," said Ste. Marie, "she won't see the humour of it.
Good God, what a thing to happen! You know well enough what
she'll think of me.
"At five o'clock this afternoon," he said bitterly, "I left her with a
great many fine high-sounding words about the quest I was to give
my days and nights to—for her sake. I went away from her like a—
knight going into battle—consecrated. I tell you, there were tears in
her eyes when I went. And now, now, at midnight, she sees me
riding a galloping pig in a street fête with a girl from the boulevards
sitting on the pig with me and holding me round the neck before a
thousand people. What will she think of me? What but one thing can
she possibly think? Oh, I know well enough! I saw her face before
she turned away.
 "And," he cried, "I can't even go to her and explain—if there's
anything to explain, and I suppose there is not. I can't even go to
her. I've sworn not to see her."
"Oh, I'll do that," said the other man. "I'll explain it to her, if any
explanation's necessary. I think you'll find that she will laugh at it."
But Ste. Marie shook his head.
"No, she won't," said he. And Hartley could say no more, for he
knew Miss Benham, and he was very much afraid that she would not
laugh.
They found a fiacre at the side of the square and drove home at
once. They were almost entirely silent all the long way, for Ste.
Marie was buried in gloom, and the Englishman, after trying once or
twice to cheer him up, realised that he was best left to himself just
then, and so held his tongue. But in the Rue d'Assas as Ste. Marie
was getting down—Hartley kept the fiacre to go on to his rooms in
the Avenue de l'Observatoire—he made a last attempt to lighten the
man's depression. He said—
"Don't you be a silly ass about this! You're making much too
much of it, you know. I'll go to her to-morrow or next day and
explain, and she'll laugh—if she hasn't already done so.
"You know," he said, almost believing it himself, "you are paying
her a dashed poor compliment in thinking she's so dull as to
misunderstand a little thing of this kind. Yes, by Jove, you are!"
Ste. Marie looked up at him, and his face, in the light of the
cab-lamp, showed a first faint gleam of hope.
"Do you think so?" he demanded. "Do you really think that?
Maybe I am. But— O Lord, who would understand such an idiocy?
Sacred imbecile that I am: why was I ever born? I ask you." He
turned abruptly and began to ring at the door, casting a brief "Good-
night," over his shoulder. And, after a moment, Hartley gave it up
and drove away.
Above, in the long shallow front room of his flat, with the three
windows overlooking the Gardens, Ste. Marie made lights, and after
much rummaging unearthed a box of cigarettes of a peculiarly
delectable flavour, which had been sent him by a friend in the
Khedivial household. He allowed himself one or two of them now
and then, usually in sorrowful moments, as an especial treat. And
this seemed to him to be the moment for smoking all there were
left. Surely his need had never been greater. In England he had, of
course learnt to smoke a pipe, but pipe smoking always remained
with him a species of accomplishment; it never brought him the
deep and ruminative peace with which it enfolds the Anglo-Saxon
heart. The vieux Jacob of old-fashioned Parisian Bohemia inspired in
him unconcealed horror, of cigars he was suspicious because, he
said, most of the unpleasant people he knew smoked cigars: so he
soothed his soul with cigarettes, and he was usually to be found with
one between his fingers.
He lighted one of the precious Egyptians, and after a first
ecstatic inhalation went across to one of the long windows, which
was open, and stood there with his back to the room, his face to the
peaceful fragrant night. A sudden recollection came to him of that
other night a month before, when he had stood on the Pont des
Invalides with his eyes upon the stars, his feet upon the ladder
thereunto. His heart gave a sudden exultant leap within him when
he thought how far and high he had climbed, but after the leap it
shivered and stood still when this evening's misadventure came
before him.
Would she ever understand? He had no fear that Hartley would
not do his best with her. Hartley was as honest and as faithful as
ever a friend was in this world. He would do his best. But even then
—— It was the girl's inflexible nature that made the matter so
dangerous. He knew that she was inflexible, and he took a curious
pride in it. He admired it. So must have been those calm-eyed
ancient ladies for whom other Ste. Maries went out to do battle. It
was wellnigh impossible to imagine them lowering their eyes to silly
revelry. They could not stoop to such as that. It was beneath their
high dignity. And it was beneath hers also. As for himself, he was a
thing of patches. Here a patch of exalted chivalry—a noble patch—
there a patch of bourgeois child-like love of fun; here a patch of
melancholic asceticism, there one of something quite the reverse. A
hopeless patchwork he was. Must she not shrink from him when she
knew? He could not quite imagine her understanding the wholly
trivial and meaningless impulse that had prompted him to ride a
galloping pig and cast paper serpentines at the assembled world.
Apart from her view of the affair he felt no shame in it. The
moment of childish gaiety had been but a passing mood. It had in
no way slackened his tense enthusiasm, dulled the keenness of his
spirit, lowered his high flight. He knew that well enough. But he
wondered if she would understand, and he could not believe it
possible. The mood of exaltation in which they had parted that
afternoon came to him, and then the sight of her shocked face as he
had seen it in the laughing crowd in the Place Blanche.
 "What must she think of me?" he cried aloud. "What must she
think of me?"
So for an hour or more he stood in the open window staring
into the fragrant night, or tramped up and down the long room, his
hands behind his back, kicking out of his way the chairs and things
which impeded him—torturing himself with fears and regrets and
fancies, until at last in a calmer moment he realised that he was
working himself up into an absurd state of nerves over something
which was done and could not now be helped. The man had an odd
streak of fatalism in his nature—that will have come of his southern
blood—and it came to him now in his need. For the work upon which
he was to enter with the morrow he had need of clear wits, not
scattered ones; a calm judgment, not disordered nerves. So he took
himself in hand, and it would have been amazing to any one
unfamiliar with the abrupt changes of the Latin temperament, to see
how suddenly Ste. Marie became quiet and cool and master of
himself.
"So for an hour or more he stood in the open window
staring into the fragrant night."
 "It is done," he said with a little shrug, and if his face was for a
moment bitter it quickly enough became impassive. "It is done, and
it cannot be undone—unless Hartley can undo it. And now, revenons
à nos moutons!
"Or at least," said he, looking at his watch—and it was between
one and two—"at least to our beds!"
So he went to bed, and, so well had he recovered from his fit of
excitement, he fell asleep almost at once. But, for all that, the
jangled nerves had their revenge. He who commonly slept like the
dead, without the slightest disturbance, dreamed a strange dream.
It seemed to him that he stood spent and weary in a twilit place, a
waste place at the foot of a high hill. At the top of the hill She sat
upon a sort of throne, golden in a beam of light from heaven—
serene, very beautiful, the end and crown of his weary labours. His
feet were set to the ascent of the height whereon she waited, but he
was withheld. From the shadows at the hill's foot a voice called to
him in distress, anguish of spirit—a voice he knew, but he could not
say whose voice. It besought him out of utter need, and he could
not turn away from it.
Then from those shadows eyes looked upon him, very great and
dark eyes, and they besought him too; he did not know what they
asked, but they called to him like the low voice, and he could not
turn away.
He looked to the far height, and with all his power he strove to
set his feet towards it—the goal of long labour and desire—but the
eyes and the piteous voice held him motionless, for they needed
him.
 From this anguish he awoke trembling. And after a long time,
when he was composed, he fell asleep once more, and once more
he dreamed the dream.
So morning found him pallid and unrefreshed. But by daylight
he knew whose eyes had besought him, and he wondered, and was
a little afraid.

CHAPTER IX
STE. MARIE GOES UPON A JOURNEY AND
RICHARD HARTLEY PLEADS FOR HIM

It may as well be admitted at the outset that neither Ste. Marie nor
Richard Hartley proved themselves to be geniuses, hitherto
undeveloped, in the detective science. They entered upon their self-
appointed task with a fine fervour, but, as Miss Benham had
suggested, with no other qualifications in particular. Ste. Marie had a
theory that when engaged in work of this nature you went into
questionable parts of the city, ate and drank cheek by jowl with
questionable people; if possible got them drunk while you remained
sober (difficult feat), and sooner or later they said things which put
you on the right road to your goal, or else confessed to you that
they themselves had committed the particular crime in which you
were interested. He argued that this was the way it happened in
books, and that surely people didn't write books about things of
which they were ignorant.
 Hartley, on the other hand, preferred the newer or scientific
methods. You sat at home with a pipe and a whisky and water—if
possible in a long dressing-gown with a cord round its middle. You
reviewed all the known facts of the case, and you did mathematics
about them with Xs and Ys and many other symbols, and in the end,
by a system of elimination, you proved that a certain thing must
infallibly be true. The chief difficulty for him in this was, he said, that
he had been at Oxford instead of at Cambridge, and so the
mathematics was rather beyond him.
In practice, however, they combined the two methods, which
was doubtless as well as if they hadn't, because for some time they
accomplished nothing whatever, and so neither one was able to
sneer at the other's stupidity.
This is not to say that they found nothing in the way of clues.
They found an embarrassment of them, and for some days went
about in a fever of excitement over these; but the fever cooled when
clue after clue turned out to be misleading. Of course Ste. Marie's
first efforts were directed towards tracing the movements of the
Irishman, O'Hara, but the efforts were altogether unavailing. The
man seemed to have disappeared as noiselessly and completely as
had young Arthur Benham himself. He was unable even to settle
with any definiteness the time of the man's departure from Paris.
Some of O'Hara's old acquaintances maintained that they had seen
the last of him two months before, but a shifty-eyed person in rather
cheaply smart clothes came up to Ste. Marie one evening in Maxim's,
and said he had heard that Ste. Marie was making inquiries about M.
O'Hara. Ste. Marie said he was, and that it was an affair of money,
whereupon the cheaply smart individual declared that M. O'Hara had
left Paris six months before to go to the United States of America,
and that he had had a picture postal card, some weeks since, from
New York. The informant accepted an expensive cigar and a
Dubonnet by way of reward, but presently departed into the night,
and Ste. Marie was left in some discouragement, his theory badly
damaged.
He spoke of this encounter to Richard Hartley, who came on
later to join him, and Hartley, after an interval of silence and smoke,
said—
"That was a lie. The man lied."
"Name of a dog, why?" demanded Ste. Marie, but the
Englishman shrugged his shoulders.
"I don't know," he said. "But I believe it was a lie. The man
came to you, sought you out to tell his story, didn't he? And all the
others have given a different date? Well, there you are! For some
reason this man or some one behind him—O'Hara himself, probably
—wants you to believe that O'Hara is in America. I dare say he's in
Paris all the while."
"I hope you're right," said the other. "And I mean to make sure,
too. It certainly was odd, this strange being hunting me out to tell
me that. I wonder, by the way, how he knew I'd been making
inquiries about O'Hara. I've questioned only two or three people,
and then in the most casual way. Yes, it's odd."
It was about a week after this—a fruitless week, full of the
alternate brightness of hope and the gloom of disappointment—that
he met Captain Stewart, to whom he had been more than once on
the point of appealing. He happened upon him quite by chance one
morning in the Rue Royale. Captain Stewart was coming out of a
shop, a very smart-looking shop, devoted, as Ste. Marie, with some
surprise and much amusement, observed, to ladies' hats, and the
price of hats must have depressed him, for he looked in an ill
humour and older and more yellow than usual. But his face altered
suddenly when he saw the younger man, and he stopped, and shook
Ste. Marie's hand with every evidence of pleasure.
"Well met! well met!" he exclaimed. "If you are not in a hurry,
come and sit down somewhere and tell me about yourself."
They picked their way across the street to the terrace of the
Taverne Royale, which was almost deserted at that hour, and sat
down at one of the little tables well back from the pavement, in a
corner.
"Is it fair?" queried Captain Stewart, "is it fair as a rival
investigator to ask you what success you have had?" Ste. Marie
laughed rather ruefully and confessed that he had as yet no success
at all.
"I've just come," said he, "from pricking one bubble that
promised well, and Hartley is up in Montmartre destroying another, I
fancy. Oh well, we didn't expect it to be child's play."
Captain Stewart raised his little glass of dry vermouth in an old-
fashioned salute, and drank from it.
"You," said he, "you were—ah, full of some idea of connecting
this man, this Irishman, O'Hara, with poor Arthur's disappearance.
You've found that not so promising, as you went on, I take it."
"Well, I've been unable to trace O'Hara," said Ste. Marie. "He
seems to have disappeared as completely as your nephew. I
suppose you have no clues to spare? I confess I'm out of them, at
the moment."
 "Oh, I have plenty," said the elder man. "A hundred. More than
I can possibly look after." He gave a little chuckling laugh.
"I've been waiting for you to come to me," he said. "It was a
little ungenerous perhaps, but we all love to say, 'I told you so.' Yes,
I have a great quantity of clues, and, of course, they all seem to be
of the greatest and most exciting importance. That's a way clues
have." He took an envelope from an inner pocket of his coat, and
sorted several folded papers which were in it.
"I have here," said he, "memoranda of two chances, shall I call
them?—which seem to me very good, though, as I have already
said, every clue seems good. That is the maddening, the heart-
breaking part of such an investigation. I have made these brief notes
from letters received, one yesterday, one the day before, from an
agent of mine who has been searching the bains de mer of the north
coast. This agent writes that some one very much resembling poor
Arthur has been seen at Dinard and also at Deauville, and he urges
me to come there, or to send a man there at once to look into the
matter. You will ask, of course, why this agent himself does not
pursue the clue he has found. Unfortunately he has been called to
London upon some pressing family matter of his own; he is an
Englishman."
"Why haven't you gone yourself?" asked Ste. Marie. But the
elder man shrugged his shoulders and smiled a tired deprecatory
smile.
"Oh, my friend," said he, "if I should attempt personally to
investigate one half of these things, I should be compelled to divide
myself into twenty parts. No, I must stay here. There must be, alas!
the spider at the centre of the web. I cannot go, but if you think it
worth while I will gladly turn over the memoranda of these last clues
to you. They may be the true clues, they may not. At any rate, some
one must look into them. Why not you and your partner—or shall I
say assistant?"
"Why, thank you!" cried Ste. Marie. "A thousand thanks. Of
course I shall be—we shall be glad to try this chance. On the face of
it, it sounds very reasonable. Your nephew, from what I remember
of him, is much more apt to be in some place that is amusing—some
place of gaiety—than hiding away where it is merely dull, if he has
his choice in the matter, that is—if he is free. And yet——" he turned
and frowned thoughtfully at the elder man.
"What I want to know," said he, "is how the boy is supporting
himself all this time. You say he had no money, or very little, when
he went away. How is he managing to live, if your theory is correct—
that he is staying away of his own accord? It costs a lot of money to
live as he likes to live."
Captain Stewart nodded.
"Oh, that," said he, "that is a question I have often proposed to
myself. Frankly it's beyond me. I can only surmise that poor Arthur,
who had scattered a small fortune about in foolish loans, managed,
before he actually disappeared (mind you, we didn't begin to look for
him until a week had gone by), managed to collect some of this
money, and so went away with something in pocket. That, of course,
is only a guess."
"It is possible," said Ste. Marie doubtfully, "but—I don't know. It
is not very easy to raise money from the sort of people I imagine
your nephew to have lent it to. They borrow but they don't repay."
He glanced up with a half-laughing half-defiant air.
 "I can't," said he, "rid myself of a belief that the boy is here in
Paris and he is not free to come or go. It's only a feeling, but it is
very strong in me. Of course I shall follow out these clues you've
been so kind as to give me. I shall go to Dinard and Deauville, and
Hartley, I imagine, will go with me; but I haven't great confidence in
them."
Captain Stewart regarded him reflectively for a time, and in the
end he smiled.
"If you will pardon my saying it," he said, "your attitude is just a
little womanlike. You put away reason for something vaguely
intuitive. I always distrust intuition myself." Ste. Marie frowned a
little and looked uncomfortable. He did not relish being called
womanlike—few men do—but he was bound to admit that the elder
man's criticism was more or less just.
"Moreover," pursued Captain Stewart, "you altogether ignore the
point of motive—as I may have suggested to you before. There
could be no possible motive, so far as I am aware, for kidnapping or
detaining or in any way harming my nephew except the desire for
money; but, as you know, he had no large sum of money with him,
and no demand has been made upon us since his disappearance.
I'm afraid you can't get round that."
"No," said Ste. Marie, "I'm afraid I can't. Indeed, leaving that
aside (and it can't be left aside), I still have almost nothing with
which to prop up my theory. I told you it was only a feeling."
He took up the memoranda which Captain Stewart had laid
upon the marble-topped table between them, and read the notes
through.
 "Please," said he, "don't think I am ungrateful for this chance. I
am not. I shall do my best with it, and I hope it may turn out to be
important." He gave a little wry smile.
"I have all sorts of reasons," he said, "for wishing to succeed as
soon as possible. You may be sure that there won't be any delays on
my part. And now I must be going on. I am to meet Hartley for
lunch on the other side of the river, and, if we can manage it, I
should like to start north this afternoon or evening.
"Good!" said Captain Stewart, smiling. "Good! that is what I call
true promptness. You lose no time at all. Go to Dinard and Deauville,
by all means, and look into this thing thoroughly. Don't be
discouraged if you meet with ill success at first. Take Mr. Hartley with
you and do your best." He paid for the two glasses of aperatif, and
Ste. Marie could not help observing that he left on the table a very
small tip. The waiter cursed him audibly as the two walked away.
"If you have returned by a week from to-morrow," he said, as
they shook hands, "I should like to have you keep that evening—
Thursday—for me. I am having a very informal little party in my
rooms. There will be two or three of the opera people there, and
they will sing for us, and the others will be amusing enough. All
young. All young. I like young people about me." He gave his odd
little mewing chuckle. "And the ladies must be beautiful as well as
young. Come if you are here! I'll drop a line to Mr. Hartley also." He
shook Ste. Marie's hand and went away down the street towards the
Rue du Faubourg St. Honoré, where he lived.
Ste. Marie met Hartley as he expected to do, at lunch, and they
talked over the possibilities of the Dinard and Deauville expedition.
In the end they decided that Ste. Marie should go alone, but that he
was to telegraph, later on, if the clue looked promising. Hartley had
two or three investigations on foot in Paris, and stayed on to
complete these. Also he wished, as soon as possible, to see Helen
Benham and explain Ste. Marie's ride on the galloping pigs. Ten days
had elapsed since that evening, but Miss Benham had gone into the
country the next day to make a visit at the de Saulnes' chateau on
the Oise.
So Ste. Marie packed a portmanteau with clothes and things,
and departed by a mid-afternoon train to Dinard, and, towards five,
Richard Hartley walked down to the Rue de l'Université. He thought
it just possible that Miss Benham might by now have returned to
town, but if not he meant to have half an hour's chat with old David
Stewart, whom he had not seen for some weeks.
At the door he learnt that Mademoiselle was that very day
returned and was at home. So he went in to the drawing-room,
reserving his visit to old David until later. He found the room divided
into two camps. At one side Mrs. Benham conversed in melancholic
monotones with two elderly French ladies, who were clad in
depressing black of a dowdiness surpassed only in English provincial
towns. It was as if the three mourned together over the remains of
some dear one who lay dead amongst them. Hartley bowed low with
an uncontrollable shiver, and turned to the tea-table, where Miss
Benham sat in the seat of authority, flanked by a young American
lady, whom he had met before, and by Baron de Vries, whom he had
not seen since the evening of the de Saulnes' dinner party.
Miss Benham greeted him with evident pleasure, and to his
great delight remembered just how he liked his tea—three pieces of
sugar and no milk. It always flatters a man when his little tastes of
this sort are remembered. The four fell at once into conversation
together, and the young American lady asked Hartley why Ste. Marie
was not with him.
"I thought you two always went about together," she said.
"Were never seen apart and all that—a sort of modern Damon and
Phidias." Hartley caught Baron de Vries' eye and looked away again
hastily.
"My—ah, Phidias," said he, resisting an irritable desire to correct
the lady, "got mislaid to-day. It shan't happen again, I promise you.
He's a very busy person just now, though. He hasn't time for social
dissipation. I'm the butterfly of the pair." The lady gave a sudden
laugh.
"He was busy enough the last time I saw him," she said,
crinkling her eyelids. She turned to Miss Benham.
"Do you remember that evening we were going home from the
Madrid, and motored round by Montmartre to see the fête?"
"Yes," said Miss Benham, unsmiling, "I remember."
"Your friend, Ste. Marie," said the American lady to Hartley,
"was distinctly the lion of the fête—at the moment we arrived,
anyhow. He was riding a galloping pig and throwing those paper
streamer things—what do you call them?—with both hands, and a
genial lady in a blue hat was riding the same pig and helping him
out. It was just like the Vie de Bohême and the other books. I found
it charming."
Baron de Vries emitted an amused chuckle.
"That was very like Ste. Marie," he said. "Ste. Marie is a very
exceptional young man. He can be an angel one moment, a child
playing with toys the next, and—well, a rather commonplace social
favourite the third. It all comes of being romantic—imaginative. Ste.
Marie—I know nothing about this evening of which you speak—but
Ste. Marie is quite capable of stopping on his way to a funeral to ride
a galloping pig—or on his way to his own wedding.
"And the pleasant part of it is," said Baron de Vries, "that the
lad would turn up at either of these two ceremonies not a bit the
worse, outside or in, for his ride."
"Ah, now that's an oddly close shot," said Hartley. He paused a
moment, looking towards Miss Benham, and said—
"I beg pardon! Were you going to speak?"
"No," said Miss Benham, moving the things about on the tea-
table before her, and looking down at them. "No, not at all!"
"You came oddly close to the truth," the man went on, turning
back to Baron de Vries. He was speaking for Helen Benham's ears,
and he knew she would understand that, but he did not wish to
seem to be watching her.
"I was with Ste. Marie on that evening," he said. "No! I wasn't
riding a pig, but I was standing down in the crowd throwing
serpentines at the people who were. And I happen to know that he
—that Ste. Marie was on that day, that evening, more deeply
concerned about something, more absolutely wrapped up in it,
devoted to it, than I have ever known him to be about anything
since I first knew him. The galloping pig was an incident that made,
except for the moment, no impression whatever upon him." Hartley
nodded his head.
"Yes," said he, "Ste. Marie can be an angel one moment and a
child playing with toys the next. When he sees toys he always plays
with them, and he plays hard, but when he drops them they go
completely out of his mind."
The American lady laughed.
"Gracious me!" she cried. "You two are emphatic enough about
him, aren't you?"
"We know him," said Baron de Vries. Hartley rose to replace his
empty cup on the tea-table. Miss Benham did not meet his eyes, and
as he moved away again she spoke to her friend about something
they were going to do on the next day, so Hartley went across to
where Baron de Vries sat at a little distance, and took a place beside
him on the chaise longue. The Belgian greeted him with raised
eyebrows and the little half-sad half-humorous smile which was
characteristic of him in his gentler moments.
"You were defending our friend with a purpose," he said in a
low voice. "Good! I am afraid he needs it—here." The younger man
hesitated a moment. Then he said—
"I came on purpose to do that. Ste. Marie knows that she saw
him on that confounded pig. He was half wild with distress over it
because—well, the meeting was singularly unfortunate, just then. I
can't explain——"
"You needn't explain," said the Belgian gravely. "I know. Helen
told me some days since, though she did not mention this encounter.
Yes, defend him with all your power, if you will. Stay after we others
have gone and—have it out with her. The Phidias lady (I must
remember that mot, by the way) is preparing to take her leave now,
and I will follow her at once. She shall believe that I am enamoured
—that I sigh for her.
 "Eh!" said he, shaking his head. And the lines in the kindly old
face seemed to deepen, but in a sort of grave tenderness. "Eh, so
love has come to the dear lad at last! Ah! of course, the hundred
other affairs! Yes, yes. But they were light. No seriousness in them.
The ladies may have loved. He didn't very much. This time, I'm
afraid——"
Baron de Vries paused as if he did not mean to finish his
sentence, and Hartley said—
"You say 'afraid'! Why, afraid?"
The Belgian looked up at him reflectively.
"Did I say 'afraid'?" he asked. "Well—perhaps it was the word I
wanted. I wonder if these two are fitted for each other. I am fond of
them both. I think you know that, but—she's not very flexible, this
child. And she hasn't much humour. I love her, but I know those
things are true. I wonder if one ought to marry Ste. Marie without
flexibility and without humour."
"If they love each other," said Richard Hartley, "I expect the
other things don't count. Do they?"
Baron de Vries rose to his feet, for he saw that the Phidias lady
was going.
"Perhaps not," said he; "I hope not. In any case, do your best
for him with Helen. Make her comprehend if you can. I am afraid
she is unhappy over the affair." He made his adieux and went away
with the American lady, to that young person's obvious excitement.
And after a moment the three ladies across the room departed also,
Mrs. Benham explaining that she was taking her two friends up to
her own sitting-room to show them something vaguely related to the
heathen. So Hartley was left alone with Helen Benham.
 It was not his way to beat about the bush, and he gave battle at
once. He said, standing to say it more easily—
"You know why I came here to-day. It was the first chance I've
had since that—unfortunate evening. I came on Ste. Marie's
account."
Miss Benham said a weak—
"Oh!" And because she was nervous and overwrought and
because the thing meant so much to her, she said cheaply—
"He owes me no apologies. He has a perfect right to act as he
pleases, you know."
The Englishman frowned across at her.
"I didn't come to make apologies," said he. "I came to explain.
Well, I have explained—Baron de Vries and I together. That's just
how it happened, and that's just how Ste. Marie takes things. The
point is, that you've got to understand it. I've got to make you."
The girl smiled up at him dolefully.
"You look," she said, "as if you were going to beat me if
necessary. You look very warlike."
"I feel warlike," the man said, nodding. He said—
"I'm fighting for a friend to whom you are doing, in your mind,
an injustice. I know him better than you do, and I tell you you're
doing him a grave injustice. You're failing altogether to understand
him."
"I wonder," the girl said, looking very thoughtfully down at the
table before her.
"I know," said he.
Quite suddenly she gave a little overwrought cry, and she put
up her hands over her face.
 "Oh, Richard!" she said, "that day when he was here! He left me
—— Oh, I cannot tell you at what a height he left me! It was
something new and beautiful. He swept me to the clouds with him.
And I might—perhaps I might have lived on there. Who knows? But
then that hideous evening! Ah, it was too sickening, the fall back to
common earth again!"
"I know," said the man gently, "I know. And he knew, too.
Directly he'd seen you he knew how you would feel about it. I'm not
pretending that it was of no consequence. It was unfortunate, of
course. But—the point is it did not mean in him any slackening, any
stooping, any letting go. It was a moment's incident. We went to the
wretched place by accident after dinner. Ste. Marie saw those
childish lunatics at play, and for about two minutes he played with
them. The lady in the blue hat made it appear a little more extreme,
and that's all."
Miss Benham rose to her feet and moved restlessly back and
forth.
"Oh, Richard!" she said, "the golden spell is broken—the
enchantment he laid upon me that day. I'm not like him, you know.
Oh, I wish I were! I wish I were! I can't change from hour to hour. I
can't rise to the clouds again after my fall to earth. It has all—
become something different.
"Don't misunderstand me," she cried; "I don't mean that I've
ceased to care for him. No, far from that! But I was in such an
exalted heaven, and now I'm not there any more. Perhaps he can lift
me to it again. Oh yes, I'm sure he can when I see him once more;
but I wanted to go on living there so happily while he was away! Do
you understand at all?"
 "I think I do," the man said, but he looked at her very curiously
and a little sadly; for it was the first time he had ever seen her
swept from her superb poise by any emotion, and he hardly
recognised her. It was very bitter to him to realise that he could
never have stirred her to this, never under any conceivable
circumstances.
The girl came to him where he stood and touched his arm with
her hand.
"He is waiting to hear how I feel about it all, isn't he?" she said.
"He is waiting to know that I understand. Will you tell him a little lie
for me, Richard? No! You needn't tell a lie; I will tell it. Tell him that I
said I understood perfectly. Tell him that I was shocked for a
moment, but that afterwards I understood and thought no more
about it. Will you tell him I said that? It won't be a lie from you,
because I did say it. Oh, I will not grieve him or hamper him now
while he is working in my cause! I'll tell him a lie rather than have
him grieve."
"Need it be a lie?" said Richard Hartley. "Can't you truly believe
what you've said?"
She shook her head slowly.
"I'll try," said she, "but—my golden spell is broken, and I can't
mend it alone. I'm sorry."
He turned with a little sigh to leave her, but Miss Benham
followed him towards the door of the drawing-room.
"You're a good friend, Richard," she said, when she had come
near. "You're a good friend to him."
"He deserves good friends," said the man stoutly. "And besides,"
said he, "we're brothers in arms nowadays. We've enlisted together
to fight for the same cause."
The girl fell back with a little cry. "Do you mean," she said, after
a moment, "do you mean that you are working with him—to find
Arthur?"
Hartley nodded.
"But," said she stammering, "but, Richard——"
The man checked her. "Oh, I know what I'm doing," said he.
"My eyes are open. I know that I'm not—well, in the running. I work
for no reward except a desire to help you and Ste. Marie. That's all.
It pleases me to be useful."
He went away with that, not waiting for an answer; and the girl
stood where he had left her, staring after him.

CHAPTER X
CAPTAIN STEWART ENTERTAINS

Ste. Marie returned, after three days, from Dinard in a depressed

and somewhat puzzled frame of mind. He had found no trace
whatever of Arthur Benham either at Dinard or at Deauville, and,
what was more, he was unable to discover that any one even
remotely resembling that youth had been seen at either place. The
matter of identification, it seemed to him, should be a rather simple
one. In the first place, the boy's appearance was not at all French,
nor for that matter English: it was very American. Also he spoke
French—so Ste. Marie had been told—very badly, having for the
language that scornful contempt peculiar to Anglo-Saxons of a
certain type. His speech, it seemed, was, like his appearance, ultra-
American, full of strange idioms and oddly pronounced. In short,
such a youth would be rather sure to be remembered by any hotel
management and staff with which he might have come in contact.
At first Ste. Marie pursued his investigations quietly and, as it
were, casually, but, after his initial failure, he went to the
managements of the various hotels and lodging-houses and to the
cafés and bathing establishments, and told them with all frankness a
part of the truth—that he was searching for a young man whose
disappearance had caused great distress to his family. He was not
long in discovering that no such young man could have been either
in Dinard or Deauville.
The thing which puzzled him was that, apart from finding no
trace of the missing boy, he also found no trace of Captain Stewart's
agent—the man who had been first on the ground. No one seemed
able to recollect that such a person had been making inquiries, and
Ste. Marie began to suspect that his friend was being imposed upon.
He determined to warn Stewart that his agents were earning their
fees too easily.
So he returned to Paris more than a little dejected and sore over
this waste of time and effort. He arrived by a noon tram, and drove
across the city in a fiacre to the Rue d'Assas. But as he was in the
midst of unpacking his portmanteau, for he kept no servant (a
woman came in once a day to "do" the rooms), the door-bell rang. It
was Baron de Vries, and Ste. Marie admitted him with an
exclamation of surprise and pleasure.
"You passed me in the street just now," explained the Belgian,
"and, as I was a few minutes early for a lunch engagement, I
Welcome to our website – the perfect destination for book lovers and
knowledge seekers. We believe that every book holds a new world,
offering opportunities for learning, discovery, and personal growth.
That’s why we are dedicated to bringing you a diverse collection of
books, ranging from classic literature and specialized publications to
self-development guides and children's books.

More than just a book-buying platform, we strive to be a bridge

connecting you with timeless cultural and intellectual values. With an
elegant, user-friendly interface and a smart search system, you can
quickly find the books that best suit your interests. Additionally,
our special promotions and home delivery services help you save time
and fully enjoy the joy of reading.

Join us on a journey of knowledge exploration, passion nurturing, and

personal growth every day!

ebookbell.com