Enterprise Vault Best Practice

Guide
Implementing Enterprise Vault
on VMware
This document provides design and deployment considerations for implementing Enterprise Vault on
VMware.

If you have any feedback or questions about this document, please email them to [email protected] stating
the document title.

This document applies to the following version(s) of Enterprise Vault: 10, 11, 12

© 2016 Veritas Technologies LLC. All rights reserved. Veritas and the Veritas Logo are trademarks or registered trademarks of Veritas
Technologies LLC or its affiliates in the U.S. and other countries. Other names may be trademarks of their respective owners. This
document is provided for informational purposes only and is not intended as advertising. All warranties relating to the information
in this document, either express or implied, are disclaimed to the maximum extent allowed by law. The information in this
document is subject to change without notice.
Enterprise Vault Best Practice Guide – Implementing Enterprise Vault on VMware

Document Control
Contributors

Who Contribution
Dan Strydom Author

Revision History

Version Date Changes

1.0 January 2011

2.0 September 2016 Updated formatting and minor changes to content

Related Documents

Version
Document Title Source
/ Date
Installing and Configuring EV Product Media\Documentation EV 12

Performance Guide for EV 12 www.veritas.com/docs/000100768 EV 12

1
Enterprise Vault Best Practice Guide – Implementing Enterprise Vault on VMware

Table of Contents
Scope of Document ............................................................................................................................................. 3
Intended Audience ............................................................................................................................................... 3
Terminology used in this document ..................................................................................................................... 3
Choosing the Right Platform for Your Environment ............................................................................................ 4
Sizing Enterprise Vault for VMware ..................................................................................................................... 5
ESX Host Configuration ....................................................................................................................................... 6
Optimizing VMware for Enterprise Vault .............................................................................................................. 7
CPU configuration ............................................................................................................................................ 7
Memory Configuration ...................................................................................................................................... 8
Disk Configuration ............................................................................................................................................ 9
Number of virtual machines per LUN ......................................................................................................... 12
Enterprise Vault per Server Storage Locations .......................................................................................... 13
Choosing Suitable Storage for Index Locations ......................................................................................... 14
General ESXi Storage Recommendations ................................................................................................. 16
Network Configuration.................................................................................................................................... 17
Virtualizing SQL Server ..................................................................................................................................... 17
Other Deployment Recommendations .............................................................................................................. 18
General VMware Recommendations ............................................................................................................. 18
Disable Anti-Virus for Index Locations ........................................................................................................... 18
Pagefile Size .................................................................................................................................................. 19
Index Volume Fragmentation ......................................................................................................................... 19
Ongoing Maintenance........................................................................................................................................ 20
Performance Monitoring ................................................................................................................................. 20
Windows Performance Counters ................................................................................................................... 21

2
Enterprise Vault Best Practice Guide – Implementing Enterprise Vault on VMware

Scope of Document
This document aims to provide guidance on designing and deploying Enterprise Vault on the VMware vSphere
platform.

This document should be used in conjunction with other performance and best practice guides as outlined in
the “Related Documents” section of this document.

Intended Audience
This document is aimed at system administrators, solutions architects, and consultants. It is assumed that the
reader has a thorough understanding of the architecture and operational aspects of Enterprise Vault. It is also
assumed that the reader has experience and understanding of VMware vSphere.

Terminology
Term Description
A virtual machine is an isolated software container that can run its
Virtual Machine own operating systems and applications as if it were a physical
computer
A hypervisor, also called a virtual machine manager (VMM), is a
Hypervisor program that allows multiple operating systems to share a single
hardware host
Formerly known as VMware Infrastructure, vSphere is compromises
VMware vSphere a suite of tools required to manage and support the virtual
infrastructure

vCPU CPU allocated to a virtual machine

Minimize downtime with automated restart of Virtual Machines on a

VMware High Availability (HA)
different physical host in the event of hardware failure
Reduce downtime from planned maintenance by migrating virtual
VMware vMotion
machines between different physical hosts
Provides continuous downtime protection by keeping a live shadow
VMware Fault Tolerance (FT)
copy of a virtual machine running on a separate host
Virtual Symmetric Multi- Enhances virtual machine performance by allowing a virtual machine
Processing (Virtual SMP) to use multiple physical processor cores simultaneously
Non-Uniform Memory Access (NUMA) is a computer memory design
NUMA
used for Multiprocessing

3
Enterprise Vault Best Practice Guide – Implementing Enterprise Vault on VMware

Choosing the Right Platform for Your Environment

Virtualization technology has helped many customers introduce cost savings both in terms of lowered data
center power consumption and cooling requirements. Virtualization typically also simplifies the datacenter
landscape through server consolidation, requiring less hardware to provide the same service to end users with
the added benefit of application independent high availability.

Application architectures are rapidly evolving towards highly distributed, loosely-coupled applications. The
conventional x86 computing model, in which applications are tightly coupled to physical servers, is too static
and restrictive to efficiently support most modern applications. With a virtual deployment, the architecture can
be as modular as is appropriate, without expanding the hardware footprint. The dynamic nature of virtual
machines mean that the design can grow and adapt as required, without the need for an initial “perfect” design.
Virtual deployments typically take minutes, can share currently deployed hardware, and can be adjusted “on
the fly” when more resources are required.

Certain server applications however are less suitable for virtualization, especially those requiring heavy use of
physical server resources such as CPU and memory. Traditionally customers have been reluctant to place
applications with high service level agreements such as Microsoft Exchange Server and SQL Server on a virtual
platform, not only because the application’s demand on resources meant that only one or two virtual machines
could co-exist on a single server, but also because the server could not offer the same performance it would
have on a physical server.

More powerful hardware, enhancements in virtualization technology and better support from application
vendors now mean that customers are looking to virtualize even the top-end applications.

A number of factors should be considered before deploying Enterprise Vault in a VMware environment:

 Enterprise Vault is heavily dependent on CPU resources. In a typical physical server configuration, it is
not unusual for the CPU to run at 90% or higher utilization while archiving is being performed. Generally,
the more powerful the processor, the better the ingestion and retrieval rates
 The recommended CPU and memory configuration for Enterprise Vault is 8 CPU cores and 16GB RAM
 It is recommended that CPU and Memory resources are dedicated (reserved) to the Enterprise Vault
server, and not shared with other virtual machines on the host. This aligns with VMware
recommendations for virtualized Microsoft Exchange Server and SQL Server implementations
 Other system components such as network and storage need to be sized accordingly to prevent them
from becoming a bottleneck

If the above considerations are acceptable and supported by the customer environment, then it is likely that
virtualizing the Enterprise Vault environment will be a good fit for the organization.

4
Enterprise Vault Best Practice Guide – Implementing Enterprise Vault on VMware

Sizing Enterprise Vault for VMware

One of the most important considerations when sizing Enterprise Vault is a thorough understanding of the
expected workload on each of the Enterprise Vault servers. However, the initial design of Enterprise Vault
should be done independent of whether it will run on physical servers or virtual machines, with the main
consideration being the customer requirements for archiving and eDiscovery.

It is outside the scope of this document to provide a design and sizing introduction to Enterprise Vault, but in
general terms, once the customer requirements are understood, a close look at the archive targets will help
determine what server resources will be require to not only archive the backlog but also keep up with the daily
change (also known as the “steady-state”).

Enterprise Vault sizing utilities such as Exchange Mailbox Analyzer, Domino Mail File Reporter, File System
Analyzer, SharePoint Analyzer and PST Analyzer will help provide a better understanding of what data is held
within the archive targets and what the impact of different archiving policies will be. The data collected from
these sizing utilities can then be used in the Enterprise Vault Sizing Estimator Tool. From here the tool will
provide a number of recommendations, including number of Enterprise Vault servers, expected performance
for different CPU configurations, SQL database recommendations and estimated Vault Store and Index storage
figures over a 3 year period.

The Sizing Estimator tool is able to adjust recommendations and expected performance if the option to deploy
on a virtualized environment is selected. Choosing this option will reduce the expected performance of each
Enterprise Vault server by 30%. This reduction is based on a sub-optimally configured VMware environment -
with an optimal VMware configuration the virtual machine performance will be close to that of a physical server.

The above sections represent an over simplified view of the design process as there will be many environment
specific factors that will affect a design.

The most common mistake when designing Enterprise Vault is to size for capacity, as opposed to sizing for
performance. The following sections in this guide will provide detail on how to design the various components
for optimal configuration.

5
Enterprise Vault Best Practice Guide – Implementing Enterprise Vault on VMware

ESX Host Configuration

Each ESX host should provide enough physical hardware resources to accommodate the planned workload
and provide some headroom in the event of a VMware HA failover or planned VMware vMotion migration of
live virtual machines for host hardware maintenance.

If planning to use VMware vMotion, confirm that the CPU supports vMotion (which in turn affects DRS) and
VMware Fault Tolerance. Confirm that the selected hardware is supported by searching the VMware product
compatibility database.

The following general recommendations can be made regarding the ESX host:

 Confirm that the host is running the latest version of the BIOS available
 Enable the “Turbo Boost” option in the BIOS, if supported by the CPU
 Some NUMA-capable systems provide an option in the BIOS to disable NUMA by enabling node
interleaving. Generally, the best performance is achieved by disabling node interleaving (in other words,
leaving NUMA enabled)
 Ensure that any hardware-assisted virtualization features (VT-x, AMD-V, EPT, RVI, etc.) are enabled
in the BIOS
 Disconnect or disable any physical hardware devices that will not be used. These might include devices
such as COM ports, LPT ports, USB controllers, floppy drives, network interfaces and storage
controllers. Disabling hardware devices can free interrupt resources. Additionally, some devices, such
as USB controllers, operate on a polling scheme that consumes extra CPU resources. Some PCI
devices reserve blocks of memory, making that memory unavailable to the ESX host
 Set the ESX host power policy to “Maximum performance” or disable power management altogether. It
is also recommended to disable C1E and other C-states in BIOS.

6
Enterprise Vault Best Practice Guide – Implementing Enterprise Vault on VMware

Optimizing VMware for Enterprise Vault

The following sections will cover recommendations regarding the VMware and Enterprise Vault components.
ESX resource configuration is vitally important to ensure applications such as Enterprise Vault run with optimal
performance.

Not all recommendations will be suitable to all environments – these recommendations are aimed at creating
the best possible platform specifically tuned for Enterprise Vault.

CPU configuration
VMware Virtual Symmetric Multi-Processing (Virtual SMP) enhances virtual machine performance by allowing
a virtual machine to use multiple physical processor cores simultaneously. VMware ESXi 4.1 supports the use
of up to 8 virtual CPUs per virtual machine, ESXi 5.0 supports up to 32 virtual cores. Virtual SMP allows virtual
machines to use multiple processors to execute multiple tasks concurrently, thereby increasing throughput.

The virtual processors are co-scheduled and run simultaneously, and (providing physical cores are available
on the host) each vCPU is mapped one-to-one to physical processors. In practice what this means is that if one
vCPU is running, a second vCPU is co-scheduled so that they execute nearly synchronously. Even when the
operating system is idle, the vCPUs will still perform a minimal amount of work and needs to be managed by
the ESX host, effectively competing with other vCPUs for system resources.

A common misconception is that by assigning more than 2 CPU cores per virtual machine you actually reduce
the performance of that server. While this may have been true in earlier versions of VMware, a number of
improvements in the VMware 4.1 and 5.0 co-scheduling algorithm means that the virtual machines with multiple
vCPUs are now more scalable and perform better, and the effect of idle multi-vCPU machines are reduced. In
ESX 4.1 and 5.0 the larger 4-core and 8-core vCPU systems not only scale better, but it also means that it’s
easier to scale the virtual machines up, compared to scaling out1.

The following general recommendations can be made regarding vCPU allocation for Enterprise Vault:

 Allocate 8 CPU cores to each Enterprise Vault server. For smaller or non-production environments with
less than 500 users 4 CPU cores can be allocated
 Set a CPU Reservation - this guarantees the CPU is dedicated to the virtual machine. Generally this
practice is not recommended because the reserved resources is then not available to other virtual
machines, but VMware recommends this practice in cases where service level agreements needs to
be guaranteed. The processor priority and bandwidth should be set to provide the virtual machine with
full utilization of the selected CPUs
 Ensure that the total number of vCPUs assigned to the virtual machines is equal or less than the total
number of cores on the ESX host
 Do not enable Hyperthreading – in most cases this provides little or no benefit to multi-CPU virtual
machines, internal testing have shown that Hyperthreading provides no performance benefit to
Enterprise Vault.

1 For more details see ESX CPU consideration sections in VMware Performance Best Practices available at
http://www.vmware.com/pdf/Perf_Best_Practices_vSphere4.0.pdf
and http://www.vmware.com/pdf/Perf_Best_Practices_vSphere5.0.pdf
7
Enterprise Vault Best Practice Guide – Implementing Enterprise Vault on VMware

Memory Configuration
Symantec recommends that each Enterprise Vault 10 server is allocated 16GB of memory. For smaller or non-
production environments (less than 500 user) 8GB of memory should be sufficient. The memory requirements
do however increase depending on the search requirements - specifically in environments where regular
eDiscovery searches are run the memory should be increased to 32GB.

Virtual machines also require memory beyond the amount allocated to account for memory overhead, this
memory needs to be available to the physical host. Memory overhead includes space reserved for virtual
machine devices, such as SVGA frame buffers and internal data structures. The amount of overhead required
depends on the number of vCPUs allocated per virtual machine, the amount of configured memory and whether
the operating system is 32 or 64-bit. As an example, a virtual machine with 2 vCPUs and 32GB of memory will
consume approximately 500MB of memory overhead. This memory overhead must be available on the ESX
host, and any other ESX host potentially hosting the Enterprise Vault virtual server.

The following diagram describes the use of memory settings available for a virtual machine:

Figure 1 - Memory Settings per virtual machine

The memory settings include the following parameters:

 Configured memory: The amount of memory allocated to the virtual machine when it was created
 Touched memory: The amount of memory actually in use by the virtual machine
 Swappable: Memory that can be reclaimed and reallocated by the vSphere balloon driver. If this
memory is being used by the virtual machine the balloon driver will cause the guest operating system
to swap. This value is the size of the per virtual machine swap file that is created on the VMFS file
system (as a .vswp file).

The following best practice recommendations can be made regarding memory allocation:

 Set the memory reservation to the configured size of the memory allocated to the virtual machine. Note
that by setting the reservation to the configured size of the virtual machine the vmkernel swap file will
be 0 bytes. This will consume less storage and help increase performance by eliminating ESX host-
level swapping. Note that Windows will still maintain its own separate swap/page file on the guest
operating system
 Setting memory reservations as described above may limit VMware vMotion as a virtual machine can
only be migrated if the target ESX host has sufficient free physical memory

8
Enterprise Vault Best Practice Guide – Implementing Enterprise Vault on VMware

 Use ESX performance counters to measure actual memory usage, see later section in this document
on the use of “Active”, “SwapIn” and “SwapOut” counters
 Ensure that the Enterprise Vault server does not reside on an ESX host where the memory have been
over committed
 It is generally recommended not to disable the memory balloon driver on the virtual machine (installed
with VMware Tools)
 If available, enable vSphere Dynamic Resource Scheduler (DRS) within the ESX cluster. Together with
reservations DRS can guarantee the resource availability
 Configure the virtual machine memory size to be greater than the average memory usage of the
Enterprise Vault server. This will avoid guest operating system swapping. In a typical Enterprise Vault
environment around 6GB of RAM will be used in core activities during the day, increasing to the full
available 16GB recommended memory with journaling and searching (end user searches, Virtual Vault
activity, etc.). For more details on expected server throughput for different CPU and memory
configurations refer to the EV Performance Guide.
 If the Enterprise Vault virtual server requires more memory than has been allocated, the performance
will reduce as the guest operating system will have to perform swap operations. It is recommended that
the host is allocated a swap file size 1.5x that of the memory assigned to the virtual machine (as per
standard recommendation for physical hosts).

The recommendations relating to memory are purposely conservative to avoid kernel swapping between ESX
and the guest operating system. Once the workload is known and predictable, if VMware vCenter reports that
steady state active memory usage is below the amount of memory on the ESX host, then the reservation
settings may be relaxed to the steady state active memory value. To get a better understanding of this scenario,
refer to VMworld content “TA2627 – Understanding “Host” and “Guest” Memory Usage and Related Memory
Management Concepts”2.

Disk Configuration
Storage performance depends on many factors, including the workload, hardware, RAID level, cache size,
stripe size, etc. The storage vendor documentation should always be consulted along with recommendations
from VMware.

VMware Virtual Machine File System (VMFS) is a high performance cluster file system developed to scale
beyond the use of a single system. VMFS provide multiple virtual machines shared access to a consolidated
pool of clustered storage, and provide the foundation for enabling services such as vMotion, Distributed
Resource Scheduler (DRS) and VMware High Availability. Each virtual machine is encapsulated in a small set
of files; in most cases these files are stored using the VMFS file system on physical SCSI disks.

VMware supports Fibre-Channel, iSCSI, and NAS shared-storage protocols. VMware storage virtualization can
be categorized into three layers of storage technology.

2 http://www.vmworld.com/docs/DOC-3817 (VMworld access is required, apply for a free account)

9
Enterprise Vault Best Practice Guide – Implementing Enterprise Vault on VMware

Figure 2 - VMware Storage Virtualization Stack

The storage array (consisting of physical disks) is presented as logical disks in the form of storage array volumes
or LUNs to the ESX server. Storage array LUNs are then formatted as VMFS volumes in which virtual disks
can be created. Virtual machines consist of virtual disks that are presented to the guest operating system as
disks that can be partitioned and used in file systems.

The best way to configure a LUN for a given VMFS volume is to size for throughput first and capacity second.
That is, you should aggregate the total I/O throughput for all applications or virtual machines that might run on
a given shared pool of storage; then make sure you have provisioned enough back-end disk spindles (disk
array cache) and appropriate storage service to meet the requirements.

Aside from VMFS, ESXi also supports raw device mapping (RDM) as referenced in the above diagram. RDM
effectively allows management and access of raw SCSI disks or LUNs as VMFS files. RDM is a special file on
a VMFS volume that acts as a proxy for a raw device. The file contains metadata used to manage the disk, and
is able to communicate direct access to the physical device. The mapping file, not the raw volume, is referenced
in the virtual machine configuration.

RDMs can be used in virtual or physical compatibility mode. Virtual mode specifies full virtualization of the
mapped drive, allowing the guest operating system to use the RDM like any other virtual disk file in a VMFS
volume. Physical mode specifies minimal SCSI virtualization of the mapped drive, giving the most flexibility to
SAN management software.

It is often a difficult decision when it comes to choosing between the VMFS and RDM for data volumes. They
both provide similar performance characteristics - random access (50% read, 50% write) performance is very
similar, but for sequential read operations RDM provides more I/O per second. Test results vary depending on

10
Enterprise Vault Best Practice Guide – Implementing Enterprise Vault on VMware

the data size used, for a more detailed analysis refer to the “Performance Characterization of VMFS and RDM
Using a SAN”3 whitepaper by VMware.

The following table described some of the main differences between VMFS and RDM.

VMFS RDM
Many virtual machines can be hosted on one Maps a single LUN to one virtual machine
volume (can be dedicated to one virtual machine)
Better flexibility, easier management Requires more LUNs, consideration of 256 LUN
limit per ESX host
Large third-party VMware ecosystem with V2P RDM volumes can help facilitate migrating
products to aid in certain support scenarios physical servers to virtual machines using the LUN
swing method
Cannot be used for in-guest clustering, does not Required for in-guest clustering. Cluster data and
support quorum disks as required by MSCS quorum disks should be configured using RDM
Supports VMware VMotion, HA and Distributed RDM guarantees no other virtual machine is able
Resource Scheduler (DRS) and VMware Site to use the LUNs
Recovery Manager (SRM)
Required to leverage array-level backup and
replication tools (VSS) integrated
Supports VMware VMotion, HA and Distributed
Resource Scheduler (DRS) and VMware Site
Recovery Manager (SRM)

Table 1 - VMFS and RDM Trade-offs

3
http://www.vmware.com/files/pdf/performance_char_vmfs_rdm.pdf

11
Enterprise Vault Best Practice Guide – Implementing Enterprise Vault on VMware

Depending on existing storage practices it may be advantageous to mix VMFS and RDM under the following
circumstances:

 Where 3rd party storage management software is already in use RDM can be used to leverage that
infrastructure, for example if using storage-based backups to disk
 RDM is required when using clustering technologies like Microsoft Clustering Services (MSCS)
 One advantage of using RDM volumes is that it gives the administrator the ability to point both virtual
and physical machines to the same storage.

Both VMFS and a mixed VMFS/RDM storage configuration are suitable for Enterprise Vault. In a mixed
configuration the guest operating system is installed on VMFS, while the Index volumes can reside on RDM.
The Vault Store partitions can also be stored on RDM, or if preferred on a NAS or other iSCSI storage partition.

For more information and a more detailed comparison between VMFS and RDM see the VMware VMFS Best
Practices Whitepaper4.

Number of virtual machines per LUN

You can assign multiple virtual machines to a VMFS LUN, however for best performance it is recommended
that a single LUN is dedicated to a virtual machine. The following diagram discusses the differences.

4 www.vmware.com/pdf/vmfs-best-practices-wp.pdf

12
Enterprise Vault Best Practice Guide – Implementing Enterprise Vault on VMware

Figure 3 - One versus many virtual machines in a LUN

It’s considered best practice to separate heavy I/O workloads from the shared pool of storage in order to
optimize the performance of applications requiring high transactional throughput. This approach is best
characterized as “consolidation with some level of isolation”.

When looking at the number of VMs per LUN, there is no exact rule to determine the limits of performance and
scalability, as workloads can vary significantly. These limits also depend on the number of ESX Servers sharing
concurrent access to a given VMFS volume. The key is to remember the upper limit of 256 LUNs per ESX
Server and consider that this number can limit the consolidation ratio if you take the concept of “1 LUN per VM”
too far.

Enterprise Vault per Server Storage Locations

The following table details storage areas within Enterprise Vault that can affect the application performance.
For optimal performance, each of the storage areas should be treated as separate areas and their individual
needs catered for.

13
Enterprise Vault Best Practice Guide – Implementing Enterprise Vault on VMware

Per server storage area Type of access Expected Size Recommendation

Index Locations Sequential Access. Depending on Fast SAN based storage – FC or
Large flat file area Indexing Level – iSCSI, can be mapped as VMFS
consisting of many Brief 3% of total or RDM. Storage should be
small files information capable of supporting 400+
archived, Full IOPS, 2000+ IOPS for
Indexing will eDiscovery environments. NAS
require 13% storage not considered
appropriate in most scenarios
(refer to Indexing Best Practice
paper for more information)
Vault Store Partitions Random Access. Very large but split Slower, lower tier storage such
Large flat file area up in smaller “vault as NAS is appropriate. Low IOPS
consisting of many partitions”. requirement
small files
MSMQ Data Folder Random Access 40GB Fast SAN based or local RAID1
disk. Ideally split onto a different
spindle set from MSMQ Log area
MSMQ Log Folder Sequential Access 10GB Fast SAN based or local RAID1
EMC Centera Collections Random Access 100GB Fast SAN based protected disk,
RAID1 or better performing stripe
Vault Server Cache Area Random Access 50GB Fast SAN based disk, RAID1 or
better performing stripe. Storage
should be capable of supporting
400 IOPS for typical
environment. Virtual Vault
environments can expect up to
1,000 IOPS during busy periods
Windows Temp Directory Random Access 2GB Fast SAN based disk, used for
temporary EV operations RAID1
or better performing stripe. Used
during archiving to process large
files
Table 2 - Enterprise Vault per Server Storage Locations

Choosing Suitable Storage for Index Locations

When choosing a suitable storage device, the whole storage solution should be considered. The supported
IOPS is just one aspect of performance and other areas should also be looked at such as connectivity – for
example Fiber and iSCSI are preferred over CIFS.

As per previous versions of Enterprise Vault, a high speed storage device is recommended for Index locations.
Lower tier NAS devices are generally not recommended for index locations, and should definitely not be used
to host indexes where any type of eDiscovery search is used.

NAS devices connected over CIFS shares is not suitable to host index volumes for certain environments due
to the slower connectivity speed.

The best devices are local storage, direct attached storage, or SAN LUNs.

In the case of local or direct attached storage:

 Use multiple controllers supporting multiple channels to distribute the load between index file locations
and provide sufficient throughput
 Provide battery-backed read and write cache to aid performance.

14
Enterprise Vault Best Practice Guide – Implementing Enterprise Vault on VMware

Before using partitions on a SAN, consider the I/O load together with any other applications that are already
using the SAN to ensure that the performance can be maintained. Ideally, the implementation should be
discussed with the SAN hardware vendor to ensure that optimum performance is achieved. Typically, LUNs
should be created across as many suitable disks as possible, using entire disks rather than partial disks to
prevent multiple I/O-intensive applications from using the same disks.

15
Enterprise Vault Best Practice Guide – Implementing Enterprise Vault on VMware

General ESXi Storage Recommendations

 Monitor storage on a regular basis. I/O latency statistics can be monitored using esxtop (or resxtop),
which reports device latency, time spent in the kernel, and latency seen by the guest operating system
 Check that the average latency for storage devices is not too high. Latency can be seen in esxtop (or
resxtop) by looking at the GAVG/cmd metric. A reasonable upper value for average latency depends
on the storage subsystem. This value should be confirmed with the storage vendor
 For iSCSI and NFS, if the network switch deployed for the data path supports VLAN, it might be
beneficial to create a VLAN just for the ESXi host's vmknic and the iSCSI/NFS server. This minimizes
network interference from other packet sources
 Local storage performance might be improved with write-back cache. Ensure that write-back cache
installed and contains a functional battery module
 To optimize storage array performance, spread I/O loads over the available paths to the storage (that
is, across multiple host bus adapters (HBAs) and storage processors)
 The default virtual storage adapter in ESXi 5.0 is either BusLogic Parallel, LSI Logic Parallel, or LSI
Logic SAS, depending on the guest operating system and the virtual hardware version. However, ESXi
also includes a paravirtualized SCSI storage adapter, PVSCSI (also called VMware Paravirtual). The
PVSCSI adapter offers a significant reduction in CPU utilization as well as potentially increased
throughput compared to the default virtual storage adapters, and would therefore be the best choice
for environments with very I/O-intensive guest applications. In order to use PVSCSI, your virtual
machine must be using virtual hardware version 7 or later
 If you choose to use the BusLogic Parallel virtual SCSI adapter, and are using a Windows guest
operating system, you should use the custom BusLogic driver included in the VMware Tools package
 The depth of the queue of outstanding commands in the guest operating system SCSI driver can
significantly impact disk performance. A queue depth that is too small, for example, limits the disk
bandwidth that can be pushed through the virtual machine. See the driver-specific documentation for
more information on how to adjust these settings – this is a very common performance bottleneck
 Ensure that an adequate number of spindles is available to support the I/O requirements with a
acceptable latency
 Make sure the disk partitions within the guest are aligned
 Use up-to-date HBA drivers recommended by the storage vendor
 Ensure that the storage array firmware is up to the latest recommended level

Make sure to give thought to the growth strategy up front. As the Vault Store and Index volume grows, how will
the growth of data files / LUNs / RAID groups be managed? It is much better to design for this up front than to
rebalance data files or LUN(s) later in a production deployment

16
Enterprise Vault Best Practice Guide – Implementing Enterprise Vault on VMware

Network Configuration
The following general best practice recommendations can be made regarding VMware network configuration
for Enterprise Vault:

 Separate virtual machine and infrastructure traffic – Keep virtual machine and VMkernel or service
console traffic separate. This can be accomplished physically using separate virtual switches that uplink
to separate physical NICs, or virtually using VLAN segmentation
 Use NIC Teaming – Use two physical NICs per vSwitch, and if possible, uplink the physical NICs to
separate physical switches. Teaming provides redundancy against NIC failure and, if connected to
separate physical switches, against switch failures. NIC teaming does not necessarily provide higher
throughput
 Enable PortFast on ESX host uplinks – Failover events can cause spanning tree protocol recalculations
that can set switch ports into a forwarding or blocked state to prevent a network loop. This process can
cause temporary network disconnects. To prevent this situation, set the switch ports connected to ESX
hosts to PortFast, which immediately sets the port back to the forwarding state and prevents link state
changes on ESX hosts from affecting the STP topology. Loops are not possible in virtual switches
 Converge Network and Storage I/O with 10Gbps Ethernet – When possible consolidating storage and
network traffic can provide simplified cabling and management over having to maintain separate
switching infrastructures.

Virtualizing SQL Server

Follow the recommendations of your VMware and Microsoft when you size and configure the environment for
SQL Server. The following general recommendations can be made when virtualizing SQL for Enterprise Vault:

 In a typical virtualized infrastructure, local disks would be used for the hypervisor and SAN-based
storage for the guest operating system images and data file locations. The operating system and data
storage partitions should be independent dedicated locations.
 Virtual hard disks should be created as fixed size and not dynamic
 The required memory capacity should be dedicated and prioritized to the virtual machine to prevent
dynamic allocation or sharing
 Avoid the use of hyper-threading by the hyper-visor
 The processor cores should be exclusively dedicated to the virtual machine, and the processor priority
and bandwidth set to provide the virtual machine with full utilization of the selected CPUs

17
Enterprise Vault Best Practice Guide – Implementing Enterprise Vault on VMware

Other Deployment Recommendations

General VMware Recommendations
Generally it is recommended that any unnecessary programs or graphic effects (such as screen savers and
Window animations) and disable in virtual machines.

The following recommendations can be made regarding VMware servers:

 Install the latest version of VMware Tools in the guest operating system. Make sure to update VMware
Tools after each ESXi upgrade. Installing VMware Tools in Windows guests updates the BusLogic SCSI
driver included with the guest operating system to the VMware-supplied driver. The VMware driver has
optimizations that guest-supplied Windows drivers do not
 Schedule backups and virus scanning programs in virtual machines to run at off-peak hours. Avoid
scheduling them to run simultaneously in multiple virtual machines on the same ESXi host. For
workloads such as backups and virus scanning, where the load is predictable, this is easily achieved
by scheduling the jobs appropriately
 For the most accurate timekeeping, consider configuring your guest operating system to use NTP,
Windows Time Service or the VMware Tools time-synchronization option. On any particular machine it
is recommended that either the VMware Tools time-synchronization option is used or another
timekeeping utility, but not both
 Unused or unnecessary virtual hardware devices can impact performance and should be disabled. For
example, Windows guest operating systems poll optical drives (that is, CD or DVD drives) quite
frequently. When virtual machines are configured to use a physical drive, and multiple guest operating
systems simultaneously try to access that drive, performance could suffer. This can be reduced by
configuring the virtual machines to use ISO images instead of physical drives, and can be avoided
entirely by disabling optical drives in virtual machines when the devices are not needed
 ESXi 5.0 introduces virtual hardware version 8. By creating virtual machines using this hardware
version, or upgrading existing virtual machines to this version, a number of additional capabilities
become available. Some of these, such as support for virtual machines with up to 1TB of RAM and up
to 32 vCPUs, support for virtual NUMA, and support for 3D graphics, can improve performance for
some workloads

The following general recommendations apply to Enterprise Vault index locations:

 If indexes are stored on NetApp devices, and possibly other NAS systems, opportunistic locking must
be turned off for volumes that contain indexes
 Disable Windows file indexing on the drives that contain Enterprise Vault indexes.

Disable Anti-Virus for Index Locations

Ensure that there is no Anti-Virus scanning the index locations, real-time or scheduled. Failing to exclude the
indexes from scanning can cause index corruption and will result in reduced server performance. A new
requirement in EV10 is to exclude the Indexing meta-data folder, default location “<install
path>\EVIndexing\data\indexmetadata”.

18
Enterprise Vault Best Practice Guide – Implementing Enterprise Vault on VMware

Pagefile Size
The Pagefile should conform to Microsoft’s recommendation of 1.5 times the amount of memory. It is
recommended that the page file is placed on a different partition and different physical hard disk drive from the
system partition so that Windows can handle multiple I/O requests more efficiently.

Additionally, if you set the “Initial size” of the Pagefile to the same as the “Maximum size” it will prevent
unnecessary fragmentation file. An un-fragmented paging file leads to faster virtual memory access.

Index Volume Fragmentation

The index files quickly become fragmented on disk, even if there is a large volume of free storage capacity.
This file fragmentation can cause severe performance problems which need to be managed on any index
storage device. Either an automated background file defragmentation product or scheduled device
defragmentation must be employed.

19
Enterprise Vault Best Practice Guide – Implementing Enterprise Vault on VMware

Ongoing Maintenance
Performance Monitoring
When measuring performance from within virtual machines any time-based performance counter is likely to be
inaccurate, especially when the processor is overcommitted. Generally it is safe to assume the results are no
more than 10% in error if CPU utilization stays below 80%.

To monitor the CPU usage on the host, use vSphere esxtop or resxtop. To interpret the esxtop data:

 If the load average on the first line of the esxtop CPU panel is equal to or greater than 1, this indicates
that the system is overloaded
 The usage percentage for the physical CPUs on the PCPU line can be another indication of a possibly
overloaded condition. In general, 80% usage is a reasonable ceiling and 90% should be a warning that
the CPUs are approaching an overloaded condition.

Enterprise Vault administrators should pay particular attention to the following CPU performance
metrics:

Esxtop Metric Description Implication

%RDY The percentage of time a A high %RDY time (use 20% as a starting point)
vCPU in a run queue is may indicate the virtual machine is under
waiting for the CPU resource contention. Monitor Enterprise Vault
scheduler to let it run on a performance, if the archiving rates are as
physical CPU. expected a higher threshold may be tolerated.

%MLMTD Percentage of time a vCPU A high %MLMTD time may indicate a CPU limit
was ready to run but was is holding the virtual machine in a ready to run
deliberately not scheduled state. If the application is running slow consider
due to CPU limits. increasing or removing the CPU limit.
%CSTP Percentage of time a vCPU A high %CSTP time usually indicates that
spent in read, co- vCPUs are not being used in a balanced way –
descheduled state (only evaluate the necessity for multiple vCPUs in
applicable for virtual smaller environments.
machines with more than one
vCPU).

20
Enterprise Vault Best Practice Guide – Implementing Enterprise Vault on VMware

Windows Performance Counters

The following counters may be useful to monitor within the virtual machine:
Objects and Counters Description
Processor

% Processor Time Shows processor usage over a period of time. If this counter is
consistently too high it is likely that the system performance will be
impacted. You can measure the utilization on each processor to achive
balanced performance between cores.
Disk
Avg. Disk Queue Length Shows the average number of read and write requests that were queued
for the selected disk during the sample interval. A bigger disk queue
length may not be a problem as long as disk reads/writes are not
suffering and the system is working in a steady state without expanding
queuing.
Avg. Disk Read Queue The average number of read requests that are queued.
Length
Avg. Disk Write Queue The average number of write requests that are queued.
Length
Disk Reads/sec The number of reads to disk per second.
Disk Wites/sec The number of writes to disk per second.
Memory
Available Mbytes Shows the amount of physical memory available for allocation.
Insufficient memory is likely to cause excessive use of the page file and
an increase in the number of page faults per second.
Cache Faults/sec Shows the rate at which faults occur when a page is sought in the file
system cache and is not found. This may be a soft fault, when the page is
found in memory or a hard fault when the page is on disk.
The use of cache for read and write operations can have a significant
impact on server performance. Monitor for increased cache failures,
indicated by a reduction in the Async Fast Reads/sec or Read
Aheads/sec.
Pages/sec Shows the rate at which pages are read from or written to disk to resolve
hard page faults. If this rises, it indicates system-wire performance
problems.

Objects and Counters Description

Paging File

% Used and % Used Peak The server paging file holds “virtual” memory addresses on disk. Page
faults occur when a process has to stop and wait while required “virtual”
resources are retrieved from disk into memory. These are more frequent
if the physical memory is inadequate.
Network
Total Bytes/sec The rate at which data is sent and received on the network interface. If
the rate is over 50% capacity you should investigate for issues. To
troubleshoot monitor Bytes Received/sec and Bytes Sent/sec.
Avg. Disk Read Queue The average number of read requests that are queued.
Length
Avg. Disk Write Queue The average number of write requests that are queued.
Length
Disk Reads/sec The number of reads to disk per second.
Disk Wites/sec The number of writes to disk per second.

21
About Veritas
Veritas Technologies LLC enables
organizations to harness the power
of their information, with solutions
designed to serve the world’s largest
and most complex heterogeneous
environments. Veritas works with
86 percent of Fortune 500 companies
today, improving data availability and
revealing insights to drive competitive
advantage. More information is available
at www.veritas.com.

© 2016 Veritas Technologies

LLC. All rights reserved.
Veritas and the Veritas Logo
are trademarks or registered
trademarks of Veritas
Technologies LLC or its
affiliates in the U.S. and other
countries. Other names may be
trademarks of their respective
owners.
This document is provided for
informational purposes only
and is not intended as
advertising. All warranties
relating to the information in
this document, either express
or implied, are disclaimed to the
For specific country offices and Veritas World Headquarters maximum extent allowed by
contact numbers, please visit 500 East Middlefield Road law. The information in this
our Web site: Mountain View, CA 94043 USA document is subject to change
www.veritas.com +1 (650) 933 1000 without notice.