v1.

5 | March 2021 | BP-2105

BEST PRACTICES

Linux on Nutanix AHV

Copyright
Copyright 2021 Nutanix, Inc.
Nutanix, Inc.
1740 Technology Drive, Suite 150
San Jose, CA 95110
All rights reserved. This product is protected by U.S. and international copyright
and intellectual property laws. Nutanix and the Nutanix logo are registered
trademarks of Nutanix, Inc. in the United States and/or other jurisdictions.
All other brand and product names mentioned herein are for identification
purposes only and may be trademarks of their respective holders.
 Linux on Nutanix AHV

Contents

1. Executive Summary.......................................................................................... 5

2. Introduction.........................................................................................................6
Audience.............................................................................................................................................................. 6
Purpose................................................................................................................................................................ 6

3. Nutanix Enterprise Cloud Overview..........................................................8

Nutanix HCI Architecture............................................................................................................................. 9

4. Nutanix AHV Best Practices........................................................................ 11

Nutanix Model Configurations....................................................................................................................11
Nutanix AHV Cluster Networking.............................................................................................................11
Nutanix Distributed Storage Fabric........................................................................................................12

5. Linux Operating System Kernel Settings...............................................13

6. Linux Networking............................................................................................ 15

7. Nutanix Volumes............................................................................................. 16
Jumbo Frames.................................................................................................................................................16
iSCSI Settings...................................................................................................................................................16

8. Configuration for Volumes (vDisks)....................................................... 26

Nutanix Native vDisks..................................................................................................................................27
Nutanix Volume Groups............................................................................................................................. 27

9. Logical Volume Manager (LVM) Configuration..................................29

File System Mount Options for ext4 and XFS..................................................................................30

10. Linux Disk Device Settings........................................................................31

11. Conclusion.........................................................................................................33

Appendix....................................................................................................................... 34
References........................................................................................................................................................34
About Nutanix................................................................................................................................................ 34

List of Figures..............................................................................................................................................................35

List of Tables............................................................................................................................................................... 36
 Linux on Nutanix AHV

1. Executive Summary
The Nutanix enterprise cloud OS provides a complete software-defined
datacenter infrastructure solution for applications running on Linux, eliminating
the complexities and inefficiencies of traditional multitier datacenter
environments. Whether you are virtualizing critical tier-1 applications or running
them on bare metal, Nutanix solutions bring the predictable performance,
availability, scalability, and cost benefits of web-scale architecture to your Linux
applications.
Nutanix provides the freedom to choose the hypervisor you want for your
applications, including Nutanix AHV, VMware vSphere, and Microsoft Hyper-
V. Solutions built on the Nutanix enterprise cloud deliver the performance
required for business-critical applications, while powerful self-healing, data
protection, and disaster recovery capabilities keep your applications running
and your vital data well protected. Nutanix provides near-instantaneous local
and remote data protection using snapshots. You can use these snapshots
for offloading database backups to tape and disk or WORM (write once,
read many) for offsite retention. Nutanix also enables one-click cloning, so
administrators can easily and quickly apply these snapshots to refresh a test or
development database from production.
This best practices guide collects recommended Nutanix AHV cluster settings
and Linux OS settings, which include those for Oracle Linux, Red Hat Enterprise
Linux, and CentOS. It covers vDisk (LUN) configuration and settings, Logical
Volume Manager (LVM) configuration, file system settings for ext4 and XFS,
and kernel parameters. Most of the recommendations in this guide also apply
generally to other hypervisor environments on Nutanix.

© 2021 Nutanix, Inc. All rights reserved | 5

 Linux on Nutanix AHV

2. Introduction

Audience
This best practices guide is part of the Nutanix Solutions Library. We wrote
it for architects and system administrators responsible for designing and
maintaining a robust Linux environment and its infrastructure. Readers should
already be familiar with AHV-based Nutanix infrastructure and the Linux OS.

Purpose
In this document, we cover the following topics:
• An overview of Nutanix.
• Nutanix AHV best practices.
• Linux OS kernel settings.
• Linux networking.
• Nutanix Volumes.
• Configuration for volumes (vDisks).
• LVM configuration.
• File system mount options for ext4 and XFS.
• Linux disk device settings.

Table 1: Document Version History

Version
Published Notes
Number
1.0 June 2018 Original publication.

© 2021 Nutanix, Inc. All rights reserved | 6

 Linux on Nutanix AHV

Version
Published Notes
Number
Updated the Nutanix AHV Cluster Networking
1.1 July 2018
and Nutanix Volumes sections.
Updated product information and the Linux Disk
1.2 January 2019
Device Settings section.
1.3 December 2019 Updated LVM creation instructions.
Updated product information and jumbo frames
1.4 June 2020
guidance.
1.5 March 2021 Updated load balancing recommendations.

© 2021 Nutanix, Inc. All rights reserved | 7

 Linux on Nutanix AHV

3. Nutanix Enterprise Cloud Overview

Nutanix delivers a web-scale, hyperconverged infrastructure solution purpose-
built for virtualization and both containerized and private cloud environments.
This solution brings the scale, resilience, and economic benefits of web-scale
architecture to the enterprise through the Nutanix enterprise cloud platform,
which combines the core HCI product families—Nutanix AOS and Nutanix Prism
management—along with other software products that automate, secure, and
back up cost-optimized infrastructure.
Available attributes of the Nutanix enterprise cloud OS stack include:
• Optimized for storage and compute resources.
• Machine learning to plan for and adapt to changing conditions automatically.
• Intrinsic security features and functions for data protection and cyberthreat
defense.
• Self-healing to tolerate and adjust to component failures.
• API-based automation and rich analytics.
• Simplified one-click upgrades and software life cycle management.
• Native file services for user and application data.
• Native backup and disaster recovery solutions.
• Powerful and feature-rich virtualization.
• Flexible virtual networking for visualization, automation, and security.
• Cloud automation and life cycle management.
Nutanix provides services and can be broken down into three main
components: an HCI-based distributed storage fabric, management and
operational intelligence from Prism, and AHV virtualization. Nutanix Prism
furnishes one-click infrastructure management for virtual environments running
on AOS. AOS is hypervisor agnostic, supporting two third-party hypervisors

© 2021 Nutanix, Inc. All rights reserved | 8

 Linux on Nutanix AHV

—VMware ESXi and Microsoft Hyper-V—in addition to the native Nutanix

hypervisor, AHV.

Figure 1: Nutanix Enterprise Cloud OS Stack

Nutanix HCI Architecture

Nutanix does not rely on traditional SAN or network-attached storage (NAS)
or expensive storage network interconnects. It combines highly dense storage
and server compute (CPU and RAM) into a single platform building block. Each
building block delivers a unified, scale-out, shared-nothing architecture with no
single points of failure.
The Nutanix solution requires no SAN constructs, such as LUNs, RAID groups,
or expensive storage switches. All storage management is VM-centric, and I/
O is optimized at the VM virtual disk level. The software solution runs on nodes
from a variety of manufacturers that are either entirely solid-state storage
with NVMe for optimal performance or a hybrid combination of SSD and HDD
storage that provides a combination of performance and additional capacity.
The storage fabric automatically tiers data across the cluster to different
classes of storage devices using intelligent data placement algorithms. For best

© 2021 Nutanix, Inc. All rights reserved | 9

 Linux on Nutanix AHV

performance, algorithms make sure the most frequently used data is available in
memory or in flash on the node local to the VM.
To learn more about the Nutanix enterprise cloud, visit the Nutanix Bible and
Nutanix.com.

© 2021 Nutanix, Inc. All rights reserved | 10

 Linux on Nutanix AHV

4. Nutanix AHV Best Practices

This section discusses how to select Nutanix nodes that have the CPU clock
speed you need for optimal performance. We also discuss networking and
storage best practices for a Nutanix cluster running on AHV, as proper
configuration eliminates infrastructure performance bottlenecks. For more
detailed information on AHV, please review our AHV best practices guide.

Nutanix Model Configurations

For a database workload, select Nutanix cluster nodes with a higher clock
speed and fewer cores. Higher clock speed means more I/O and lower latency,
while having fewer cores reduces licensing costs. We recommend selecting
nodes with at least 2.6 GHz of CPU.

Nutanix AHV Cluster Networking

When you set up a Nutanix AHV cluster, use only the 10 GbE, 25 GbE, or 40
GbE network interfaces for the bond0. Set the balance mode for br0-up to
balance-slb, with the next rebalance at 30 seconds.
Log on to a CVM and issue the following commands:
• Add the 10 GbE NICs to bond0:
nutanix@CVM:~$ allssh manage_ovs --bridge_name br0 --bond_name br0-up --
interfaces 10g update_uplinks

• Set the bond balance mode to balance-slb:

nutanix@CVM:~$ allssh ssh [email protected] "ovs-vsctl set port br0-up
bond_mode=balance-slb"

• Set next rebalance to 30 seconds:

nutanix@CVM:~$ allssh ssh [email protected] "ovs-vsctl set port br0-up
other_config:bond-rebalance-interval=30000"

© 2021 Nutanix, Inc. All rights reserved | 11

 Linux on Nutanix AHV

Nutanix Distributed Storage Fabric

Create a storage container with a replication factor of at least 2 for data
redundancy. This setting guarantees that the cluster doesn’t lose any data in
the event of a single-node failure. For more efficient space usage, enable inline
compression, deduplication, or erasure coding on the container.
Note: Don’t enable deduplication on the container if you run any database workload.

The following table details the recommended Nutanix storage configuration.

Table 2: Recommended Nutanix Storage Configuration

Nutanix Value Rationale

Required for replication factor
Number of Nutanix nodes Minimum 3
2.
Nutanix storage pool 1 Standard AHV practice.
Standard practice for both VM
Container 1
and database data.
Standard practice for space
Compression Enable
savings.
Not recommended for
databases but can be
Erasure coding Disable
enabled if usage capacity is a
constraint.
Not recommended for
Deduplication Disable
databases.

© 2021 Nutanix, Inc. All rights reserved | 12

 Linux on Nutanix AHV

5. Linux Operating System Kernel

Settings
The following table outlines the recommended OS kernel settings for a Linux
VM. Specific applications or databases may require additional settings for
optimal performance, so consult their documentation. You can find the
following settings in the documentation for /proc/sys/vm/*.

Table 3: Kernel Settings

Settings Value Purpose

vm.overcommit_memory 1 Disables memory overcommit handling.
As a percentage. Contains the number
of pages at which the background
vm.dirty_background_ratio 5
kernel flusher threads start writing out
dirty data.
As a percentage. Contains the number
of pages at which a process that is
vm.dirty_ratio 15
generating disk writes starts writing out
dirty data itself.
This tunable defines when dirty data is
vm.dirty_expire_centisecs 500 old enough to be eligible for writeout
by the kernel flusher threads.
The kernel flusher threads periodically
wake up and write old data out to disk.
vm.dirty_writeback_centisecs 100
This tunable expresses the interval
between those wakeups.

© 2021 Nutanix, Inc. All rights reserved | 13

 Linux on Nutanix AHV

Settings Value Purpose

This tunable defines how aggressively
the kernel swaps memory pages. A
value of 0 instructs the kernel not to
vm.swappiness 0
initiate swap until the amount of free
and file-backed pages is less than the
high-water mark in a zone.

© 2021 Nutanix, Inc. All rights reserved | 14

 Linux on Nutanix AHV

6. Linux Networking
When you create VMs for applications or databases, Nutanix highly
recommends that you use at least 10 GbE on the Nutanix cluster. If you require
high bandwidth and low latency, Nutanix recommends using multiple interfaces
(10 GbE, 25 GbE, or 40 GbE) in a bonded fashion, if your network switches
support doing so.

© 2021 Nutanix, Inc. All rights reserved | 15

 Linux on Nutanix AHV

7. Nutanix Volumes
Nutanix Volumes allows bare-metal servers or VMs to access vDisks in a
Nutanix volume group (VG) natively on AHV or through iSCSI. For client iSCSI
access, the Nutanix cluster provides a single data services IP address. If you
require high bandwidth and low latency, you can use volume groups with load
balancing (VGLB) on AHV or Nutanix Volumes iSCSI access to achieve the
performance you need. For more information about Volumes, see the Nutanix
Volumes section of the Prism Web Console Guide.

Jumbo Frames
The Nutanix CVM uses the standard Ethernet MTU (maximum transmission
unit) of 1,500 bytes for all the network interfaces by default. The standard
1,500 byte MTU delivers excellent performance and stability. Nutanix does not
support configuring the MTU on a CVM's network interfaces to higher values.
You can enable jumbo frames (MTU of 9,000 bytes) on the physical network
interfaces of AHV, ESXi, or Hyper-V hosts and user VMs if the applications on
your user VMs require them. If you choose to use jumbo frames on hypervisor
hosts, be sure to enable them end to end in the desired network and consider
both the physical and virtual network infrastructure impacted by the change.

iSCSI Settings
If you use Nutanix Volumes, configure the following iSCSI settings on the guest
OS in the /etc/iscsi/iscsid.conf file and restart the iscsid process.
node.session.timeo.replacement_timeout = 120
node.conn[0].timeo.noop_out_interval = 5
node.conn[0].timeo.noop_out_timeout = 10
node.session.cmds_max = 2048
node.session.queue_depth = 1024
node.session.iscsi.ImmediateData = Yes

© 2021 Nutanix, Inc. All rights reserved | 16

 Linux on Nutanix AHV

node.session.iscsi.FirstBurstLength = 1048576
node.session.iscsi.MaxBurstLength = 16776192
node.conn[0].iscsi.MaxRecvDataSegmentLength = 1048576
discovery.sendtargets.iscsi.MaxRecvDataSegmentLength = 1048576

When you use Nutanix Volumes, you can configure single path (single
iSCSI NIC) or multipath (two iSCSI NICs). For bare-metal configurations, we
recommend using multipath to avoid a single point of failure in the client NIC. If
you use Volumes with a VM, you can only configure with a single NIC.

Set Up a Single iSCSI NIC

If you have multipath configured on a VM with a single NIC (which can occur
if you use an organization-standard VM image or migrate a VM from a system
that requires multipath) you should either disable multipath or increase the
fast_io_fail_tmo. If you leave multipath enabled, increase fast_io_fail_tmo to
120 seconds to allow clean cluster failover on the client VM:
# cat /etc/multipath.confdefaults { fast_io_fail_tmo 120 }
<…snip…>

What follows is an example of how to set up a single NIC for iSCSI on a VM.
After you create the VM with a NIC dedicated for iSCSI traffic with an IP
address, follow this procedure. Assume the iSCSI NIC is ETH1.
• Create an iface:
[root@localhost ~]# iscsiadm -m iface -I iface1 --op=new
[root@localhost ~]# iscsiadm -m iface -I iface1 --op=update -n
iface.net_ifacename -v eth1

• Create a VG.

© 2021 Nutanix, Inc. All rights reserved | 17

 Linux on Nutanix AHV

Figure 2: Create a VG

• Add disks to the VG.

© 2021 Nutanix, Inc. All rights reserved | 18

 Linux on Nutanix AHV

Figure 3: Choose a Storage Container and Size

• Save the VG.

© 2021 Nutanix, Inc. All rights reserved | 19

 Linux on Nutanix AHV

Figure 4: Save the VG

• Get the initiator IQN on the client.

Figure 5: Initiator IQN

• Update the VG. This step allows you to specify the initiator IQN. Click Update.

© 2021 Nutanix, Inc. All rights reserved | 20

 Linux on Nutanix AHV

Figure 6: Update the VG

• Click Add New Client.

Figure 7: Add a New Client

• Enter the client IQN and click Add.

© 2021 Nutanix, Inc. All rights reserved | 21

 Linux on Nutanix AHV

Figure 8: Client IQN

• Click Save.

© 2021 Nutanix, Inc. All rights reserved | 22

 Linux on Nutanix AHV

Figure 9: Save Changes to VG

• Run the iSCSI discover and login commands using the Nutanix external data
service IP address:
[root@localhost ~]# iscsiadm -m discovery -t st -p <DSIP>
[root@localhost ~]# iscsiadm -m node - -login

Set Up Multiple iSCSI NICs

If you use multiple NICs for iSCSI traffic on your bare-metal servers, you need
to configure dm-multipath. After you set up your iSCSI NICs using the above
steps, follow this procedure to set up multipath on your bare-metal servers.
• Edit the /etc/multipath.conf file. If this file doesn’t exist, create a new one.
Enter the following in this file:
devices {

© 2021 Nutanix, Inc. All rights reserved | 23

 Linux on Nutanix AHV

device {
vendor "NUTANIX"
product "Server"
path_grouping_policy multibus
path_selector "round-robin 0"
features "1 queue_if_no_path"
path_checker tur
rr_min_io_rq 20
rr_weight priorities
failback immediate
}
}
multipaths {
multipath {
wwid 1NUTANIX_NFS_1_0_4424_ed6ae886_4ecc_473d_aadc_765f706bca13

alias asm1
}
}

• Enter the vDisk Universally Unique Identifier (UUID) and an alias name for it.
Add all vDisks to this file. The above shows a sample of a single vDisk only.
• Start your multipathd service:
[root@localhost ~]# systemctl stop multipathd.service
[root@localhost ~]# systemctl start multipathd.service
[root@localhost ~]# multipath -r -p multibus
[root@localhost ~]# for a in $(multipath -ll | grep "NUTANIX" | awk '{print
$1}') ; do dmsetup message $a 1 "queue_if_no_path" ; done

Note: The command for loop ensures that each vDisk has a feature of 1 queue_if_no_path. You
must verify that each vDisk has this feature by running the multipath -ll command.

The for loop command and the multipath -r -p multibus don’t survive a server
reboot. To make sure they’re set on reboot, add the commands to a shell script
and have it run automatically. You can put this shell script in the /etc/rc.local

© 2021 Nutanix, Inc. All rights reserved | 24

 Linux on Nutanix AHV

file. New Linux distributions may not have /etc/rc.local configured to run, so
you may have to manually configure it.

© 2021 Nutanix, Inc. All rights reserved | 25

 Linux on Nutanix AHV

8. Configuration for Volumes (vDisks)

In a traditional three-tier architecture, application and database administrators
often work with storage administrators to create a custom storage design to
suit their environment, which includes designating the RAID level and block
size for different types of files, such as database tablespaces or log files.
This process can become cumbersome to manage, especially if there are
multiple databases. Nutanix eliminates the problems associated with choosing
the optimal RAID and block size. Once you create a VG with the required
number of vDisks for your database or application, you’re ready to deploy. The
following table outlines the recommended minimum disk layout for a database
such as Oracle. For other workloads please refer to the specific application’s
best practices guide.

Table 4: Nutanix Volume Configuration

Number
Purpose Comment
of vDisks
Can be used with LVM or Standard
1 Boot disk
partition
Database datafiles / control Can be used with Oracle ASM or
8
files / redo log files Filesystem with LVM
Can be used with Oracle ASM or
4 Database archive log files
Filesystem with LVM
Can be used with Oracle ASM or
4 Database RMAN backup files
Filesystem with LVM

Tip: Nutanix recommends that you use multiple vDisks with OS-level striping for any applications
that require high performance I/O.

There are two ways to create vDisks on Nutanix for your VMs:
1. Nutanix native vDisks.

© 2021 Nutanix, Inc. All rights reserved | 26

 Linux on Nutanix AHV

2. Nutanix volume groups.

Nutanix Native vDisks

The Nutanix native vDisk option simplifies VM administration because it doesn’t
require Nutanix VGs. When you create a VM using Prism, add more vDisks to
the VM for the application or database the same way you add a vDisk to the
VM for the boot disk. If you have an application or database that doesn’t have
intensive I/O requirements, native vDisks are the best option.

Nutanix Volume Groups

Alternatively, you can attach Nutanix VGs, which are collections of vDisks, to
the VMs. VGs enable you to separate the data vDisks from the VM’s boot vDisk.
This separation allows you to create a protection domain that consists only of
the data vDisks for snapshots and cloning. In addition, VGs let you configure
a cluster with shared-disk access across multiple VMs. Supported clustering
applications include Oracle Real Application Cluster (RAC), IBM Spectrum Scale
(formerly known as GPFS), Veritas InfoScale, and Red Hat Clustering. To attach
the VG to multiple VMs when you use the aCLI, create it with a shared=true
attribute. If you use Prism, answer yes when you attach the VG to a second VM.
There are two ways to use Nutanix VGs: default VG and VGLB.

Default Volume Group

This type of VG provides the best data locality because it doesn’t load-balance
vDisks in a Nutanix cluster, which means all vDisks in default VGs have a single
CVM providing their I/O. For example, in a four-node Nutanix cluster that
includes a VG with eight vDisks attached to a VM, a single CVM owns all the
vDisks and all I/O to the eight vDisks goes through this CVM.

Volume Group with Load Balancing (VGLB)

Note: This feature is available with AHV (AOS 5.6 and later). If you use ESXi or Hyper-V, you can
use Nutanix Volumes with iSCSI to achieve this functionality.

VGLBs distribute ownership of the vDisks across all the CVMs in the cluster,
which can provide better performance. Use the VGLB feature if your

© 2021 Nutanix, Inc. All rights reserved | 27

 Linux on Nutanix AHV

applications or databases require better performance than the default VG

provides. Run the following command in the Nutanix aCLI to create a VGLB.

Figure 10: Create a VGLB

© 2021 Nutanix, Inc. All rights reserved | 28

 Linux on Nutanix AHV

9. Logical Volume Manager (LVM)

Configuration
This section discusses Linux Logical Volume Manager (LVM) best practices.
When you create a file system for any applications or databases, use LVM
striping across all vDisks for a specific mount point. The following example
shows how to create an LVM VG with eight vDisks for a database datafiles
mount point.
• Create the physical volumes.
[root@localhost ~]# pvcreate /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/
sdh /dev/sdi /dev/sdj
Physical volume "/dev/sdc" successfully created
Physical volume "/dev/sdd" successfully created
Physical volume "/dev/sde" successfully created
Physical volume "/dev/sdf" successfully created
Physical volume "/dev/sdg" successfully created
Physical volume "/dev/sdh" successfully created
Physical volume "/dev/sdi" successfully created
Physical volume "/dev/sdj" successfully created

• Create a VG.
[root@localhost ~]# vgcreate vgdata /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/
sdg /dev/sdh /dev/sdi /dev/sdj
Volume group "vgdata" successfully created

• Create a logical volume. Make sure to use the -i (lowercase i) option, which
specifies the number of vDisks to stripe across, and the -I (uppercase i)
option, which specifies the stripe size. The recommended stripe size is 512
KB.
[root@localhost ~]# lvcreate -l 383994 -i 8 -I 512 -n vol1 vgdata
Logical volume "vol1" created.

• Create the file system. In this example, we are creating an ext4 file system.

© 2021 Nutanix, Inc. All rights reserved | 29

 Linux on Nutanix AHV

[root@localhost ~]# mkfs.ext4 /dev/vgdata/vol1

• Mount the ext4 file system. If you are creating an XFS file system, please
refer to the Mount Options table.
[root@localhost ~]# mount /dev/vgdata/vol1 /u01/oradata -o
noatime,nodiratime,discard,barrier=0

Note: To give your file systems a friendly name, try xfs_admin for xfs or e2label for ext4. Use the
LABEL= option in the /etc/fstab file for easy management.

File System Mount Options for ext4 and XFS

The following table lists the recommended mount options for ext4 and XFS file
systems.

Table 5: Mount Options for ext4 and XFS File Systems

File System Type Mount Options

ext4 noatime,nodiratime,discard,barrier=0
XFS noatime,nodiratime,discard,nobarrier,logbufs=8

© 2021 Nutanix, Inc. All rights reserved | 30

 Linux on Nutanix AHV

10. Linux Disk Device Settings

To optimize performance for your Linux VMs, change the following disk
parameters from their default settings:
• Change the max_sectors_kb to 1024 (the default is 512). For newer Linux
distributions, the default may already be set to 1024. Change the disk timeout
to 60 (default is 30).
[root@localhost ~]# lsscsi | grep NUTANIX | awk '{print $NF}' | awk -F"/" '{print
$NF}' | grep -v "-" | while read LUN
do
echo 1024 > /sys/block/${LUN}/queue/max_sectors_kb
done
[root@localhost ~]# lsscsi | grep NUTANIX | awk '{print $NF}' | awk -F"/" '{print
$NF}' | grep -v "-" | while read LUN
do
echo 60 > /sys/block/${LUN}/device/timeout
done

• Either put this command in the /etc/rc.local file, so that it runs the next time
the server reboots, or use UDEV rules. For UDEV rules, you can create a file
with the following content under the /etc/udev/rules.d directory:
ACTION=="add", SUBSYSTEMS=="scsi", ATTRS{vendor}=="NUTANIX ",
ATTRS{model}=="VDISK", RUN+="/bin/sh -c 'echo 1024 >/sys$DEVPATH/queue/
max_sectors_kb'"
ACTION=="add", SUBSYSTEMS=="scsi", ATTRS{vendor}=="NUTANIX ",
ATTRS{model}=="VDISK", RUN+="/bin/sh -c 'echo 60 >/sys$DEVPATH/device/timeout'"

• Change the disk io_scheduler to noop. The default is deadline.

[root@localhost ~]# lsscsi | grep NUTANIX | awk '{print $NF}' | awk -F"/" '{print
$NF}' | grep -v "-" | while read LUN
do
echo noop > /sys/block/${LUN}/queue/scheduler
done

© 2021 Nutanix, Inc. All rights reserved | 31

 Linux on Nutanix AHV

If you put this command in the /etc/rc.local file, it runs the next time the server
reboots. Alternatively, you can put it in the grub configuration file. Follow these
steps for GRUB2 configuration:
• Edit the /etc/default/grub file, add “elevator=noop” to the
GRUB_CMDLINE_LINUX line, and save the file.

Figure 11: Add elevator=noop

• Disable transparent_hugepage.
• In the same edit window, add “transparent_hugepage=never” to the end of
the elevator=noop line and save the file.
• If the Linux kernel supports the blk_mq (block multiqueue) option, add the
parameter “scsi_mod.use_blk_mq=1” to enable blk_mq and remove the
elevator=noop option.

Figure 12: Add scsi_mod.use_blk_mq=1

• Run grub2-mkconfig to generate a new grub file, then reboot.

[root@localhost ~]# grub2-mkconfig –o /boot/grub2/grub.cfg

© 2021 Nutanix, Inc. All rights reserved | 32

 Linux on Nutanix AHV

11. Conclusion
This best practices guide outlines our recommended settings for a Linux VM
running on Nutanix with AHV. With the proper settings for Nutanix cluster
networking, as well as for volume, LVM, kernel configuration, and proper
file system mount options, you can maximize Linux performance for any
application. The Nutanix enterprise cloud OS removes the complexity of
constantly managing and optimizing the underlying compute, network, and
storage architecture, so you can focus on higher-value tasks for your business.
For feedback or questions, please contact us using the Nutanix NEXT
Community forums.

© 2021 Nutanix, Inc. All rights reserved | 33

 Linux on Nutanix AHV

Appendix

References
1. Nutanix Volumes best practices guide
2. AHV best practices guide
3. Documentation for /proc/sys/vm/*
4. Prism Web Console Guide, Nutanix Volumes
5. Nutanix Volumes Guide

About Nutanix
Nutanix makes infrastructure invisible, elevating IT to focus on the applications
and services that power their business. The Nutanix enterprise cloud software
leverages web-scale engineering and consumer-grade design to natively
converge compute, virtualization, and storage into a resilient, software-
defined solution with rich machine intelligence. The result is predictable
performance, cloud-like infrastructure consumption, robust security, and
seamless application mobility for a broad range of enterprise applications.
Learn more at www.nutanix.com or follow us on Twitter @nutanix.

© 2021 Nutanix, Inc. All rights reserved | 34

 Linux on Nutanix AHV

List of Figures
Figure 1: Nutanix Enterprise Cloud OS Stack................................................................................................... 9

Figure 2: Create a VG............................................................................................................................................... 18

Figure 3: Choose a Storage Container and Size............................................................................................ 19

Figure 4: Save the VG............................................................................................................................................. 20

Figure 5: Initiator IQN.............................................................................................................................................. 20

Figure 6: Update the VG..........................................................................................................................................21

Figure 7: Add a New Client.................................................................................................................................... 21

Figure 8: Client IQN.................................................................................................................................................. 22

Figure 9: Save Changes to VG............................................................................................................................. 23

Figure 10: Create a VGLB....................................................................................................................................... 28

Figure 11: Add elevator=noop................................................................................................................................32

Figure 12: Add scsi_mod.use_blk_mq=1............................................................................................................32

List of Tables
Table 1: Document Version History.......................................................................................................................6

Table 2: Recommended Nutanix Storage Configuration............................................................................ 12

Table 3: Kernel Settings........................................................................................................................................... 13

Table 4: Nutanix Volume Configuration........................................................................................................... 26

Table 5: Mount Options for ext4 and XFS File Systems........................................................................... 30