Scribd
Upload
30 views

Nutanix LCM Log Collection Utility

Nutanix LCM Log Collection Utility

Uploaded by

dean.huang
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
30 views

Nutanix LCM Log Collection Utility

Nutanix LCM Log Collection Utility

Uploaded by

dean.huang
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 4

LCM Log Collection Utility

Article # 000007288 Last modified on Jul 31st 2024

Summary: Overview and instructions on using LCM Log Collection Utility.

LCM

Description:
Purpose
LCM (Life Cycle Manager) Log Collection Utility provides an independent script designed to collect all
the important log files necessary to debug any LCM issue.

NCC log_collector bundle is significant in size and contains multiple files that are irrelevant to LCM. In
contrast, Log Collection Utility collects only the files that are relevant to LCM workflow thereby
reducing the log bundle size and speeding up the process of log collection.
The utility is available in LCM version 2.2 or later by default. It could also be used on earlier versions
of LCM.

Functionality

Collects LCM log files.


Detects if the node is stuck in Phoenix and collects kernel logs.
Collects logs for the services with which LCM interacts (Example: Foundation, Prism, etc.).
The utility state is independent of such of the LCM hence it is capable of collecting logs even if the
framework is down.
Parses LCM configurations from Zookeeper nodes.

Caveats
The script cannot pull logs from the node booted into Phoenix when the node's IP address is not
reachable over the network. In that case, apply the respective CVM (Controller VM) IP address to the
Phoenix instance using IPMI following KB 5346 before running the script. Once the network is
restored, /tmp/phoenix.log should also be collected from Phoenix for troubleshooting purposes.

Files and information collected

Node level information

LCM leader
Foundation version
LCM version
Hypervisor type/version
LCM configuration
/etc/nutanix/hardware_config.json
/etc/nutanix/factory_config.json

Logs from CVM

/home/nutanix/data/logs/genesis.out*
/home/nutanix/data/logs/lcm_ops.out*
/home/nutanix/data/logs/lcm_op.trace
/home/nutanix/data/logs/lcm_wget.log
/home/nutanix/data/logs/foundation*
/home/nutanix/data/logs/catalog.out*
/home/nutanix/data/logs/prism_gateway.log
/home/nutanix/data/logs/hera.out*
/home/nutanix/data/logs/ergon.out*

Command output from Phoenix

$ date
$ uptime
$ ifconfig
$ cat /proc/cmdline
$ /usr/sbin/smartctl -a /dev/sdX
$ dmesg -T > dmesg
$ journalctl -u sshd > journalctl_sshd

Solution:
Starting LCM-3.0:

Logbay serves as the primary tool for automated log collection in the event of an LCM failure.
However, for logs related to Phoenix-based failures or Prism Central or Mixed Clusters, will rely on the
LCM log collector.

These logs will be collected separately and later appended to the Logbay bundle.

To run the LCM log collector manually in LCM 3.0 or later:

nutanix@CVM$ /home/nutanix/cluster/bin/lcm/lcm_log_collector
For LCM 2.3.1 or later (automatic log collection)

LCM 2.3.1 automatically triggers the LCM log collector utility when any LCM operation fails.
Log bundle will be created in /home/nutanix/data/log_collector/.
For LCM 2.3.1.1 and later, LCM prints Log location along with the failure message as below. Log into
the CVM IP to fetch the LCM log bundle and upload it to the Support case for review.

Operation failed. Reason: LCM operation kLcmUpdateOperation failed on


phoenix, ip: [xx.xx.x.xx] due to Multistage upgrade encountered an
error: ('The read operation timed out',).
Logs have been collected and are available to download on xx.xx.x.xx at
/home/nutanix/data/log_collector/lcm_logs__xx.xx.x.xx__2020-05-05_08-48-
19.113101.tar.gz

For LCM 2.2.x or later (manual log collection)

It can be used to trigger a new log collection.

Run the following command from a CVM to begin log collection.

nutanix@CVM$ python /home/nutanix/cluster/bin/lcm/lcm_log_collector.py

Sometimes you might see the error message below:

nutanix@CVM$ python /home/nutanix/cluster/bin/lcm/lcm_log_collector.py


/usr/bin/python2.7: can't open file
'/home/nutanix/cluster/bin/lcm/lcm_log_collector.py': [Errno 2] No such
file or directory

If you get the above error, please look for the file python lcm_log_collector.py instead and run the
command shown below:

nutanix@CVM:~/cluster/bin/lcm$ python lcm_log_collector.py

Log bundle will be created in /home/nutanix directory on the node from which the script is executed,
with the name in the following format:

lcm_logs__{lcm_leader}__{time_stamp}.tar.gz
For LCM 2.2.x, log bundle will be created in /home/nutanix/data/logs/lcm_logs/.

For LCM 2.3 and later, the log bundle will be created in /home/nutanix/data/log_collector/.

For Uploading the logs


Once the logs are collected refer to KB-1294 for different methods to upload the LCM logs to the case.

You might also like