0% found this document useful (0 votes)
51 views103 pages

PowerStore 3.0 Administration - Troubleshooting - Participant Guide

The PowerStore 3.0 Administration - Troubleshooting guide provides essential troubleshooting techniques for PowerStore systems, focusing on hardware issues, CLI usage, and support material collection. It includes detailed information on using LEDs for hardware diagnostics, understanding alert severity levels, and gathering support materials. The document serves as a comprehensive resource for administrators to effectively manage and troubleshoot PowerStore systems.

Uploaded by

Salome Marimba
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
0% found this document useful (0 votes)
51 views103 pages

PowerStore 3.0 Administration - Troubleshooting - Participant Guide

The PowerStore 3.0 Administration - Troubleshooting guide provides essential troubleshooting techniques for PowerStore systems, focusing on hardware issues, CLI usage, and support material collection. It includes detailed information on using LEDs for hardware diagnostics, understanding alert severity levels, and gathering support materials. The document serves as a comprehensive resource for administrators to effectively manage and troubleshoot PowerStore systems.

Uploaded by

Salome Marimba
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 103

POWERSTORE 3.

0
ADMINISTRATION -
TROUBLESHOOTING

PARTICIPANT GUIDE

PARTICIPANT GUIDE

[email protected]
[email protected]
PowerStore 3.0 Administration - Troubleshooting

© Copyright 2022 Dell Inc. Page i


[email protected]
Table of Contents

PowerStore Basic Troubleshooting ...................................................................................... 2


Introduction .......................................................................................................................... 3

Troubleshoot Hardware Issues ................................................................................. 4


Troubleshoot Components Using LEDs ............................................................................... 5
Reference PowerStore Alerts for Troubleshooting ............................................................. 16
PowerStore Hardware Troubleshooting Key Points............................................................ 31

Troubleshoot Using PowerStore CLI...................................................................... 32


Use PowerStore CLI to Display System Information .......................................................... 33
Troubleshoot Issues Using PowerStore CLI ....................................................................... 40
Troubleshooting with PowerStore CLI Key Points .............................................................. 57

PowerStore Troubleshooting Tools ....................................................................... 58


Service Container .............................................................................................................. 59
NVMe NVRAM Failure Handling ........................................................................................ 78
PowerStore Troubleshooting Tools Key Points .................................................................. 81

Gather Support Materials ........................................................................................ 83


Gather Support Materials Using PowerStore Manager ....................................................... 84
Gather Support Materials Using Service Container ............................................................ 91
Data Collection Key Points................................................................................................. 95

Appendix ................................................................................................. 97

PowerStore 3.0 Administration - Troubleshooting

© Copyright 2022 Dell Inc.


Page ii [email protected]
Troubleshoot Hardware Issues

PowerStore 3.0 Administration - Troubleshooting

© Copyright 2022 Dell Inc. Page 1


[email protected]
Troubleshoot Hardware Issues

PowerStore Basic Troubleshooting

PowerStore 3.0 Administration - Troubleshooting

Page 2 © Copyright 2022 Dell Inc.


[email protected]
Troubleshoot Hardware Issues

Introduction

This course provides basic troubleshooting techniques in PowerStore, including:


 Using LEDs to troubleshoot and identify hardware
 Collecting logs and data from the system
 Collecting logs and data using CLI

PowerStore 3.0 Administration - Troubleshooting

© Copyright 2022 Dell Inc. Page 3


[email protected]
Troubleshoot Hardware Issues

Troubleshoot Hardware Issues

PowerStore 3.0 Administration - Troubleshooting

Page 4 © Copyright 2022 Dell Inc.


[email protected]
Troubleshoot Hardware Issues

Troubleshoot Components Using LEDs

Base Enclosure Front View LEDs

There are LEDs on the front of the base enclosure. The drives are in 2.5 inch (6.35
cm) carriers and are hot-swappable. Enable drive identification from the CLI or
PowerStore Manager to locate a specific drive.

To identify a particular appliance in the rack, select BLINK LED from Hardware >
APPLIANCES > Appliance > COMPONENTS > DRIVES in PowerStore Manager.
Once the button is activated, the button is relabeled to STOP BLINK LED.

Base enclosure power-on LED:


Blue: Power on (normal)
Amber: Fault
Blinking amber: Discovery state

Drive fault LED (amber) Drive ready/activity LED (blue)


Solid: Fault Solid: Power up
Blinking: Drive identification Blinking: Drive activity

Base Enclosure Front LEDs

Embedded I/O (Personality) Module LEDs

Module Fault

Embedded module fault

Embedded module fault LED

PowerStore 3.0 Administration - Troubleshooting

© Copyright 2022 Dell Inc. Page 5


[email protected]
Troubleshoot Hardware Issues

LED State Description

Embedded module Solid amber Embedded module


fault has faulted.

Off No fault has


occurred—normal
operation.

Ethernet Link

Ethernet port link

Ethernet port link LED

LED State Description

Ethernet port link Solid green Link established.

Off No link established.

Ethernet Activity

Ethernet port activity

Ethernet port activity LED

PowerStore 3.0 Administration - Troubleshooting

Page 6 © Copyright 2022 Dell Inc.


[email protected]
Troubleshoot Hardware Issues

LED State Description

Ethernet port Blinking amber Port activity


activity
Off No port activity

SAS Activity

SAS port/activity link

SAS port/activity link LEDs

LED State Description

SAS port/activity Solid blue SAS port link is up.


link
Off No link established.

Port link

Port link

Port link LEDs

LED State Description

PowerStore 3.0 Administration - Troubleshooting

© Copyright 2022 Dell Inc. Page 7


[email protected]
Troubleshoot Hardware Issues

Port link Solid green Link up with full speed.

Solid amber Link up with degraded


speed.

Off Link down.

Node Fault

Node fault

Node fault LED

LED State Description

Node fault Solid amber Fault has occurred.

Solid blue Node in Degraded


Mode.

Blinking amber or System is booting.


blue

Blue and amber System not initialized.


alternating (blue A management IP
for 3 s) address has not been
assigned.

Blue and amber Node in Service Mode.


alternating at 1 s
intervals

PowerStore 3.0 Administration - Troubleshooting

Page 8 © Copyright 2022 Dell Inc.


[email protected]
Troubleshoot Hardware Issues

Off No fault has occurred;


normal operation.

Node Power

Node power

Node fault LED

LED State Description

Node power Solid Green Node is on (main


power).

Blinking Green Node is initializing a


serial over LAN session
(Standby Mode).

Off Node is off.

Unsafe to Remove

Unsafe to remove

Unsafe to remove LED

PowerStore 3.0 Administration - Troubleshooting

© Copyright 2022 Dell Inc. Page 9


[email protected]
Troubleshoot Hardware Issues

LED State Description

Unsafe to Solid white Do not remove the node


remove or embedded module.
Improper removal could
cause data loss. Lights
when the other node is
not working.

Off Safe to remove the node


or embedded module
when the embedded
module has been
properly prepared.

Fibre Channel Module LEDs

Add I/O modules in pairs: One module in node A and one module in node B. Both
nodes must have the same type of I/O modules in the same slots.

The port link LEDs are below the ports. The power fault LED is on the right, below
the ejector handle.

To learn more, click each area that is highlighted in red.

Fibre channel module LEDs

1 2

1: Port link LEDs

PowerStore 3.0 Administration - Troubleshooting

Page 10 © Copyright 2022 Dell Inc.


[email protected]
Troubleshoot Hardware Issues

 Blue: Link up.


 Off: Link down.
 Unsupported SFP: Blink interval=1 s.

2: Power fault LED

 Green: I/O module turned on.


 Amber: I/O module power fault.
 Off: I/O module powered off.

LEDs on AC Power Supply

The image below shows the AC Power Supplies from the back.

 The top LED (~) is the AC Power LED.


 The middle LED (О)—DC Power Output LED—is not supported. DC power
supplies are covered on the next page.
 The bottom LED (∆) is the Power Supply/Cooling Module Fault LED.

The node includes a Lithium-ion (Li-ion) internal battery that powers the associated
node during a power event.

AC Power Supply LEDs

1: Power Supply/Cooling Module Fault LED

 Solid amber: Power supply or cooling module fault

PowerStore 3.0 Administration - Troubleshooting

© Copyright 2022 Dell Inc. Page 11


[email protected]
Troubleshoot Hardware Issues

 Blinking amber: Undervoltage protection (UVP) fault


 Off: No fault or power off

2: DC Power (output) LED: Not supported

3: AC Power LED (input)

 Solid green: AC power is on.


 Off: AC power is off. Verify source power.

LEDs on PowerStore 500T DC Power Supply

The PowerStore 500T has an AC model and a DC model. This page only applies to
PowerStore 500T DC models.

The image below shows the DC Power Supply from the back.

 The left LED ( ) is the DC Power Input LED.

 The middle LED ( ) is the DC Power Output LED.

 The bottom LED ( ) is the Power Supply/Cooling Module Fault LED.

DC powered models have DC battery backup module.

PowerStore 500T DC Power Supply LEDs

3 2 1

1: Power Supply/Cooling Module Fault LED

 Solid amber: Power supply fault. Check cable connection.

PowerStore 3.0 Administration - Troubleshooting

Page 12 © Copyright 2022 Dell Inc.


[email protected]
Troubleshoot Hardware Issues

 Blinking amber: Over temperature fault.


 Off: No fault.

2: DC Power (output) LED

 Green: Power supply is operating properly.


 Off: Power supply is not operating properly.

3: DC Power LED (input)

 Solid green: DC power is on.


 Off: DC power is off. Verify source power.

LEDs on Expansion Components

The enclosure power and fault LEDs on expansion enclosures are slightly different
from the LEDs on the base enclosure. The drives are in 2.5-inch carriers and are
hot-swappable.

Expansion enclosure LEDs

1 4

2 3

1: Expansion enclosure fault LED

 Amber: Fault
 Blue: No fault

2: Drive status/activity

 Blue: Powering and powered up


 Blue blinking: Drive activity

3: Drive fault

PowerStore 3.0 Administration - Troubleshooting

© Copyright 2022 Dell Inc. Page 13


[email protected]
Troubleshoot Hardware Issues

 Amber: Fault
 Off: No fault

4: Expansion enclosure power status LED

 Blue: Powering and powered up


 Off: Powered off

System Identification Tag

 The Dell Service Tag (DST) is a serialized label that allows Support to track
hardware in the field.
 The DST is a black pull-out tag between the drives in slots 16 and 17.
 The left side of the tag contains the service tag and the Express Service Code
information. The QRL code is on the right:

Service tab pull-out

PowerStore 3.0 Administration - Troubleshooting

Page 14 © Copyright 2022 Dell Inc.


[email protected]
Troubleshoot Hardware Issues

QRL pull-out

 Provide the service tag when contacting Support.


 To access the Dell support site, scan the QRL with the Dell Quick Resource
Locator mobile phoneapp or other QR reader.

PowerStore 3.0 Administration - Troubleshooting

© Copyright 2022 Dell Inc. Page 15


[email protected]
Troubleshoot Hardware Issues

Reference PowerStore Alerts for Troubleshooting

Cluster Alerts in PowerStore Manager

PowerStore generates alerts at four severity levels:

Icon Severity Indicates

Info An event has occurred that does not impact system functions. No action is required.

An error has occurred to be aware of but does not have a significant impact on the system. For example, a
Minor
component is working, but its performance is not optimum.

May have an impact on the system and should be remedied as soon as possible. For example, the last
Major
synchronization time for a resource does not match the time that its protection policy indicates.

An error has occurred that has a significant impact on the system and should be remedied immediately. For
Critical
example, a component is missing or has failed and recovery may not be possible.

PowerStore alert levels

Cluster Alerts Dashboard

Alerts about the overall cluster are visible on the Dashboard Overview tab or
under Monitoring > Alerts. Select alert headings to show only that type of alert as
shown here:

PowerStore 3.0 Administration - Troubleshooting

Page 16 © Copyright 2022 Dell Inc.


[email protected]
Troubleshoot Hardware Issues

PowerStore alert lists

Alert Details

On the Dashboard Overview tab, click an alert title to see its details and the
actions to take. Focus on critical alerts first, and then major alerts. Minor and Info
alerts do not necessarily require any action. On the alert details slide-out panel,
acknowledge that the issue which triggered the alert has been addressed.

Copy and paste the event code into the monitoring, events, or filters to find all
timestamps where this event has occurred for further troubleshooting. For example,
select 0x00a00301.

Important: Acknowledging an alert does not make the problem go


away, it makes the alert go away. Since some alerts do not require
action, the system may automatically clear those alerts.

Alert Information

General information about the alert, including:


1. Alert Description—a detailed description of why the alert was triggered.
2. Event Code—Unique hexadecimal code used by PowerStore OS to refer to this
event type

PowerStore 3.0 Administration - Troubleshooting

© Copyright 2022 Dell Inc. Page 17


[email protected]
Troubleshoot Hardware Issues

3. Resource Name—the name of the appliance or device resource reporting the


alert
4. Associated Events—number of and link to PowerStore OS Events that are
linked to this alert. Events are lower-level logs that trigger an alert when certain
thresholds are reached, or criticality flags are raised.
5. Impact—if applicable, additional information on the impact of the event on the
cluster
6. Repair Details—suggested steps to fix the system state reported in the alert
7. Support—links to additional resources such as the Dell support site, knowledge
base, or PowerStore diagnostics

PowerStore 3.0 Administration - Troubleshooting

Page 18 © Copyright 2022 Dell Inc.


[email protected]
Troubleshoot Hardware Issues

Alert details

PowerStore 3.0 Administration - Troubleshooting

© Copyright 2022 Dell Inc. Page 19


[email protected]
Troubleshoot Hardware Issues

Associated Events

Events that are related to the alert:


 Includes information about earlier alerts that are related to this issue, and how
the situation has changed.
 To organize a long list of related events, click column headings to sort, use the
filter, or modify the columns of information visible.

List of events associated with the alert

Service and Support

How to get help:


1. Next Steps—recommended steps to fixing the system state which caused the
Alert.
2. Knowledge Base Articles—link to launch the PowerStore KB in a new browser
tab
3. GatherSupport Materials—link to gather PowerStore system logs,
configuration details, and other diagnostic information

PowerStore 3.0 Administration - Troubleshooting

Page 20 © Copyright 2022 Dell Inc.


[email protected]
Troubleshoot Hardware Issues

PowerStore 3.0 Administration - Troubleshooting

© Copyright 2022 Dell Inc. Page 21


[email protected]
Troubleshoot Hardware Issues

Details of an event associated with the alert

Monitoring

All alerts are listed on the Monitoring page. There are two ways to get to this
screen. Select Monitoring from the menu bar. Or, from the Dashboard Overview,
click VIEW ALL ALERTS.

1. Click any heading on this page to sort the table by that column. Click again to
reverse the sort order. Sort by Severity, Code, Description, Resource Type
or Name, Updated Date, or Timestamp.
a. In this example, the list has been sorted by severity to show the critical alerts
at the top.
2. Click the Add Filters button to add filters to the columns. Filtering limits the list
to show only those alerts that match the filters.
a. In this example, Last Updated has been filtered to show alerts that were
updated in last 30 days.
b. Acknowledged alerts are not displayed on the list. To show acknowledged
alerts, filter the Acknowledged button.
3. Select one or more check boxes to acknowledge multiple alerts at once.
4. Click the description of any alert to view its details.

Events filtered by Last Updated field

PowerStore 3.0 Administration - Troubleshooting

Page 22 © Copyright 2022 Dell Inc.


[email protected]
Troubleshoot Hardware Issues

Component-Specific Alerts

The Dashboard Overview shows alerts for the entire cluster. Alerts can also be
found when they are specific to compute, storage, protection, and hardware. If
these objects have generated alerts, see the severity icon in the Alerts column.

Click through the tabs to see an example of each set of alerts.

Virtual Machine Alerts

From the Compute menu, select Virtual Machines to see alerts related to
each vVol-hosted Virtual Machine

Alerts that are related to virtual machines

PowerStore 3.0 Administration - Troubleshooting

© Copyright 2022 Dell Inc. Page 23


[email protected]
Troubleshoot Hardware Issues

Storage Alerts

From the Storage menu, select Volumes, Volume Groups, Storage Containers, File Systems, or NAS Servers to
see lerts related to each of those specific storage objects.

Alerts that are related to storage

Remote System Alerts

From the Protection menu, select Remote Systems to see Alerts related to each the
PowerStore cluster's remote replication partner clusters.

Alerts that are related to remote systems

PowerStore 3.0 Administration - Troubleshooting

Page 24 © Copyright 2022 Dell Inc.


[email protected]
Troubleshoot Hardware Issues

Appliance Alerts

Select the Hardware menu to see Alerts related to each the PowerStore
cluster's Appliances and Front End Ports.

Alerts that are related to an appliance

Storage Resource Fault Containment

 If a fault occurs on a volume, all objects within the affected volume


family/families are marked offline.
 Only the affected volume family (including affected snapshots and clones) is
marked offline.
 An internal alert is generated for each volume and clone affected, for
log/tracking purposes–Any hidden/internal volume or clones will also log an
internal alert.
 If a volume family is marked offline, the following operations on the volume,
clones, and snapshots are blocked:

 Map, unmap, clone, modify, delete, restore, refresh, and create snapshot.

PowerStore 3.0 Administration - Troubleshooting

© Copyright 2022 Dell Inc. Page 25


[email protected]
Troubleshoot Hardware Issues

Fault information

List of critical faults

ENS24 Troubleshooting Installation and Field Replaceable Units

Q: Deepak is performing an initial deployment of a PowerStore 500 with a


single ENS24 expansion enclosure. He has been unable to add the ENS24 to
the base enclosure. What could be preventing the ENS24 expansion
enclosure from being added to the configuration?

PowerStore 3.0 Administration - Troubleshooting

Page 26 © Copyright 2022 Dell Inc.


[email protected]
Troubleshoot Hardware Issues

A: Most likely, there is a cabling or hardware fault causing the issue. The
installation of the ENS24 expansion enclosure occurs after the cluster is created
successfully.
 The addition of the ENS24 expansion enclosure is blocked if:

 There is a hardware fault or issue on the ENS24 expansion enclosure.


 There are missing cables.
 There is a cabling issue such as crossed cables or cable faults.
Blocking the installation is beneficial to ensure that the cluster is healthy before
expanding capacity.
If the expansion enclosure was being added after the initial deployment of the
PowerStore Cluster, the same events could prevent the expansion enclosure from
being added. The difference is that auto-discovery starts when new Storage
Access Module (SAM) IPv6 addresses are found, after cabling and turning on the
new enclosure.

Q: Rebecca sees a replacement alert on an ENS24 expansion enclosure.


Which components can be replaced on the ENS24 Expansion enclosure?

PowerStore 3.0 Administration - Troubleshooting

© Copyright 2022 Dell Inc. Page 27


[email protected]
Troubleshoot Hardware Issues

Hot-Swappable FRUs include:


1. Storage Access Modules (SAMs)
2. Power Supply Units (PSUs)
3. Quad Small Form Factor Pluggable (QSFP) ports

SAM 1
T

SAM 2

ENS24 Expansion Enclosure Rear View

4. Fan packs
5. Clock Distribution Boards (CDBs)

NVMe Expansion Enclosure with Cover Removed

6. Dual Inline Memory Modules (DIMMs)

PowerStore 3.0 Administration - Troubleshooting

Page 28 © Copyright 2022 Dell Inc.


[email protected]
Troubleshoot Hardware Issues

The ENS24 chassis can also be replaced, but this requires downtime for the
enclosure as well as the connected appliance if the enclosure is already in use.

Hardware that requires replacing at the SAM level usually results in a failover to the
other SAM in the enclosure.
Follow the normal process for replacing components for the NVMe expansion
enclosure components.

ENS24 Troubleshooting - Thermal Monitoring

Several components in the NVMe expansion enclosure have thermal temperature


thresholds that the BMC monitors. The BMC controls the fan RPMs and is set
separately per fan.

When a thermal temperature threshold has been surpassed, an alert is


sent.
 A 5-minute grace period starts and the system continues
to monitor the temperatures.
 If after 5 minutes the temperature continues to exceed the threshold, the
expansion enclosure is shut down for 5 minutes.
 If the temperature goes back below the threshold during the 5 minutes, the
alert is dismissed.
 During the shut down process the I/O is halted and caching occurs on the
PowerStore appliance.
 The caching mechanisms on the PowerStore appliance help protect from
any data corruption or loss.
 Access the to the data is lost during the shut down.
 After 5 minutes of being powered off, the expansion enclosure powers back on
for 30 seconds to check the temperature.
 This cycle of 5 minutes of and 30 seconds on continues until the temperature is
under the threshold.

Conditions that generate thermal threshold alerts:


 Ambient temperature

PowerStore 3.0 Administration - Troubleshooting

© Copyright 2022 Dell Inc. Page 29


[email protected]
Troubleshoot Hardware Issues

 PSU temperature
 Three or more SSD temperature exceeded
 Three or more fan rotor failures

Components that generate thermal threshold alerts, but only the SAM shuts
down when the threshold is surpassed:
 SAMs
 DIMMs
 QSFPs
 PCI switches

Traffic is redirected to the other SAM while the SAM is shut down.

PowerStore 3.0 Administration - Troubleshooting

Page 30 © Copyright 2022 Dell Inc.


[email protected]
Troubleshoot Hardware Issues

PowerStore Hardware Troubleshooting Key Points

PowerStore Alerts:
 May or may not indicate a problem requiring user intervention. Focus on Critical
and Major alerts.
 Can be sorted and filtered.
 Includes details helpful in solving the problem:

 Links to repair flows


 Links to support sites
 Error code
 Information about specific hardware affected by the alert
PowerStore LEDs:
 Are in various locations.
 May be steady or blink in different rates.
 Use different colors for different meanings.

Deep Dive: See the PowerStore Service Scripts Guide on


PowerStore: Info Hub.

PowerStore 3.0 Administration - Troubleshooting

© Copyright 2022 Dell Inc. Page 31


[email protected]
Troubleshoot Using PowerStore CLI

Troubleshoot Using PowerStore CLI

PowerStore 3.0 Administration - Troubleshooting

Page 32 © Copyright 2022 Dell Inc.


[email protected]
Troubleshoot Using PowerStore CLI

Use PowerStore CLI to Display System Information

Connect to PowerStore CLI from Client System

PowerStore CLI is intended to manage a PowerStore system, or to use commands


in scripts for automating routine tasks. Choosing the PowerStore CLI or the
PowerStore Manager is a matter of personal preference. Most of the tasks can be
performed with either interface.

Run PowerStore CLI by establishing an SSH session to the primary PowerStore


node using PuTTY. Log in to the service account, and then invoke PSTCLI.

Another option is to install PowerStore CLI on a Linux or Windows workstation and


connect to the primary PowerStore node.

Install and launch the PowerStore CLI client on a Microsoft Windows, Linux, or
UNIX system. PowerStore CLI sends commands to the system through the secure
HTTPS protocol using the PowerStore REST API.

To install the PowerStore CLI client:


1. Go to the Dell Support website for PowerStore Downloads.
2. Download the PowerStore Command Line Interface (CLI) tool for Windows,
Linux, or UNIX.
3. Perform these steps based on the operating system:

 On Windows, double-click the installer and follow the prompts. The default
installation location is:
– 64-bit systems: C:\Program Files\Dell EMC\PowerStore CLI

– 32-bit systems:C:\Program Files (x86)\Dell EMC\PowerStore


Manager CLI

Important: Select Register in system %PATH% environment


variable.

PowerStore 3.0 Administration - Troubleshooting

© Copyright 2022 Dell Inc. Page 33


[email protected]
Troubleshoot Using PowerStore CLI

 On UNIX or Linux, type: rpm -ihv filename, where filenameis the


name of the installer. The default installation location is:
/opt/dellemc/pstcli-version/bin/, where versionis the version
of the client.

Important: Add the installation directory to the PATH variable in the


shell initialization file.

Run PowerStore CLI from Client

Overview

To run PowerStore CLI from an SSH session on the primary PowerStore node, first
connect to the service account. Once the connection is made, the steps are the
same for an SSH session or running from a workstation. Logging directly into the
admin account does not work.

Run PowerStore CLI against the PowerStore primary node—the node running the
management stack. Connecting to a nonprimary node, causes the system to
display an error message:The system was unable establish a secure
connection to the storage server. If the connection fails on all nodes,
the management stack may be down or unresponsive. Contact the support provider
for assistance.

If the PATH variable includes the installation directory, invoke PowerStore CLI from
Windows Command Prompt or UNIX or Linux shell with the pstcli command. If
the PATH variable does not include the installation directory, include the directory in
the command.

The PowerStore CLI client contacts the server that issued the certificate—the
trusted certificate authority (CA)—and confirms the validity of the certificate before
proceeding. When the certificate is verified, PowerStore CLI and its backend server
establish the connection, and begin to exchange data.

PowerStore 3.0 Administration - Troubleshooting

Page 34 © Copyright 2022 Dell Inc.


[email protected]
Troubleshoot Using PowerStore CLI

Command Syntax

The syntax of an example command line:

pstcli [options] object_type [object_qualifier] action


[action_qualifiers]

Entering only pstcli, without any options or commands, lists the options and their
descriptions.

Syntax options:
 -u username
The username on the PowerStore system, such as admin.
 -d hostname or ipaddress
Name or address of the primary node. Use localhost for the primary node,
such as when using an SSH session to the primary PowerStore node.
 [-p password]
Optional: password for the specified account. If the password is not on the
command line, the system prompts for it, as shown in the examples.
 [-session]
Optional: Opens a persistent session with the cli> prompt. Or enter a
command to run and return context back to the workstation.
 [-ssl value] Optional. Where value is one of the following:

 interactive — Prompt to accept the certificates for the current session


(default).
 reject— Automatically reject the certificates.

 accept — Automatically accept the certificates.

 store — Automatically accept and store the certificates.

PowerStore 3.0 Administration - Troubleshooting

© Copyright 2022 Dell Inc. Page 35


[email protected]
Troubleshoot Using PowerStore CLI

Important: From the Windows cmd prompt or UNIX or Linux shell


prompt, start each command with PSTCLI. With the -session
option, PowerStore CLI stays active and displays the cli>
prompt. With the -session option and no command, the
PowerStore CLI session stays open and displays the cli>
prompt. Do not use the pstcli command at the cli> prompt.

Command Examples

For example:

pstcli command with -session option, but without a subcommand:

Starting a session

pstcli command without -session option, but with a subcommand:

Single command option

PowerStore CLI Filtering

The PowerStore Manager CLI or pstcli filtering feature filters fields with different
datatype and array values. Filtering is supported in both session mode and single
action mode. The syntax is similar to SQL WHERE statement.

Syntax: –show -query [not] condition1 [ { and | or } [not]


condition2 ] ...

PowerStore 3.0 Administration - Troubleshooting

Page 36 © Copyright 2022 Dell Inc.


[email protected]
Troubleshoot Using PowerStore CLI

Example: alert show –query “severity == Info and


description_l10n like *upgrade*”

Filtering alert show output

Limitations:

 Not all command output can be filtered.


 Limited to most show parameters where the output is in list format.
 Use –help to determine if a command supports filtering.

PowerStore CLI Help Options

Basic Help

Get help on any command using the help command. Commands are divided into
categories. Get a list of the commands in a category, or a list of all the commands
with the help all command. For example, help all shows that at here are
commands that are named alert and event under Monitoring.

List of PSTCLI commands

PowerStore 3.0 Administration - Troubleshooting

© Copyright 2022 Dell Inc. Page 37


[email protected]
Troubleshoot Using PowerStore CLI

Command Help

The command help alert shows that there is an alert show command, with
an option of -id. Get more information by using help alert show. Note the
syntax that is shown for the highlighted show subcommand. The -id option goes
before show. All other options follow show.

Note the highlighted -limit option. The default limit is 100 records.

Example of a help command

Tab Key

When in "session mode," use the keyboard Tab key to display the options.

PowerStore 3.0 Administration - Troubleshooting

Page 38 © Copyright 2022 Dell Inc.


[email protected]
Troubleshoot Using PowerStore CLI

For example, when connected in "session mode," press the Tab key to display all
the available commands. Type volume and press Tab it displays all the options.
Select one option and press Tab to display the sub-options.

Pressing Tab key from session mode

PowerStore 3.0 Administration - Troubleshooting

© Copyright 2022 Dell Inc. Page 39


[email protected]
Troubleshoot Using PowerStore CLI

Troubleshoot Issues Using PowerStore CLI

Use PowerStore CLI to Show Alerts

 Alert categories and meanings are described in PowerStore Hardware


Troubleshooting - Alerts.

Alert Help

Example of the help alert command,

Help command for alerts

To show all alerts on the system, use the alert showcommand. For example,
Windows clients may be having problems authenticating. An alert states that a
node has changed life cycles. To show the details about this specific alert, select
and copy the id field from the display.

PowerStore 3.0 Administration - Troubleshooting

Page 40 © Copyright 2022 Dell Inc.


[email protected]
Troubleshoot Using PowerStore CLI

Selecting specific alert fields

Show Specific Alert

The show subcommand is after the -id option. This output shows that the node is
now healthy. If there were a problem to correct, the repair_flow_l10n would
show a suggestion of how to correct the problem. Since there is no problem, the
repair_flow_l10n and system_impact_l10n fieldsare empty.

Displaying a specific alert

Sorting by Fields

The default format for show commands is table. The width of the columns is limited
due to the width of your window. Note the hyphen (-) after -sort

PowerStore 3.0 Administration - Troubleshooting

© Copyright 2022 Dell Inc. Page 41


[email protected]
Troubleshoot Using PowerStore CLI

severity_l10n-. The default sort order is ascending. The hyphen changes it to


sort by severity in descending order.

Sorting output by field value

NVP Format

 Instead of the default table, show events or alerts are in name-value pair (NVP),
comma-separated values (CSV), or JavaScript Object Notation (JSON). NVP
format shows the full value of each field name and its value, since it uses one
field per line. To display specific fields and severities of events, use the options
that are available for alerts and events.
 Here are four events showing specific fields in NVP format, which is sorted by
severity in descending order:

PowerStore 3.0 Administration - Troubleshooting

Page 42 © Copyright 2022 Dell Inc.


[email protected]
Troubleshoot Using PowerStore CLI

Named-Value pair (NVP) format

CSV Format

Output the same data to CSV format for use with a spreadsheet, such as Microsoft
Excel. PSTCLI does not allow redirection of output the way Windows command
prompt and UNIX or Linux shells do. To output to a CSV file on your client system:

1. Exit from session mode back to the Windows command prompt or UNIX or
Linux shell.
2. Run PSTCLI from the prompt, providing destination, username, password, and
selection fields. Omit the -session option. Redirect the output to a file. Note: If
the password is not on the command line, the system displays a blank line and
waits. Enter the password to continue.
PSTCLI exits after the command runs.
3. View the .csv file in the client system directory.

PowerStore 3.0 Administration - Troubleshooting

© Copyright 2022 Dell Inc. Page 43


[email protected]
Troubleshoot Using PowerStore CLI

Redirecting PSTCLI output to a file

Use PowerStore CLI to Show Events

Events are similar to alerts. An alert is a summation of one or more events that
need or needed attention. Many events do not generate an alert, so there are more
events than alerts. The same system could have 100,000 events, so scrolling
through them may be impractical.

Event Show Command Syntax

Use the help event show command to list the syntax of the command. It uses
most of the same options as the alert show command. The highlighted options,
-sort, and -select, can be used with the fields listed. Use -limit <value> to
control the number of events that are displayed.

PowerStore 3.0 Administration - Troubleshooting

Page 44 © Copyright 2022 Dell Inc.


[email protected]
Troubleshoot Using PowerStore CLI

event show command options

Default Show Events

By default, event show displays the first 100 events in a table.

PowerStore 3.0 Administration - Troubleshooting

© Copyright 2022 Dell Inc. Page 45


[email protected]
Troubleshoot Using PowerStore CLI

Event show command

NVP Format

 Like alerts, events can be displayed in the more readable NVP format.
 Here are the first four events, showing the most important fields in NVP format
sorted by severity:

Select and sort alerts in NVP format

PowerStore 3.0 Administration - Troubleshooting

Page 46 © Copyright 2022 Dell Inc.


[email protected]
Troubleshoot Using PowerStore CLI

Use PowerStore CLI to Show Hardware

Hardware Help

To display the list of available hardware command options, use the help
hardware command. As with other commands, select which fields to display and
sort by most fields. Here is a partial list of the help for that command:

help hardware command

Show All Hardware

Here is truncated list of all the hardware on a two-appliance cluster.

PowerStore 3.0 Administration - Troubleshooting

© Copyright 2022 Dell Inc. Page 47


[email protected]
Troubleshoot Using PowerStore CLI

...

Hardware show -all command

Use PowerStore CLI to Show Network

If there appears to be a problem with the network, show a listing of the network
status. As with other commands, there are many options available. Use the help
network command to see the list of options.

Network Help

There are options to change and show network settings. Note the multiple options
for ip_pool_addresses highlighted below.

PowerStore 3.0 Administration - Troubleshooting

Page 48 © Copyright 2022 Dell Inc.


[email protected]
Troubleshoot Using PowerStore CLI

Network help

Network Show Command

Here is an example network show command with specific fields:

Network show

PowerStore 3.0 Administration - Troubleshooting

© Copyright 2022 Dell Inc. Page 49


[email protected]
Troubleshoot Using PowerStore CLI

Use PowerStore CLI to Show Licenses

Trial license expiration:


 After a trial license expires, storage objects continue to function. However, no
new resources can be created until the license is rectified.
 If multiple appliances in single cluster are added in different times, the trial
license expires based on the first appliance that was added.

To check the status of the license from the PowerStore CLI:


 Create a PowerStore CLI session: pstcli -d <PowerStore_address> -
session -u admin -p <password>
 Show the license status:

 license show

 For example:

This system is using a temporary license

 When a trial license expires, PowerStore switches into Read-Only mode, but
allows a one-time extension of trial period for an additional seven days.
 When a second extension is requested, a dialog indicates that no additional trial
extension is possible. Install a permanent license.

SSL Certificate Import for Management

By default, PowerStore uses its own Dell-certified certificate, which is often


unknown by a browser. When PowerStore Manager opens, a warning message
about an unsecure connection is displayed.

Use a public key infrastructure (PKI) or certificate authority (CA) to generate a


corporate-wide trusted certificate for the system. SSL Certificate Import is
supported on PowerStore T Block Optimized systems.

PowerStore 3.0 Administration - Troubleshooting

Page 50 © Copyright 2022 Dell Inc.


[email protected]
Troubleshoot Using PowerStore CLI

Configure third-party SSL certificates for management connections using


PowerStore CLI.

Use the x509_certificate csr command to generate the Certificate Sign


Request. Then use the x509_certificatesetcommand to install the signed
certificate.
1. Generate a Certificate Sign Request (CSR) from the PowerStore, providing the
following fields:
 service (Management_HTTP)
 scope (customer-defined value)
 common_name (customer-defined value)
 dns_name (PowerStore FQDN)
 ip_addresses (of PowerStore)
 organizational_unit (customer-defined value)
 organization (customer-defined value)
 locality (city)
 state
 country
 key_length (2048–4096)
2. The generated CSR contains a CSR ID. Make a note of it. Use it when
importing the certificate.
3. Present CSR to customer CA to issue the certificate.
4. Install and enable certificate on PowerStore system using the PSTCLI set
command.
5. When PowerStore Manager opens, the browser compares CA signature in
certificate with known CA signatures.
6. Once a match is found, new connections are trusted.

The x509_certificate command has the following options for csr:

PowerStore 3.0 Administration - Troubleshooting

© Copyright 2022 Dell Inc. Page 51


[email protected]
Troubleshoot Using PowerStore CLI

csr command value options from x509_certificate -help command.

Deep Dive: Certificates can also be managed using REST API. For
more information, see the REST API Reference Guide on
PowerStore: Info Hub

PowerStore CLI NAS Commands

To view the status of SMB or NFS volumes set up on a PowerStore T system, use
the commands that are shown here.

List NAS Servers

If there are NAS servers running on the PowerStore T system, list them with the
nas_server show command.

NAS Servers in normal mode

One server is intended for NFS while the other is intended for SMB. Each server
uses its own IP address.

A NAS server is degraded when a node on the NAS appliance is unavailable and
all NAS servers are hosted on the remaining node. A degraded NAS server is
displayed as follows:

PowerStore 3.0 Administration - Troubleshooting

Page 52 © Copyright 2022 Dell Inc.


[email protected]
Troubleshoot Using PowerStore CLI

NAS Server in degraded mode

List SMB Services

The smb_server show command displays any SMB servers and their
corresponding domain or workgroup and NAS servers.

smb_server show command

The smb_share show command displays SMB shares. In this example, only one
share exists on the system. Add -help to either command for a complete list of
options.

smb_share show command

List NFS Services

To list NFS servers, use the nfs_server show command:

nfs_server show command

PowerStore 3.0 Administration - Troubleshooting

© Copyright 2022 Dell Inc. Page 53


[email protected]
Troubleshoot Using PowerStore CLI

To list NFS exports, use the nfs_export show command:

nfs_export show command

Show File Systems

To list all the file systems, use the file_system show command:

file_system show command

To see what other fields are available to select with this command, type
file_system show -help.

Access PowerStore CLI from PowerStore Service Container

Connect to the PowerStore CLI using a downloaded program on a workstation or


from a Service Container session on the node. The commands are intended to be
the same on both versions. However, the version of PowerStore CLI installed on
the maintenance node may be different from the version that is installed on a
workstation. Commands may vary between versions.

Connect to PowerStore CLI

Example of creating a PowerStore CLI session from the Service Container.

PowerStore 3.0 Administration - Troubleshooting

Page 54 © Copyright 2022 Dell Inc.


[email protected]
Troubleshoot Using PowerStore CLI

Log in to PowerStore CLI from PowerStore Service Container

 Selecting option [1] opens the session. The system prompts again on the next
login.
 Selecting option [2] closes the session.
 Selecting option [3] opens the session. The system does not prompt for the
certificate on subsequent logins.

Set Default Credentials

 Store PowerStore CLI credentials for a specific destination using the -


save_cred option.
 Use the -default option to store the destination as the default.
 To connect from the Service Controller to PowerStore CLI on the same device,
use either the self or localhost keywords in place of the address.

Important: Save credentials on a customer system or customer


array only with their permission. Never use this feature for connecting
remotely over SRS or SupportAssist.

Example of these features:

PowerStore 3.0 Administration - Troubleshooting

© Copyright 2022 Dell Inc. Page 55


[email protected]
Troubleshoot Using PowerStore CLI

Saving default credentials

PowerStore 3.0 Administration - Troubleshooting

Page 56 © Copyright 2022 Dell Inc.


[email protected]
Troubleshoot Using PowerStore CLI

Troubleshooting with PowerStore CLI Key Points

PowerStore CLI:
 Uses a restricted version of the Linux BASH shell.
 Is downloaded from the documentation website.
 Can be used to perform many functions similar to the functions in the
PowerStore Manager.
 Can show hardware, network, and NAS properties.
 Can show alerts and events.

Go to: CLI User Guide on PowerStore: Info Hub

PowerStore 3.0 Administration - Troubleshooting

© Copyright 2022 Dell Inc. Page 57


[email protected]
PowerStore Troubleshooting Tools

PowerStore Troubleshooting Tools

PowerStore 3.0 Administration - Troubleshooting

Page 58 © Copyright 2022 Dell Inc.


[email protected]
PowerStore Troubleshooting Tools

Service Container

Service Container Overview

Use commands to get basic information about the system from the Service
Container CLI. The CLI uses a restricted version of the Linux Bash shell. Only
certain commands are allowed using the service account.

There are two CLI programs available for managing PowerStore systems:
 Service Container
 PSTCLI

 Download PSTCLI to a client or run it from Service Container.

See the Service Scripts Guide on the PowerStore Info Hub.

Connect to PowerStore Service Container

Connect to SSH Session

To access the Service Container, use a terminal emulator to connect to one of the
management addresses or hostname using SSH on port 22. This example uses
PuTTY:
1. Open PuTTY.
2. Connect to one of the management addresses or hostname using SSH on port
22:

PowerStore 3.0 Administration - Troubleshooting

© Copyright 2022 Dell Inc. Page 59


[email protected]
PowerStore Troubleshooting Tools

PuTTY configuration window

Log in to service Account

3. Log in to the service account:

PowerStore 3.0 Administration - Troubleshooting

Page 60 © Copyright 2022 Dell Inc.


[email protected]
PowerStore Troubleshooting Tools

Service account login

svc_diag Command

PowerStore Container commands run scripts of the same name. The svc_diag
command (script) displays many different pieces of system information, depending
on which subcommands and options are used.

Command Syntax

To list svc_diag command options, add --help after the command.

Help command output

PowerStore 3.0 Administration - Troubleshooting

© Copyright 2022 Dell Inc. Page 61


[email protected]
PowerStore Troubleshooting Tools

Add an action subcommand, such as list, from basic help to get more specific
help.

Help command with action subcommand output

Get System Information

To get the Node ID, Appliance Name, Service Tag, Model, IP address, and other
information, use svc_diag list --info command.This command only
displays information about the current node.

PowerStore 3.0 Administration - Troubleshooting

Page 62 © Copyright 2022 Dell Inc.


[email protected]
PowerStore Troubleshooting Tools

System information of the current node

Show Cluster Topology

To show cluster topology, appliances, nodes, software versions, and other


information, use the command: svc_diag list --basic. Scroll back to see
the information at the top.

In the example, the cluster has one model 1000T appliance, which has an ID of A1.
The service tag for the appliance, 3BP42W2, is the same as that for the cluster
displayed on the previous tab.

PowerStore 3.0 Administration - Troubleshooting

© Copyright 2022 Dell Inc. Page 63


[email protected]
PowerStore Troubleshooting Tools

Cluster topology information

Show Fault Status

To list fault status for various hardware components, use the command: svc_diag
list --hardware --sub_options fault_status.

PowerStore 3.0 Administration - Troubleshooting

Page 64 © Copyright 2022 Dell Inc.


[email protected]
PowerStore Troubleshooting Tools

Fault status for hardware components

Networking Service Scripts

Networking commands are important tools to check network interfaces,


connectivity, and capabilities. In PowerStoreOS 3.0, network service scripts start
earlier in the startup process.

The following service scripts and parameters are available:


 svc_networkcheck:
 tcpdump - Trace network traffic

 ethtool - Check interface status

 traceroute - Show path to a network device

 arp - List cache of known Layer 2 addresses

 telnet - Connect or check TCP connection to network device

 dig - DNS server query


 svc_tcpdump: This service script opens a given interface, retrieves TCP/IP
configuration information, and saves output to files.

PowerStore 3.0 Administration - Troubleshooting

© Copyright 2022 Dell Inc. Page 65


[email protected]
PowerStore Troubleshooting Tools

Important: The SSH service on the PowerStore must be enabled to


access service mode and scripts.

Service Script – svc_networkcheck

The svc_networkcheck service script validates network connectivity and other


network services.

Script Help

Type svc_networkcheck -h to receive help on the svc_networkcheck


command.

svc_networkcheck -h parameters

PowerStore 3.0 Administration - Troubleshooting

Page 66 © Copyright 2022 Dell Inc.


[email protected]
PowerStore Troubleshooting Tools

Network Hardware Information

Type svc_networkcheck ethtool with the correct parameters to display


information about the network device driver and hardware settings.

Check link status

Connectivity - Ping

Type svc_networkcheck ping with the correct parameters to validate network


connectivity to other hosts, including a gateway.

PowerStore 3.0 Administration - Troubleshooting

© Copyright 2022 Dell Inc. Page 67


[email protected]
PowerStore Troubleshooting Tools

svc_networkcheck ping response

Validate DNS

Type svc_networkcheck dns with the correct parameters to validate DNS


configuration.

PowerStore 3.0 Administration - Troubleshooting

Page 68 © Copyright 2022 Dell Inc.


[email protected]
PowerStore Troubleshooting Tools

svc_networkcheck dns response

Service Script – svc_tcpdump

The svc_tcpdump script attempts to open a given interface and runs a tcpdump
on it. The tcpdump command can be configured to save outputs to files.

Script Help

Help is available for the svc_tcpdump service script with the -h parameter.

PowerStore 3.0 Administration - Troubleshooting

© Copyright 2022 Dell Inc. Page 69


[email protected]
PowerStore Troubleshooting Tools

svc_tcpdump parameters

SVC_tcpdump -D

The svc_tcpdump -D script screen prints the list of available network interfaces
on the system in which tcpdump can capture packets.

PowerStore 3.0 Administration - Troubleshooting

Page 70 © Copyright 2022 Dell Inc.


[email protected]
PowerStore Troubleshooting Tools

List of available interfaces

Advanced uses of
svc_tcpdump

Advanced uses of svc_tcpdump include writing a dump file and filtering collected
tcpdump traffic.

 Write dump to file:


 Option –w <file> svc_tcpdump stores generated dumps in
/cyc_var/cyc_service/tcpdump.
 The file prefix is powerstore-tcpdump-followed by namespace host-
|sdnas-and file name from option <file>.

 If using values other than default 20 MB and five rotations, the dump file size
-C ## and dump file rotations –W #must be >= 1.

 Secure Copy (SCP) can be used to copy dump files to an external host for
analysis.
 Filter collected tcpdump traffic:

PowerStore 3.0 Administration - Troubleshooting

© Copyright 2022 Dell Inc. Page 71


[email protected]
PowerStore Troubleshooting Tools

 With option –F <file>, it is possible to filter captured data, for example,


only show specific IP or ports.
o Filter file with content dst port 22 and tcp—only show ssh (22/tcp) traffic
reaching the interface.
o Filter file with content host 192.168.18.22—only show packets that are
related to host with IP 192.168.18.22.
 If using SCP, the filter file should be placed in
/cyc_var/cyc_service/tcpdump folder.

Limitations

 Scripts only operate within a service shell session on the local node.
 SSH into a peer node or a node of a different appliance to run scripts there.
 Limits apply to some options.
 For example, ping only supports MTU 1500 or 9000.
 NAS and File use their own scripts.
 For example, svc_nas run nas_svc_tcpdump <options> to run
tcpdump on a NAS container.
 There are some limitations when running in Service Mode (due to failure or
svc_rescue_state set).

 For example:
o svc_networkcheck: tcp_port_check does not allow access to
some ports.
o svc_networkcheck: tracert could be inconsistent.

Service Script Practice

PowerStore 3.0 Administration - Troubleshooting

Page 72 © Copyright 2022 Dell Inc.


[email protected]
PowerStore Troubleshooting Tools

Troubleshoot PowerStore Manager UI Unavailable

If the PowerStore system is running, but the UI is inaccessible, here are some
troubleshooting tips.

Service commands are available using SSH and logging in with the service
credentials.

1. Check the Control Path (CP) status:

[SVC:service@FNM00xxxxxxxxx-A user]$ svc_container_mgmt


status CP
CP Container controlpath is up

Check control path

This status shows that the Control Path is running, but it may be unresponsive
or in an error state. Go to the next step to dig deeper.
2. Check the journal logs with the svc_journalctl command with the -f (or -
-follow) option, which keeps the command active so that new log entries are
displayed as they occur. Use the -g (or --grep) option to filter the command to
see only st_io_monitor status:

SVC:service@FNM00xxxxxxxxx-A user]$ svc_journalctl -f -g


"st_io_monitor"
Jan 06 11:28:48 FNM00xxxxxxxxx-A xtremapp[18033]: Jan 06
11:28:48.336143 R10 [log_id:0][6037(6185 nb_truck_2
0x7f53f1194c40)]st_io_monitor:102: IO counter:76658028988
received, 76658028733 completed, 255 in-progress, 0 aborted,
255 outstanding, 0 canceled, 257 allocated
Jan 06 11:29:48 FNM00xxxxxxxxx-A xtremapp[18033]: Jan 06
11:29:48.333946 R10 [log_id:0][6037(6185 nb_truck_2
0x7f53f1194c40)]st_io_monitor:102: IO counter: 76663405640
received, 76663405372 completed, 268 in-progress, 0 aborted,
268 outstanding, 0 canceled, 269 allocated

Check journal logs

3. Although I/O is being processed, restarting CP might restart the UI:

PowerStore 3.0 Administration - Troubleshooting

© Copyright 2022 Dell Inc. Page 73


[email protected]
PowerStore Troubleshooting Tools

[SVC:service@FNM00xxxxxxxxx-A user]$ svc_container_mgmt


restart CP
Waiting for container restart
Container controlpath restart returned: 0
Container controlpath is back up
Waiting for stack to load
Waiting for stack to load
Waiting for stack to load
Waiting for stack to load

Restarting CP

4. Connect to the PowerStore UI to see if the problem is resolved.

Collect Logs with Service Container

Use the commands shown here to collect PowerStore log files. For example, to
check the status of PowerStore T NAS service, use the commands that are shown
here.

Generate Log File

To generate a specific log file, use the svc_dc run -p profile command,
where profile is one of the following:

 essential:Collect essential data.


 detailed : Collect detailed information.
 hypervisor (PowerStore X): Collect hypervisor data.
 ControlpathHeapDump:Collect control path information.
 nas (PowerStore T): Collect NAS information.

All normal and detailed bundles include the NAS logs. If you only need the NAS
logs, you can use the NAS profile:

1. Generate a NAS-only log with the following command.

PowerStore 3.0 Administration - Troubleshooting

Page 74 © Copyright 2022 Dell Inc.


[email protected]
PowerStore Troubleshooting Tools

Generating NAS-only log

Important: Since the job may take a long time to run, add an
ampersand (&) to the end of the command to run the job in the
background. This action allows you to run other commands while
the job runs. The system notifies you when the job completes, or
you can use the jobs command to check the status.

Locate NAS Log Id

2. Since the svc_dc run command specified a profile of nas, look for a bundle
with that profile. To list and find your bundle, use the svc_dc list command:

Locate NAS log id

The nas bundle has an id of 191cc8a6-baf7-4ac0-861d-fe408ddc1642.


As a next step, look for a link to a file with that name in the directory.

Locate the file in the home Directory

3. Use the ls -l command to list the files in the directory in a long format, and
look for a file with a pointer to 191cc8a6-baf7-4ac0-861d-
fe408ddc1642:

PowerStore 3.0 Administration - Troubleshooting

© Copyright 2022 Dell Inc. Page 75


[email protected]
PowerStore Troubleshooting Tools

The "l" indicates a


This points to the physical
symbolic link.
file.

ls -l command

The l in the first column of each row means that the file is a symbolic link, or
pointer, to a file in the /cyc_var/cyc_service/data_collection folder,
indicated by the -> sign.
Note that each line wraps to the line below, so the id spans two lines.
Use an application, such as Winscp, to download the file to a system where you
can examine the contents. If the application does not accept symbolic links,
copy the original file from the subfolder of
/cyc_var/cyc_service/data_collection to the home directory. Give
the new file a different name.

SDNAS Container

PowerStore has a docker based architecture with containers like Control Path,
Data Path, Serviceability, SDNAS. Access the SDNAS container to collect logs and
run scripts in SDNAS container. Root privileges must be gained in order to access
the SDNAS container.

SDNAS scripts are made available to run within the Serviceability container using
SSH tunneling, which helps in:

 More flexibility
 Simplified workflow
 Easier troubleshooting

Service users can run specific set of NAS scripts without root access.

PowerStore 3.0 Administration - Troubleshooting

Page 76 © Copyright 2022 Dell Inc.


[email protected]
PowerStore Troubleshooting Tools

To view the list of commands that can be run as a service user, use the svc_nas
list command.

Primary Node Secondary Node

PowerStore docker based architecture with containers

PowerStore 3.0 Administration - Troubleshooting

© Copyright 2022 Dell Inc. Page 77


[email protected]
PowerStore Troubleshooting Tools

NVMe NVRAM Failure Handling

PowerStore NVMe NVRAM Failure Handling

When a node is removed from an appliance, the power is removed from half of the
NVMe NVRAM drives.
 The NVRAM drives connected to the battery backup module of the removed
node, lose battery backup.
 If the peer node is also removed, data loss can occur.

 Power is removed from the NVMe NVRAM drives, which blocks their ability
to persistently store data.
 All writes in DRAM cache memory within the node are lost.
PowerStore creates virtual drives to provide redundancy to the NVMe NVRAM
drives while a node is removed.
 Either one or two virtual drives are created, depending on the number of
NVRAM drives in that model.
 PowerStore 1000-3200 models have two NVRAM drives per system: one
virtual drive is created.
 PowerStore 5000-9200 models have four NVRAM drives per system: two
virtual drives are created.
 The virtual drives have reserved space equal to the NVRAM drives in the fsck
reserved space on data drives.
 There is no reduction in the amount of space for data.
 Data is mirrored between the NVMe NVRAM drives and the virtual drives.
 The process that creates the virtual drives and mirrors the data finishes within
30 seconds.

PowerStore 3.0 Administration - Troubleshooting

Page 78 © Copyright 2022 Dell Inc.


[email protected]
PowerStore Troubleshooting Tools

Important: A node with a fault continues to supply power to the other


node when it is fully seated.

Fault Containment for NAS Storage Resources

Fault Containment Overview

Cluster 1

Node A Node B

Appliance 1

NAS

NAS
NAS Volumes
Server

FS1 NAS V1

FS2
NAS V2

V1

PowerStore Storage Resources

Fault Containment for PowerStore Storage Resources:


 PowerStoreOS 3.0 introduces fault containment for file storage resources.

 Fault containment behavior is now consistent between file and block.


 If a file system goes offline, the following items are also placed offline:
Snapshots, clones, SMB shares, and NFS exports.
 Internally, a single PowerStore volume always backups a file system.

Offline File System

In this example, File System 01 is offline.

PowerStore 3.0 Administration - Troubleshooting

© Copyright 2022 Dell Inc. Page 79


[email protected]
PowerStore Troubleshooting Tools

Powerstore file Systems view

Offline Related Resources

Since File System 01 is offline, SMB shares and NFS export related to File
System01 are also offline.

PowerStore SMB shares and NFS export view

PowerStore 3.0 Administration - Troubleshooting

Page 80 © Copyright 2022 Dell Inc.


[email protected]
PowerStore Troubleshooting Tools

PowerStore Troubleshooting Tools Key Points

Serviceability:
 Troubleshoot PowerStore by connecting to the system over SSH or PowerStore
Manager over the Service LAN Ports.
 Use SSH connection to connect to the PowerStore T nodes when the system is
in an unknown or bad state to troubleshoot issues.
 While troubleshooting the PowerStore X, go to the service container with nc
localhost 50000.

Service containers:
 Access PowerStore Service Container using SSH to the management node.
 Run PSTCLI from Service Container.
 Collect many types of configuration and log information using Service
Container.
 In PowerStoreOS 3.0, network service scripts start earlier in the startup
process.
 The svc_networkcheck service script validates network connectivity and
other network services.
 The svc_tcpdump script attempts to open a given interface and runs a
tcpdump on it.
 Scripts only operate within a service shell session on the local node.
 Certain commands are restricted from the service account and can only be
accessed through root account. Accessing the root account is referred to as
inject root.
 Root privileges must be gained in order to access the SDNAS container.
Service users can run only a specific set of NAS scripts without the root access.

NVMe NVRAM failure handling:


 PowerStore creates virtual drives to provide redundancy to the NVMe NVRAM
drives while a node is removed.
 PowerStore provides consistent behavior of Fault containment between file and
block.

PowerStore 3.0 Administration - Troubleshooting

© Copyright 2022 Dell Inc. Page 81


[email protected]
PowerStore Troubleshooting Tools

 When a file system goes offline, the following items are also placed offline:
Snapshots, clones, SMB shares, and NFS exports.

PowerStore 3.0 Administration - Troubleshooting

Page 82 © Copyright 2022 Dell Inc.


[email protected]
Gather Support Materials

Gather Support Materials

PowerStore 3.0 Administration - Troubleshooting

© Copyright 2022 Dell Inc. Page 83


[email protected]
Gather Support Materials

Gather Support Materials Using PowerStore Manager

Data Collection Overview

The collection of support materials is one of the core PowerStore serviceability


tools. Data collection is used to gather logs, configuration data, system
dumps, and other information.

The data collections are critical to Support personnel, who can diagnose and repair
issues on PowerStore systems.

Data collections can be run manually using PowerStore Manager or the svc_dc
commands.

TIP: Support materials can be requested for one or more appliances.


Data is always collected at the appliance level. For example, a
collection for a volume collects support materials for the appliance
that contains the volume. A collection for multiple volumes collects
support materials for all the appliances that contain the volumes.

Data Collection Placement

To collect journal logs from PowerStore Manager, select Settings > Support >
Gather Support Materials:

PowerStore 3.0 Administration - Troubleshooting

Page 84 © Copyright 2022 Dell Inc.


[email protected]
Gather Support Materials

Support Materials Library

The Support Materials Library displays the existing compressed collections of ~220
MB Log files.

Collect Support Materials

Support Materials Collection - Settings

Collecting Support Materials can be initiated several ways from PowerStore


Manager. Data Collections can be started at the Cluster, Appliance, Storage
Resource, and Alerts level.

To start a data collection:


1. Select Settings from the Dashboard.
2. Select Gather Support Materials from the Support menu.
3. Select Gather Support Materials.

The collection includes support materials, such as system logs, configuration


details, and other diagnostic information. You can specify which object to collect
data for. The completed support materials package is archived on your cluster and
can be downloaded from the Support Materials Library.

PowerStore 3.0 Administration - Troubleshooting

© Copyright 2022 Dell Inc. Page 85


[email protected]
Gather Support Materials

Data is collected at the appliance level. You may include additional information in
the data collection by selecting the Advanced support materials
collectionoptions. The Advanced option collects additional system information
and requires additional space on the Node. Advanced collections can take longer
than the default data collection. The system runs one collection job at a time. The
video below describes how to Gather Support Materials.

Procedure:

 Select Send to Support if you want the system to automatically send the
collection to Support when the job completes. This option is only enabled when
remote support through Secure Remote Services is enabled on the system. The
collection may be sent to Support later.
 Select Recent Jobs to monitor the support collection job. When the collection
job completes, the system posts the job information, including its status, on the
Support Materials page.

Storage Objects - Volume Collection

Volumes, Volume Groups, and vVols

Data collections can be initiated by selecting a storage object such as a Volume,


Volume Group, or vVols. Start a data collection by checking the storage object
box and selecting MORE ACTIONS tab > Gather Support Materials. The
collection is done on the appliance that owns the object.

Alerts - Remediation

When an alert is generated, it is displayed in various locations such as the


Dashboard, Monitoring, or the Hardware page. Selecting an alert launches the
Alerts slide-out panel, which describes information about the alert and allows the
alert to be acknowledged. Select Gather Support Materials from the Service and
Support options to gather data directly from the Alerts page.

PowerStore 3.0 Administration - Troubleshooting

Page 86 © Copyright 2022 Dell Inc.


[email protected]
Gather Support Materials

Procedures:

 Select an Alert to open.


 From the Alert page, select Gather Support Materials
 From the GatherSupport Materials page, select Gather Support Materials
 From the Gather Support Materials slide out, enter the Description, select
Object Type and Object Name.

Support Materials - Hardware

In addition to the Settings > Support Materials page, you can perform Gather
Support Materials from the Hardware page. The example shows a data collection
being done on PowerStoreDemo-appliance-2.

Procedure:

 From the Hardware page, select the appliance for the data collection
 From the MORE ACTIONS drop-down, select Gather Support Materials
 From the Gather Support Materials page, provide a Description.
 Select the icon to expand Advanced support materials collection options, if
needed.
 Check the box to Include additional information.
 Select the START button.

Download Support Materials to a Windows Host

Once data collection completes, you can DOWNLOAD the file to a host, SEND TO
SUPPORT or DELETE it.

Select the checkbox for the data collection files, and then select the option that you
want. The SEND TO SUPPORT option is only available if the SupportAssist page
is configured. The below video demonstrates how to download support materials to
windows host.

PowerStore 3.0 Administration - Troubleshooting

© Copyright 2022 Dell Inc. Page 87


[email protected]
Gather Support Materials

The example shows the download process. The Initial Configuration data
collection includes files from both appliances. Once the files are selected, it is
downloaded to the default Downloads folder on the localhost. The output is a
.tgz file. Optionally, the file can be exported in either a .csv or .xlsm file to the
same Downloads folder.
1. Select the file.
2. Select DOWNLOAD.
3. Select the file to download if both service and dump files are available. There
could be dump file or regular system collect files.

Collect Journal Logs with Predefined Time Ranges

Data Collection Time Range

 PowerStoreOS allows manual and automatic generation of support materials.


 Dell Support Team uses support materials to troubleshoot issues
 Support Materials archive contains:
 Journal log
 Output from scripts, such as svc_diag

 Dumps and traces


 Array configuration
 Performance data
 Scripts to analyze data collection
 PowerStoreOS 2.x and earlier versions only create support material with a full
set of available journal files.
 PowerStoreOS 3.0 manual collection generates support material containing
journal logfiles for a specific time period.
 Automatic daily data collection covers journal log since last successful daily
data collection.
 This feature reduces file size, resulting in:

 Less space consumption for log files

PowerStore 3.0 Administration - Troubleshooting

Page 88 © Copyright 2022 Dell Inc.


[email protected]
Gather Support Materials

 Less transfer time and less bandwidth


 Faster analysis of data

Manual Log Collection

 PowerStore uses syslog journal for centralized logging.


 The syslog journal is fragmented into files of about 220 MB each.
 PowerStore Manager allows these timeframes:
 All
 Last 24 hours
 Last 48 hours
 Last week
 Last 2 weeks
 Custom - enter specific start and stop date and times.
 Support Material collection always collects complete journal files to keep
metadata with log entries.
 When each journal file reaches about 220 MB, collection starts a new file,
regardless of the custom timeframe.
 A custom timeframe can span multiple files.
 For a custom timeframe 4:00 to 8:00, three files are collected.

CustomTimeframe 4:00 to 8:00

Journal file: 1:31 to 4:23 Journal file: 4:24 to 7:51 Journal file: 7:52 to 9:16
~220 MB ~220 MB ~220 MB

Custom timeframe spanning multiple journal files

PowerStore 3.0 Administration - Troubleshooting

© Copyright 2022 Dell Inc. Page 89


[email protected]
Gather Support Materials

Storage Space Consumption

PowerStore automatic daily log collections only contain data since the last
successful automatic collection.

 PowerStore daily collections only contain data for that day:

 Day one data is 1.8 GB. Day two data is 1.6 GB. Day three data is 1.9 GB.
These three logfiles consume 10.5 GB for three days (1.8 GB + 3.4 GB + 5.3
GB).

Day 1 (1.8 GB)


Day 2 (1.6 GB)
Day 3 (1.9 GB)

Log files with data from one day

Practice Data Collection Using PowerStore Manager

The Online Course Contains an Interaction Here.

[Detailed description of the Interaction for Guides]

PowerStore 3.0 Administration - Troubleshooting

Page 90 © Copyright 2022 Dell Inc.


[email protected]
Gather Support Materials

Gather Support Materials Using Service Container

Service Commands for Data Collection

The svc_dc (data collect) command set is used to manage data collections on the
PowerStore. All commands can be found in the PowerStore Service Scripts Guide.
Service commands are available from the Service shell using SSH, Port 22, and
logging in with the service credentials. The service shell is restricted and limits the
ability to run certain commands.

svc_dc command options

Manage Data Collects with Service Commands

svc_dc run

The svc_dc run command generates a new data collection on the local
appliance using the default profile. Each appliance in a cluster generates its own
data collection archive and stores it locally on that appliance.

The example generates a new data collection on appliance FNM10203400242


using the essential and detailed profiles. Use the svc_dc run -h option for
additional details.

PowerStore 3.0 Administration - Troubleshooting

© Copyright 2022 Dell Inc. Page 91


[email protected]
Gather Support Materials

svc_dc run command

svc_dc list

The svc_dc list command lists all data collections, or details for one collection.
Each data collection is identified by an ID. Use the ID in the command to get details
of the collection. For example, svc_dc list 949af06e-7896-4b36-9f29-
d3871a13ad6.

svc_dc list command

svc_dc download

The svc_dc download command all downloads a data collection file to a Linux
host. The commands require an IP address, path, username, and password of
the remote Linux host. If the PowerStore data collection contains several files,
select the file by entering the Indexnumber of the file.

The file transfer uses the SCP protocol on TCP port 22. When the file transfer is
underway, there is no indication of progress, which could take several minutes to
an hour to complete.

In the example, the file that is indicated by Index 10 is downloaded to the Linux
host into /opt directory.

PowerStore 3.0 Administration - Troubleshooting

Page 92 © Copyright 2022 Dell Inc.


[email protected]
Gather Support Materials

The status is displayed in the Service Container when


the download is complete.

On the targget Linux system.

svc_dc download command

svc_dc delete

Since data collection requires space, delete files that are no longer needed. The
svc_dc delete command displays a list of files on the PowerStore Delete any or
all of them.

The example shows the file with Index 1 being deleted from the PowerStore
database.

svc_dc delete command

PowerStore 3.0 Administration - Troubleshooting

© Copyright 2022 Dell Inc. Page 93


[email protected]
Gather Support Materials

NAS Service Scripts for Data Collection

The table contains NAS commands available in the serviceability container.

NAS Service Script Description

nas_svc_acldb_dump This command downloads the


ACL database of an online file
system in order to troubleshoot
security issues.

nas_svc_dac This command is used for


managing Microsoft Dynamic
Access Control (DAC).

nas_svc_dc This command generates an


archive file with SDNAS
materials.

nas_svc_nas This command is used for


managing NAS server, database
maintenance, and network
troubleshooting.

nas_svc_paxstats This command displays the


statistics of NDMP and PAX
backup sessions that are in
progress in the NAS servers.

nas_svc_tcpdump This command captures the


tcpdump of a specified network
interface.

Practice Data Collection Using Service Container

The Online Course Contains an Interaction Here.

This interaction performs data collection from the service account CLI.

PowerStore 3.0 Administration - Troubleshooting

Page 94 © Copyright 2022 Dell Inc.


[email protected]
Gather Support Materials

Data Collection Key Points

Data Collection using PowerStore


 Data collection in PowerStore Manager is used to gather logs, configuration
data, system dumps, and other information.
 Data collections can be run manually using PowerStore Manager or the
svc_dc commands.
 Support materials can be requested for one or more appliances.
 By default, data collections are gathered daily (24 hrs) from all appliances.

Data Collection using Service Container:


 The svc_dc (data collect) command set is used to manage data collections on
the PowerStore.
 The svc_dc run command generates a new data collection on the local
appliance using the default profile.
 Since data collections require space, delete files that are no longer needed.

Deep Dive: See thePowerStore Service Scripts Guideon


PowerStore: Info Hub.

PowerStore 3.0 Administration - Troubleshooting

© Copyright 2022 Dell Inc. Page 95


[email protected]
[email protected]
Appendix

PowerStore 3.0 Administration - Troubleshooting

© Copyright 2021 Dell Inc. Page 97


[email protected]
Appendix

Blink LED

LEDs in PowerStore Manager

PowerStore 3.0 Administration - Troubleshooting

Page 98 © Copyright 2021 Dell Inc.


[email protected]
PowerStore 3.0 Administration - Troubleshooting

© Copyright 2022 Dell Inc. Page 99


[email protected]

You might also like