EMC ISILON CUSTOMER TROUBLESHOOTING GUIDE

HOW TO ADD A DRIVE TO A

NODE

OneFS 7.2 - 8.1.1

Abstract
This guide helps you to troubleshoot problems with replaceing a drive in a node.

March 7, 2019

1 - EMC Isilon Customer Troubleshooting Guide: How to Add a Drive to a Node

___________________________
For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.
___________________
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.
Contents and overview

Note
Follow all of these steps, in order, until you reach a resolution.

1. Follow these Before you begin

steps. Page 3

2. Perform
Start troubleshooting
troubleshooting
Page 4
steps in order.

Add the drive

Page 5

3. Appendixes
Appendix A
If you need further assistance

Appendix B
How to use this flowchart

Appendix C
Example output

Appendix D
Example output FlexProtect job

2 - EMC Isilon Customer Troubleshooting Guide: How to Add a Drive to a Node

___________________________
For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.
___________________
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.
Before you begin

CAUTION!
If the node, subnet, or pool that you are working on goes down during the course of
troubleshooting and you do not have any other way to connect to the cluster, you could
experience data unavailability.

Therefore, make sure that you have more than one way to connect to the cluster before
you start this troubleshooting process. The best method is to have a serial console
connection available. This way, if you are unable to connect through the network, you
will still be able to connect to the cluster physically.

For specific requirements and instructions for making a physical connection to the
cluster, see article 304071 on the EMC Online Support site.

Before you begin troubleshooting, confirm that you can connect through either another
subnet or pool, or that you have physical access to the cluster.

Configure screen logging through SSH

We recommend that you configure screen logging to log all session input and output during your troubleshooting session. This
log file can be shared with Isilon Technical Support, if you require assistance at any point during troubleshooting.

Note: The screen session capability does not work in OneFS 7.1.0.6 and 7.1.1.2. If you are running either of these versions, you
can configure logging by using your local SSH client's logging feature.

1. Open an SSH connection to the cluster and log in by using the root account.
Note: If the cluster is in compliance mode, use the compadmin account to log in. All compadmin commands must be
preceded by the sudo prefix.
2. Change the directory to / i f s/ dat a/ I si l on_Suppor t by running the following command:
cd / i f s/ dat a/ I si l on_Suppor t
3. Run the following command to capture all input and output from the session:
scr een - L
This will create a file named scr eenl og. 0 that will be appended to during your session.
4. Perform troubleshooting.

3 - EMC Isilon Customer Troubleshooting Guide: How to Add a Drive to a Node

___________________________
For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.
___________________
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.
Start troubleshooting

Introduction
Start troubleshooting here. For an overview of Note
the conventions used in this flowchart, see Self encrypting drives (SEDs) can
Appendix B: How to use this flowchart. stay in the PREPARI NG state for an
Start
hour or longer. This behavior is
normal when formatting a drive. Do
not interrupt the PREPARI NG state
because doing so locks the drive.

If you have not done so already, log in to

the cluster and configure screen logging
through SSH, as described on page 3.

To determine the drive health, run the following

command for your version of OneFS:

OneFS 8.0.0 - 8.1.1

i si devi ces l i st

OneFS 7.2
i si devi ces

See __________
Appendix C for example output.

Does the output

indicate that any of the
drive bays are in
SMARTFAI L, REPLACE,
EMPTY, or NEW REPLACE,
status?
SMARTFAI L EMPTY, or
NEW

Go to Page 5 Go to Page 6

4 - EMC Isilon Customer Troubleshooting Guide: How to Add a Drive to a Node

___________________________
For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.
___________________
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.
Add the drive

You could have arrived here from:

- ________________________
Page 4 - Start troubleshooting
Page
- _____________________
Page 6 - Add the drive (2)
5

Verify whether the FlexProtect job is running by running the following command:

i si j ob st at us

In the Runni ng and queued j obs section, find the Fl exPr ot ect job.
_________
See Appendix D for example output.

Is the
Yes FlexProtect job No
running?

Note the page number that you

Allow the FlexProtect job to finish.
are currently on.
After the job is successfully
Upload log files and contact Isilon
completed, the drive bay status
Technical Support, as instructed in
changes to REPLACE.
Appendix A.

After the FlexProtect job is

successfully completed, return to
page 4 to continue
troubleshooting.

Return to Page 4

5 - EMC Isilon Customer Troubleshooting Guide: How to Add a Drive to a Node

___________________________
For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.
___________________
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.
Add the drive (2)

You could have arrived here from:

- Page 4 - Start troubleshooting Page

6

Look for drives that are in a down or soft_fail status by

running the following command:

i si _gr oup_i nf o

See the example output at the bottom of this page.

Does the output

indicate that any devices
Yes are in a down or
No
sof t _f ai l ed status?

Return to Page 5 Go to Page 7

Example output
Cl ust er - 1# i si _gr oup_i nf o
ef s. gmp. gr oup: <2, 110>: { 1: 0- 33, 2: 0- 16, 18- 33, 3: 0- 14, 16- 33, 36, down: 2: 17, sof t _f ai l ed: 2: 17 }

6 - EMC Isilon Customer Troubleshooting Guide: How to Add a Drive to a Node

___________________________
For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.
___________________
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.
Add the drive (3)

You could have arrived here from:

- Page 6 - Add the drive (2) Page

7

Replace the drive by consulting the drive

replacement guide for the particular type of node.
See this search string for a list of the available
drive replacement documents.

To add the new drive, run the following command on the affected node, where
<bay#> is the number of the bay and <node#> is the number of the node to which
the drive was added:

OneFS 8.0 - 8.1.1

i si devi ces dr i ve add <bay#> - - node- l nn=<node#>

OneFS 7.2
i si devi ces - a add - d : <bay#>

If the drive is successfully added, the output will appear similar to this:
! ! The add oper at i on succeeded. A heal t hy dr i ve was f ound i n bay ##.

Is the drive
successfully Yes Go to Page 8
added?

No

Note the page number that you

are currently on.
Upload log files and contact Isilon
Technical Support, as instructed in
Appendix A.

7 - EMC Isilon Customer Troubleshooting Guide: How to Add a Drive to a Node

___________________________
For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.
___________________
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.
Add the drive (4)

You could have arrived here from: Note

Self encrypting drives
- ____________________
Page 7 - Add the drive (3) Page (SEDs) can stay in the
- _____________________
Page 10 - Add the drive (6) 8 PREPARI NG state for an hour
or longer. This behavior is
normal when formatting a
drive. Do not interrupt the
PREPARI NG state because
doing so locks the drive.
To determine the drive health, run the following
command for your version of OneFS:

OneFS 8.0.0 - 8.1.1

i si devi ces l i st

OneFS 7.2
i si devi ces

In the output, look for the bay number and the

status of PREPARI NG or HEALTHY.

Does the output

indicate that the drive
status is PREPARI NG or
No Go to Page 9
HEALTHY?

Yes

A drive status of PREPARI NG or

HEALTHY indicates that the drive
replacement is successful.

Return the non-working drive to

EMC Isilon Technical Support by
using the provide return label that
was shipped with the new drive.

End troubleshooting

8 - EMC Isilon Customer Troubleshooting Guide: How to Add a Drive to a Node

___________________________
For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.
___________________
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.
Add the drive (5)

You could have arrived here from:

- Page 8 - Add the drive (4)

Page
9

Note the page number that you

Does the output
are currently on.
indicate that the drive
status is EMPTY or
No Upload log files and contact Isilon
Technical Support, as instructed in
REPLACE?
Appendix A.

Yes

Determine whether your

workflow allows you to power
cycle the node now, without
affecting production.

Are you able to Unable to continue troubleshooting.

power cycle the node to Isilon Technical Support suggests scheduling an
finish troubleshooting, No outage window to complete the drive installation.
without affecting When you are able to continue troubleshooting,
production? return to page 10 of this guide.

Yes

Go to Page 10

9 - EMC Isilon Customer Troubleshooting Guide: How to Add a Drive to a Node

___________________________
For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.
___________________
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.
Add the drive (6)

You could have arrived here from:

- Page 9 - Add the drive (5)

Page
10

Power cycle the node.

After the node reboots, try to add the drive again by running the following command
on the affected node, where <node#> is the number of the node, and <bay#> is the
number of the bay that the drive was added to:

OneFS 8.0 - 8.1.1

i si devi ces dr i ve add <bay#> - - node- l nn=<node#>

OneFS 7.2
i si devi ces - a add - d <node#>: <bay#>

If the drive is successfully added, the output will appear similar to this:
! ! The add oper at i on succeeded. A heal t hy dr i ve was f ound i n bay ##.

Did the drive

add operation Yes Return to Page 8
succeed?

No

Note the page number that you

are currently on.
Upload log files and contact Isilon
Technical Support, as instructed in
Appendix A.

10 - EMC Isilon Customer Troubleshooting Guide: How to Add a Drive to a Node

___________________________
For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.
___________________
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.
Appendix A: If you need further assistance

Contact Isilon Technical Support

If you need to contact Isilon Technical Support during troubleshooting, reference the page or step that you need help with.
This information and the log file will help Isilon Technical Support staff resolve your case more quickly.

Upload node log files and the screen log file to Isilon Technical Support
1. When troubleshooting is complete, in the command-line interface, type exi t to end your screen session.
2. Gather and upload the node log set and include the SSH screen log file by using the command appropriate for
your method of uploading files. If you are not sure which method to use, use FTP.

ESRS:
i si _gat her _i nf o - - esr s - - l ocal - onl y - f / i f s/ dat a/ I si l on_Suppor t / scr eenl og. 0

FTP:
i si _gat her _i nf o - - f t p - - l ocal - onl y - f / i f s/ dat a/ I si l on_Suppor t / scr eenl og. 0

HTTP:
i si _gat her _i nf o - - ht t p - - l ocal - onl y - f / i f s/ dat a/ I si l on_Suppor t / scr eenl og. 0

SMTP:
i si _gat her _i nf o - - emai l - - l ocal - onl y - f / i f s/ dat a/ I si l on_Suppor t / scr eenl og. 0

SupportIQ:
Copy and past the following command.
Note: When you copy and paste the command into the command-line interface, it will appear on multiple lines
(exactly as it appears on the page), but when you press Enter, the command will run as it should.

i si _gat her _i nf o - - l ocal - onl y - f / i f s/ dat a/ I si l on_Suppor t / scr eenl og. 0 - - noupl oad \
- - syml i nk / var / cr ash/ Suppor t I Q/ upl oad/ f t p

3. ___________
If you receive a message that the upload was unsuccessful, refer to article 304567 on the Dell EMC Online
Support site for directions on how to upload files over FTP.

11 - EMC Isilon Customer Troubleshooting Guide: How to Add a Drive to a Node

___________________________
For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.
___________________
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.
Appendix B: How to use this flowchart

Introduction
Describes what the section helps you to
accomplish.

You could have arrived here from:

- Page 4 - Start Troubleshooting

Page
# Note
Provides context and additional
Directional arrows information. Sometimes a note is linked
indicate the path through to a process step with a colored dot.
the process flow.

Yes Decision diamond No

Process step with command:

Process step
CAUTION! command xyz
Caution boxes warn that
a particular step needs
to be performed with
great care, to prevent
serious consequences.
Optional process step Go to Page #

End point
Document Shape
Calls out supporting documentation
for a process step. When possible,
these shapes contain links to the
reference document.
Sometimes linked to a process step
with a colored dot.

12 - EMC Isilon Customer Troubleshooting Guide: How to Add a Drive to a Node

___________________________
For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.
___________________
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.
Appendix C: Example output

You could have arrived here from:

- Page 4 - Start Troubleshooting

Example output: isi devices list for OneFS 8.x

Cl ust er - 2# i si devi ces l i st
Lnn Locat i on Devi ce Lnum St at e Ser i al
--------------------------------------------------------
2 Bay 1 / dev/ da1 12 HEALTHY AA0000111112AA
2 Bay 2 / dev/ da2 10 HEALTHY AA0000111113AA
2 Bay 3 / dev/ da3 9 HEALTHY AA0000111111AA
2 Bay 4 / dev/ da4 8 HEALTHY AA0000111114AA
2 Bay 5 / dev/ da5 7 HEALTHY AA0000111115AA
2 Bay 6 / dev/ da6 6 HEALTHY AA0000111116AA
2 Bay 7 / dev/ da7 13 HEALTHY AA0000111117AA
2 Bay 8 / dev/ da8 4 HEALTHY AA0000111118AA
2 Bay 9 / dev/ da9 3 REPLACE AA0000111119AA
<truncated>

Example output: isi devices output for OneFS 7.2

cl ust er - 1# i si devi ces

Node 1, [ ATTN]
Bay 1 Lnum 0 [ HEALTHY] SN: Z123AAAA0000111111AA / dev/ da1
Bay 2 Lnum 5 [ HEALTHY] SN: Z124AAAA00001111AAAA / dev/ da2
Bay 3 Lnum 4 [ HEALTHY] SN: Z125AAAA00001111AAA1 / dev/ da3
Bay 4 Lnum 3 [ HEALTHY] SN: Z126AAAA000011111A11 / dev/ da4
Bay 5 Lnum 2 [ HEALTHY] SN: Z127AAAA000011111A1A / dev/ da5
Bay 6 Lnum 1 [ REPLACE] SN: Z128AAAA00001111AAA1 / dev/ da6

<truncated>

13 - EMC Isilon Customer Troubleshooting Guide: How to Add a Drive to a Node

___________________________
For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.
___________________
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.
Appendix D: Example output FlexProtect job

You could have arrived here from:

- Page 5 - Add the drive

Example output with FlexProtect job running

Cl ust er - 1# i si j ob st at us
The j ob engi ne can t empor ar i l y onl y r un Fl exPr ot ect and Fl exPr ot ect Li n j obs.
Coor di nat or : 1
Connect ed: Tr ue
Di sconnect ed Nodes: -
Down or Read- Onl y Nodes: Fal se
St at i st i cs Ready: Tr ue
Cl ust er I s Degr aded: Tr ue
Run Jobs When Degr aded: Fal se

Runni ng and queued j obs:

I D Type St at e I mpact Pr i Phase Runni ng Ti me
----------------------------------------------------------
562 Fl exPr ot ect Runni ng Medi um 1 2/ 6 3s
----------------------------------------------------------
Tot al : 1
Recent f i ni shed j obs:
I D Type St at e Ti me
--------------------------------------------------------
552 Wor mQueue Succeeded 2015- 12- 06T02: 00: 05
553 ShadowSt or ePr ot ect Succeeded 2015- 12- 06T04: 00: 13
554 ShadowSt or ePr ot ect Succeeded 2015- 12- 06T20: 00: 11
555 Wor mQueue Succeeded 2015- 12- 07T02: 00: 02

<truncated>

Example output with FlexProtect job completed

Cl ust er - 1# i si j ob st at us
The j ob engi ne i s r unni ng.

No r unni ng or queued j obs.

Recent f i ni shed j obs:
ID Type St at e Ti me
----------------------------------------------------------
553 ShadowSt or ePr ot ect Succeeded 2015- 12- 06T04: 00: 13
554 ShadowSt or ePr ot ect Succeeded 2015- 12- 06T20: 00: 11
562 Fl exPr ot ect Succeeded 2015- 12- 09T02: 07: 15
----------------------------------------------------------
Tot al : 10

14 - EMC Isilon Customer Troubleshooting Guide: How to Add a Drive to a Node

___________________________
For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.
___________________
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.
 2019 © Dell Inc. or its subsidiaries. All rights reserved.

Dell believes the information in this publication is accurate as of its publication date. The
information is subject to change without notice.

THE INFORMATION IN THIS PUBLICATION IS PROVIDED "AS-IS." DELL MAKES NO REPRESENTATIONS

OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND
SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FORA
PARTICULAR PURPOSE. USE, COPYING, AND DISTRIBUTION OF ANY DELL SOFTWARE DESCRIBED IN
THIS PUBLICATION REQUIRES AN APPLICABLE SOFTWARE LICENSE.

Dell, EMC, and other trademarks are trademarks of Dell Inc. or its subsidiaries. Other trademarks
may be the property of their respective owners.

EMC Corporation
Hopkinton, Massachusetts 01748-9103
1-508-435-1000 in North America 1-866-464-7381
www.EMC.com

15 - EMC Isilon Customer Troubleshooting Guide: How to Add a Drive to a Node

___________________________
For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.
___________________
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.