7/23/2020 574696

Summary inc establish taking ~8 hours and the lingering invalids on the R2 side
Status Assigned Severity 1 - Critical/High Code Version 5876.309.196
Responsible Assigned Engineer Product
Mubeen Mukadam Sachin Kokane DTS UCODE SSCO
Engineer
Issue Details
Problem Key SFDC number 018228154 Resolution comment
None None
Update
Problem submitter Closure SFDC
Paul Balestracci
number

Customer SR # and customer Customer Name(s)

Yunnan Rural Credit Cooperatives
Details name(s) Add

Notes

+ Add/request new update

************ Surge Request ************

SURGE Upload

******** Summary ********

Submitter: [email protected]
EE Owner: Sachin Kokane
Site:
SR #:

Upload details:
Today there are 5x M1 invalid tracks left after customer did another new round of recreation
test(split+estab) at around 03:00am 21/Jul UTC.

The issued Grp is 0x50.

2020-07-21 devices on R2 0023 || count of invalid tracks on M1
14C || 2
06:33
150 || 2
Richard Qi 151 || 1

We pick up one issued track from five tracks: V2 R1: 10B3/A5E/B and 10B3/A5E/C ---> V3 R2: 14C/2979/D

Developer B.B. assisted us and checked the R1 V2 array 0020.

Upload 10-day logall from both sides onto SR with array topology.

******** Files ********

Original Location: Local Upload
******** Uploaded Files ********
Topology for RCCB Yunnan.jpg
Logall from three arrays between 20200710 to 20200721.zip

************ Surge Request ************

SURGE Upload

******** Summary ********

Submitter: [email protected]
EE Owner: Sachin Kokane
Site:
SR #:
2020-07-20
06:27 Upload details:
Richard Qi Trace data from both R1 and R2 boxes.

******** Files ********

Original Location: Local Upload
******** Uploaded Files ********
495700019_date200720_time140505.zip
497800023_date200720_time060001.zip

✉ SURGE Support
https://surge.isus.emc.com/surge/ViewOPT.php?OPTNum=574696 1/3
7/23/2020 574696

************ Surge Request ************

Today customer did a recreation of the issue and issue repeated.

The steps of the creation test:

1. Before customer run split, we checked the devices of the groups between R1 and R2 which are consistent.
2. Around 11:00am GMT+8, customer run split , so seen from R2 V3 0023, all SRDF groups are 'split'.
3. Customer waited for 1 hour+, SAM told us after split was done, R2 side Host have some new test IO
written onto R2 side. Seen from R2 V3 0023, some devices have invalids increasing on M3 position.
4. And around 13:00pm GMT+8, customer run establish, seen from R2 side, the M1 sides have invalids marked.
But there are 20x tracks have local mirror invalids and unable to be destaged.
2020-07-20 5. Event trace info have been collected from one of the issued tracks. Upload onto Surge.
06:22 R1 0019: A33/2/2(this track was intermittently marked invalid on M3 and soon disappeared by A7,C) and
Richard Qi A33/2/3(with W/P marked with consistent writes) ---> R2 V3 box 0023: 136/9/1
6. 5 hours passed, now 5 tracks out of 20 tracks have recovered and invalids gone.
7. Now there are 15 tracks have local M1 invalids (checked @10:00am UTC). Customer agrees to leave those
invalids not fixed by D0,EC1 and for ENG to troubleshoot. SAM Guanglu said Customer stated this is very
possibly the last chance for support to set trace, in the future customer won't allow us to set trace.

R1 box CN495700019 is connected on IP Station: 10.80.211.35 (psw: raii; if account is locked, it can be
locked by psw: 'EMC2laptop')
R2 box CN497800023 can be checked by ZOOM: https://Dell.zoom.us/j/99432405306

Upload trace data from both sides of the array.

2020-07-18
10:02
Mubeen Mukadam is added to Notify Others List.
Mubeen
Mukadam

************ Surge Request ************

SURGE Upload

******** Summary ********

Submitter: [email protected]
EE Owner: Sachin Kokane
Site: Yunnan Rural Credit Cooperatives
2020-07-17 SR #: 18228154
15:15
Upload details:
Paul Balestracci Trace logs from the case.

******** Files ********

Original Location: Local Upload
******** Uploaded Files ********
EMC-74EE80BAF1.zip

************ Surge Request ************

SURGE Upload

******** Summary ********

Submitter: [email protected]
EE Owner: Sachin Kokane
Site:
2020-07-17 SR #:
06:43
Upload details:
Charlotte Ma 3 days logall from CN495700020 containing the corrective actions taken on Jul 17

******** Files ********

Original Location: Local Upload
******** Uploaded Files ********
495700020_date200717_time101100.zip

2020-07-17 This issue seems to be similar to OPT 574550 We followed draft copy of KB article which has been written
05:42 for this kind of issue. Device 14C was showing 1 local invalid for track 2979/D it had been cleared by
Sachin Kokane marking corresponding 2 tracks(10B3/A5E/B and 10B3/A5E/C) manually on R1 side with the help of recovery
team. We are working on event tracing messages to capture.
2020-07-17
Problem keys added:
04:14
CTC Recreation
Sean Crowley

2020-07-16
17:30 This looks like OPT 574550. I have sent a CTC request for a recreation based off of the prior cases
George Lettery information.

2020-07-16
14:50 Assigned Engineer for version 5876.309.196 changed from George Lettery to Sachin Kokane
Yohannes Altaye

2020-07-16
14:43
The status of this OPT has been changed to Assigned.
Paul Balestracci

✉ SURGE Support
https://surge.isus.emc.com/surge/ViewOPT.php?OPTNum=574696 2/3
7/23/2020 574696

SURGE VMAX_UCODE SSCO_RDF Escalation

******** Summary ********

Submitter: [email protected]
Site: Yunnan Rural Credit Cooperatives
SR #: 18228154
VMAX Version: 5876.309.196
Registered Builds:
MicroCode: 0054
SymmWin: 111
Serial Number: CN495700019

Where Found: Customer Site

Escalation Reason: DialHome
SFDC Number: 18228154
Escalate per SFA: No
Immediate Response Required: Yes
Engineer Assigned: George Lettery (3889)
CS recovery engineer email: [email protected]
Severity: 1 - Critical/High

======== VMAX VMAX_UCODE SSCO_RDF Escalation Details ========

Issue Summary:
inc establish taking ~8 hours and the lingering invalids on the R2 side

Issue Description:
Array SN: CN495700019 and CN497800023 ( R1: V2 495700019- SRDF/A RDFG:3F-R2: V3 CN497800023)

SRDF Group Numbers: in the array Grps 50 and 51.

inc establish taking ~8 hours and the lingering invalids on the R2 side

SRDF/A group stuck at syncinprog for long time after est from split state.
- Issue seen on multiple R1 VMAX --> R2 VMAX3 environment.

Troubleshooting Steps Performed:

In R2 array 495700023 there are local invalid tracks for devices:

2020-07-16 Devices 14C, 150 and 151 - In SRDF/A group 50 which is up, active and cycle switching.
14:43
Devices 118 and 11A in SRDF/A group 51
Paul Balestracci
Per D0MB/DRDF on these devices a REF2 was prformed today (July 16th). An incremental establish was also
performed against these devices today.

Pairings are as follows, assocaited invalid R2 tracks are in ():

14C - 10B0 - (29BE/2) R1 track: 10B4/A6F/4

150 - 1230 - (CD79/D, CE02/6) R1 tracks: 1233/335E/B & 1234/3380/C
151 - 1238 - (CD79/D)
118 - 1F09 - (66AE/8)
11A - 1F19 - (68AD/1, D9BE/2, DBE0/4)

RDF links to remote arrays 495700020 are up and alive.

The R1 devoces are Metas:

10B0 is an 8 way meta device (10B0 - 10B7).

1230 is an 8 way meta device (1230 - 1237).
1238 is an 8 way meta device (1238 - 123F).
1F09 is an 8 way meta device (1F09 - 1F10).
1F19 is an 8 way meta device (1F19 - 1F20).

Per D0MB I can see that an FDD completed for these devices today. The R2 tracks are getting updated however
the invalids are not clearing. Do we have any other option here than sneding the R1 tracks?

Current Impact:
Customer to do DR testing this weekend and needs issue resolved.

Expectations of Engineering:
Determine what is causing invalid tracks to stay around on the M2 side.

Justification why immediate response is required:

Customer doing DR testing this weekend and doesnt believe RDF is working correctly.

Customer is asking the following:

1. how possibly were these invalids generated? 2. How come the code did not automatically correct them? 3.
Is there way to correct the invalids from front end should it re-occurs during customer test?

This page was created in 0.029515027999878 seconds

✉ SURGE Support
https://surge.isus.emc.com/surge/ViewOPT.php?OPTNum=574696 3/3