ExaCC Gen 2 initial Tasks

NOTE: This doc only applies for exceptions in which no Oracle Badged FE can be sent to deploy a Gen 2 ExaCC!

X9M HW Overview
Tasks to be performed by Partner Engineer for Gen2 Deployments ( Indirect Countries ) for exceptional cases in which no Oracle Badged FE
can be sent to deploy a Gen 2 ExaCC
Initial Tasks to be performed by partner engineer after delivery of the racks to customer's DC (detailed procedure)

X9M HW Overview

(Same Rackunits apply for Gen 1 to Gen 2 upgrade on an ExaCC X7)

Tasks to be performed by Partner Engineer for Gen2 Deployments (

Indirect Countries ) for exceptional cases in which no Oracle Badged FE
can be sent to deploy a Gen 2 ExaCC
Pre Delivery Prep

Download necessary files from Shared Oradocs Site - Liase with designated Oracle SWAT Engineer for details, Partner FE will need to be
added to below site:-

Partner-FE-SharedDocs

SWAT Engineer will need to know Partner FE email address and whether they already have an SSO account linked to this email address so
he can be added to above Site
Obtain expected rack serial number related to each site from Oracle SWAT Engineer, site address & contact information.
Laptop will be required with serial connection and RJ45 network port (or adapter) - at least 8GB RAM and 100GB free disk space (SSD if
possible)
DB9 male serial port in laptop / docking station or Linux and laptop host OS compatible USB-to-serial DB9 male adapter – adapters with
Prolific PL2303 and FTDI FT232RL chipsets work fine on both Windows and Linux (example: Oracle P/N 530-4424 USB to DB9M Serial
Cable)

DB9F to RJ45F adapter to connect serial port of Cisco/PDU/ILOM RJ45 port (example: Oracle P/N 530-3100 https://support.oracle.com
/handbook_partner/Devices/Cables/530-3100.html)

2 RJ45 CAT5e straight through Ethernet cables

There are USB to RJ45 serial adapters available that can be used instead of USB-to-serial DB9M + DB9F to RJ45F + RJ45 straight through
cable. Examples (FTDI chipset):
https://www.amazon.de/dp/B07MYQPYKR/
https://www.amazon.de/dp/B07N1LZSBN/
https://www.amazon.de/dp/B01LPQM90G/
WiFi or cellular internet connectivity for the laptop on-site
Software installed on laptop: zoom, slack, ssh client (putty / MobaXterm), scp client (winscp), zip/7zip, Oracle Java
Oracle VirtualBox 6.x software installed on laptop
NFS/DHCP Linux server virtual machine image provided by SWAT team for re-imaging ExaCC installed on laptop
It is recommended to setup session logging in ssh client
for ssh
script `date "+ssh_%Y%m%d_%H%M%S.log"`
ssh user@host
for putty
 for MobaXterm

Meet delivery Day 1:

Meet and Greet - Couriers should have all the tools necessary, if not the tools needed for unpacking the rack will be in the spares box (large
rectangular box attached to outside of rack)
If available I would recommend bringing a socket set with Allan key attachments, adjustable spanner & cutters.
Obtain rack serial number from shipping sheets and confirm serial number is correct - Check tilt indiicator is ok and report any damage
immediately
Perform re-racking if required to get the rack into the datacenter
Only remove servers (dbnodes & cells). Leave switches and CMA's in place.
Velcro or tie wraps will be required to hold the CMA's and cables in place once the servers are removed whilst moving the empty
rack.
Re-racking will require 2 people
For moving of the rack in horizontal position it is advisable to remove doors and side panels so these do not get damaged
Label spares kit as ExaCC X9M/X8M (or ExaCC X7 in case of Gen 1 to Gen 2 upgrade) spares
customer has to take care of these in a safe location but they need to be available in case needed by FE for break/fix

Connect ExaCC PDU power cables with all (6) PDU breakers turned-off to customer's power sockets
Ensure PDU's are balanced for resilience and in addition for single phase:-

1. PDU_A Input_2 and PDU_B Input_0 must be on the same phase.

2. PDU_A Input_1 and PDU_B Input_1 must be on the same phase.
3. PDU_A Input_0 and PDU_B Input_2 must be on the same phase.
4. These are marked where they come out of the PDU. Connecting the cables in this manner,ensures that in the case of a failover the phases
are balanced on both the A and B sides.

Follow local policy, e.g. in some countries only certified electricians are allowed to connect power, usually I let customer's DC team
connect the power cables
Ensure correct SFP's are installed in DB & CP Servers ( if applicable) & connect network cables (follow cabling matrix provided by customer)

Installation Power Up Day 2/3:

After 24hrs up to 48hrs of acclimatization (depends on weather conditions during delivery), power-on and perform visual checks.
Perform basic Cisco admin switch config
Perform initial ROCE switch setup
Peform PDU setup

Initial Tasks to be performed by partner engineer after delivery of the racks

to customer's DC (detailed procedure)
Power Up
Turn on all breakers in PDU-B
Observe LEDS
Oracle Servers: PS1 for both the X8-2 & X8-2L servers – the right-hand LED will flash green standby.
CISCO switches: LED on left (viewed from front) will turn GREEN and the right LEDs for the other PSU will be RED
Repeat for PDU-A
 Checks all servers & switches for fault LEDS, NOTE cisco admin switch may have orange LEDS on until configured
Power on all servers cells first followed by DB servers - check ilom leds become solid green on all servers after some time ( 10
minutes or so )
Check for fault leds on servers once more after power up
Any fault light observed login to ilom using serial port ( default passwords are set ) and run >show faulty
Contact SWAT Team member if any faults are found so they can diagnose and raise SR if necessary

Below are factory IP's used on delivered racks

Cisco Configuration

NOTE:- Switch will come with no configuration, the following switch config will need to be download for this procedure:-

Cisco Admin Switch Default Configuration - See also MOS Document 2595364.1 if you have MOS account

Step Task Notes Complete

1 Connect a serial cable to the CISCO console from laptop The default serial port speed is 9600 baud, 8 bits, no parity, 1 stop bit
& no handshake

2 Enter "?" (question mark) and press return. You will see one of two outputs from this. Based on which one you see,
you will proceed with the appropriate steps below.

3A If you see a login prompt (similar to <switch_name> login: ) then Execute the steps in the section below titled, Erasing the
that means the switch has Configuration Step 3AA

been previously configured.

3B If you see the following: Type in "yes" and jump down to the section titled, System Admin
Account Setup and Basic Configuration Step 4
Abort Auto Provisioning and continue with normal setup ?(yes
/no)[n]:

3AA ERASING THE CONFIGURATION

Log in with the admin account. Login: admin Password: welcome1

Erase the configuration with the command: write erase <switch name>-adm0# write erase

Warning: This command will erase the startup-configuration.

Do you wish to proceed anyway? (y/n) [n] y

<switch name>-adm0#
 Reload the system with the command: This will cause the switch to reboot and there will be a

# reload long dialogue. Example below.

<switch name>-adm0# reload

This command will reboot the system. (y/n)? [n] y
2017 Aug 31 01:09:00 exadataX8-adm0 %$ VDC-1 %$ %
PLATFORM-2-PFM_SYSTEM_RESET: Manual
system restart from Command Line Interface
CISCO SWITCH Ver7.59
Device detected on 0:1:2 after 0 msecs
Device detected on 0:1:1 after 0 msecs
Device detected on 0:1:0 after 0 msecs
MCFrequency 1333Mhz
Relocated to memory
Time: 8/31/2017 1:9:22

<SNIP>

INIT: version 2.88 booting

<SNIP>

INIT: Entering runlevel: 3

Running S93thirdparty-script...
Populating conf files for hybrid sysmgr ...
Starting hybrid sysmgr ...
inserting /isan/lib/modules/klm_cisco_nb.o ... done

After the system reboots, you will be prompted with the following
output:
Abort Auto Provisioning and continue with normal setup ?(yes
/no)[n]: yes

---- System Admin Account Setup ----

Answer "yes" and wait about 30 seconds for the next prompt. Do
NOT press "enter"
while waiting or it will select the default value for the next
question.
After the prompt appears, you may proceed to the next section S
tep 4

4 SYSTEM ADMIN ACCOUNT SETUP AND BASIC

CONFIGURATION

5 The first question prompted is if you want to enforce the secure The default is YES, but we do NOT want this.
password standard.

Answer NO to this.

6 Set the password for the admin account and press enter. The recommended password is the default setting of "welcome1"

Confirm the password entry.

7 The Basic System Configuration Dialogue will start.

Ignore the request to register the Cisco Nexus9000 device:

---- Basic System Configuration Dialog VDC: 1 ----

<SNIP>

Would you like to enter the basic configuration dialog (yes/no): no

8 Log in and enter config mode. <switch name>-adm01 login: admin

password: welcome1

9 Set the current UTC time. orcltsw-adm01 # conf term

orcltsw-adm01 # clock set 21:00:00 28 June 2020 <<<< use a time
close to UTC time
orcltsw-adm01 # show clock
21:00:45.898 UTC Mon June 28 2020 <<<< should be close to what
was set above
Time source is Hardware Calendar
orcltsw-adm01 # end

APPLYING THE FACTORY CONFIGURATION

10 Open the switch.conf file in a file editing/viewing application (ie: If using Windows, we recommend using WordPad, because using
vi or Wordpad) to have the contents ready for copy/paste notepad or notepad++ are known to add special characters during copy
operation. /paste.

11 Log in to the switch and enter config mode using the command: c Example:
onfig t
<switch name> # config t
<switch name> (config)#

12 Enable the bash shell. Example: ->> <switch name> (config)# feature bash-shell
<switch name> (config)# exit
 <switch name># run bash
bash4.2$

13 Start a tclsh interpreter from the bash shell. Example:

bash4.2$ cd /bootflash
bash4.2$ tclsh
%

14 At the tclsh prompt (%), enter in the following, and paste the swit Press Enter when you are done to execute the command.
ch_config file downloaded earlier

% puts [open "switch.conf" w+] { <pasted content> }

Exit out of the tclsh and bash shell. % exit

bash4.2$ exit

15 Verify the generated_cisco.conf file has been created. Example <switch name># dir bootflash://
snippet -->> Directory of bootflash:/
...
25 -rw- 5387 Jul 17 2017 05:26:18 +00:00 switch.conf

16 Erase the startup config (Proceed with caution). <switch name># write erase
Warning: This command will erase the startup-configuration.
Do you wish to proceed anyway? (y/n) [n] y

17 Copy the new config to the startup config. <switch name># copy bootflash:switch.conf startup-config
Copying...
Copy complete.

18 Reboot the switch. <switch name># reload

This command will reboot the system. (y/n)? [n] y

The switch will become unresponsive for 1 to 2 minutes. Wait for it to

come up.

19 Once the switch is booted, log in to it. Since we are logging in via Serial Port you will be able to login before
the configuration has completely applied.
Wait 2 minutes for the configuration to complete before moving
onto validation steps.

20 Connect to the Cisco switch via ssh to verify connectivity: [user@laptop ~]$ ssh [email protected]

21 Basic Switch Configuration is complete Once Cisco configuration is complete confirm with Oracle
Engineer that all components can be accessed

ROCE Switch Configuration

Step Task Notes RU RU

20 22
rocea roceb

1 Obtain Roce names and network information from SWAT FE This comes from InstallationTemplate.html

2 Login as the admin user. Password: welcome1

3 Erase the existing configuration. dbm0sw-rocea0# write erase

Warning: This command will erase the startup-configuration.

Do you wish to proceed anyway? (y/n) [n] y <<<<

4 Restart the system so you can perform the automated setup. dbm0sw-rocea0# reload

This command will reboot the system. (y/n)? [n] y <<<<

NOTE: As you are on the switch console, you will see a lot of 2017 Aug 31 01:09:00 dbm0sw-rocea0 %$ VDC-1 %$ %PLATFORM-2-
console messages appear as PFM_SYSTEM_RESET: Manual

you are working. When the switch begins to boot, do NOT system restart from Command Line Interface
press enter at any time or you may
CISCO SWITCH Ver7.59
inadvertently select a default response to questions
Device detected on 0:1:2 after 0 msecs

<SNIP>

5 NOTE: Again, when the switch begins to boot, do NOT press Do you want to enforce secure password standard (yes/no) [y]: no <<<<
enter at any time or you may
For the admin password, use “welcome1”.
inadvertently select a default response to questions. Type a
question mark (?) if you need to Enter the password for "admin":

display the current question/selection again. Confirm the password for "admin":

Running S93thirdparty-script...
 Populating conf files for hybrid sysmgr ...

Starting hybrid sysmgr ...

inserting /isan/lib/modules/klm_cisco_nb.o ... done

Continue with “normal setup”. Respond “yes” for this.

Abort Auto Provisioning and continue with normal setup ? (yes

/no) [n]: yes <<<<

For installation purposes, we do NOT want to enforce secure

password standards. The

customer can change this later. Respond “no” for this.

---- System Admin Account Setup ----

6 When the Basic System Configuration Dialog appears, respond “y Would you like to enter the basic configuration dialog (yes/no): yes <<<<
es” to enter the basic

configuration dialog.

---- Basic System Configuration Dialog VDC: 1 ----

This setup utility will guide you through the basic configuration of

the system. Setup configures only enough connectivity for

management

of the system.

Please register Cisco Nexus9000 Family devices promptly with

your

supplier. Failure to register may affect response times for initial

service calls. Nexus9000 devices must be registered to receive

entitled support services.

Press Enter at anytime to skip a dialog. Use ctrl-c at anytime

to skip the remaining dialogs.

7 Respond “n” or the default choices for the first three questions. Create another login account (yes/no) [n]: n

Obtain the ROCE switch names from SWAT FE Configure read-only SNMP community string (yes/no) [n]: n

Configure read-write SNMP community string (yes/no) [n]: n

Enter the switch name : iad123456exdrocea01 <<< exdrocea01 for RU20, and
exdroceb01 for RU22 <<< Example only

8 Respond yes when asked to configure Out-of-band management Continue with Out-of-band (mgmt0) management configuration? (yes/no) [n]: yes
configuration and fill in the
Mgmt0 IPv4 address : 100.104.0.24 <<< Example only
values as prompted
Mgmt0 IPv4 netmask : 255.255.254.0 <<< Example only
Example values below.
Configure the default gateway? (yes/no) [y]: n
Obtain the ROCE switch IP from SWAT FE

9 Respond yes when asked to configure advanced IP options. Configure advanced IP options? (yes/no) [n]: yes

10 Respond no when asked to configure static route. Configure static route? (yes/no) [n]: no

11 Respond yes when asked to configure DNS and fill in the Configure the DNS IPv4 address? (yes/no) [n]: yes
appropriate entries.
DNS IPv4 address : 100.104.0.2 <<< Example only
Use the CPS01 and CPS02 Management IP’s (example:
100.104.0.2 and 100.104.0.3): Configure the default domain name? (yes/no) [n]: yes

Obtain the ROCE switch DNS info from SWAT FE Default domain name : oraclecloud.internal

12 Respond no when asked to enable telnet service. Enable the telnet service? (yes/no) [n]: no

13 Respond yes when asked to configure SSH service and the NTP Enable the ssh service? (yes/no) [y]: yes
server.
Type of ssh key you would like to generate (dsa/rsa) [rsa]: rsa
Use the CPS01 and CPS02 Management IP’s (example:
100.104.0.2 and 100.104.0.3): Number of rsa key bits <1024-2048> [1024]: 1024

Configure the ntp server? (yes/no) [n]: yes

NTP server IPv4 address : 100.104.0.2 <<< Enter eth0 IP of cps01 - Example
only

14 Select the default values for the following and make sure CoPP is Configure default interface layer (L3/L2) [L2]: <<< Default
set to “strict”.
Configure default switchport interface state (shut/noshut) [noshut]: <<< Default

Configure CoPP system profile (strict/moderate/lenient/dense) [strict]: strict

15 After reviewing the configuration, save the configuration. Would you like to edit the configuration? (yes/no) [n]:

The following configuration will be applied: Use this configuration and save it? (yes/no) [y]: yes

password strength-check [########################################] 100 %

switchname exdrocea01

ip route 100.104.8.0 255.255.254.0 100.104.10.1

vrf context management

ip route 0.0.0.0/0 100.104.10.1

exit

no feature telnet

ssh key rsa 1024 force

feature ssh

ntp server 100.104.10.1

system default switchport

no system default switchport shutdown

copp profile strict

interface mgmt0

ip address 100.104.10.21 255.255.248.0

no shutdown

If there are any errors in the above information, respond “y”

below to go through the Basic

configuration dialogue again. Otherwise, respond “n” to

move on.

16 This will drop you to a login prompt. The consensus is that it would be better to drop off the console session and ssh
in to the switch to verify that
Login as admin or ssh in to verify ssh is running.
ssh is working

17 Enable scp server feature. test123sw-roce0 # conf t

test123sw-roce0(config)# feature scp-server

18 Disable SNMP v2 due to security issues. test123sw-roce0 (config)# show snmp community

NOTE: Customer may wish to reenable this for their own use Community Group / Access context acl_filter
later, but for the installation, disable it.
_______________ __________________ _______ __________

public network-operator <<<< You may or may not see this entry.

test123sw-roce0 (config)# no snmp-server community public ro

test123sw-roce0 (config)# show snmp community

Community Group / Access context acl_filter

_______________ __________________ _______ __________

19 Set the clock. Refer to Cisco Nexus 9000 Series NX-OS – orcltsw-adm01(config)# show clock
Manually Setting the Device
17:39:32.712 GMT Thu Jun 25 2020
Clock and Configuring the Time Zone
Time source is Hardware Calendar
For ExaCC we set Timezone to UTC:
orcltsw-adm01(config)# clock timezone UTC 0 0

orcltsw-adm01(config)# show clock

17:40:25.387 UTC Thu Jun 25 2020

Time source is Hardware Calendar

Set the time in UTC, if necessary:

orcltsw-adm01(config)# clock set 17:53:00 25 June 2020

20 Due to the use of a Management VRF for management traffic, the orcltsw-adm01(config)# no ntp server 100.104.0.2
configured NTP and DNS
orcltsw-adm01(config)# no ntp server 100.104.0.3 <<< if there
servers must be removed and added back to the Management
VRF (these are example IP’s, use orcltsw-adm01(config)# ntp server 100.104.0.2 use-vrf management prefer

appropriate CPS01 and CPS02 IP’s for your engagement - get orcltsw-adm01(config)# ntp server 100.104.0.3 use-vrf management
these from SWAT FE).
Remove DNS and re-add on Management VRF:
Remove NTP and re-add on Management VRF:
orcltsw-adm01(config)# no ip name-server 100.104.0.2
 orcltsw-adm01(config)# no ip name-server 100.104.0.3 <<< if there

orcltsw-adm01(config)# ip name-server 100.104.0.2 use-vrf management

orcltsw-adm01(config)# ip name-server 100.104.0.3 use-vrf management

orcltsw-adm01(config)# end

21 Save the current configuration to flash test123sw-roce0# copy running-config startup-config

[########################################] 100%

Copy complete

22 Basic Switch Configuration is complete Handover to SWAT FE to upgrade firmware if necessary and apply golden
config

PDU Configuration

Latest PDU Firmware can be downloaded here (2.12) - PDU Firmware

Step Task Notes PDU PDU

A B

1 Obtain PDU name and network information from SWAT FE This comes from InstallationTemplate.
html

2 Connect an RS-232 cable between the SER MGT port and the host (e.g. Laptop or other
suitable system).

3 Configure the host’s terminal or terminal emulator (it is assumed that the installer knows Terminal Configuration Settings:

how to do this – otherwise see page 58 of the PDU User’s Guide). • 9600 baud

• 8 bit

• 1 stop bit

• no parity bit

• no flow control

4 At the terminal device, log in to the PDU metering unit: User = admin, pwd = adm1n

pducli->

Password may be: welcome1

5 After successful login, enter the appropriate values from the rack’s OEDA Installation pducli->set net_ipv4_dhcp=Off
Template: see step 1
pducli->set net_ipv4_ipaddr=xxx.xxx.
xxx.xxx

pducli->set net_ipv4_subnet=xxx.xxx.
xxx.xxx

pducli->set net_ipv4_gateway=xxx.
xxx.xxx.xxx

Successful commands will be

acknowledged with:

pducli->set OK => PDU-Reset required

to apply changes ('reset=yes')!!!

It is sufficient to perform the reset just

once (as shown in the task below).

The PDU can be optionally configured

for DNS with:

pducli->set net_ipv4_dns1=xxx.xxx.
xxx.xxx

pducli->set net_ipv4_dns2=xxx.xxx.
xxx.xxx

6 Reset the PDU metering unit which will cause it to reboot pducli->reset=yes

PDU reset! New login required!

7 Remove the RS-232 cable from the SER MGT port.

8 Connect PDUA network cable to admin cisco port 47 & PDUB network cable to admin
cisco port 48
9 Connect to the PDU using a browser on a laptop attached to the Cisco switch with an IP https://<NEW IP Address of PDU>
address on the same subnet as the PDU's. Alternatively connect via a crossover cable
(or standard Ethernet cable to a laptop running a 1G connection). You will probably need to accept the
security note in the web browser when
attaching to the PDU via https.

If the network configuration was

successful the web browser will display
the Metering Overview page.

10 Click on the Net Configuration link found in the upper left side of the page to view the IP You will need to login:
settings.
User = admin, pwd = adm1n

Password may be: welcome1

SETTING PDU SYSTEM TIME, NTP & INFORMATION PDU SYSTEM TIME

11 Select the System Time tab.

13 Configure the Manual Settings: Enter the current date and time and click
Submit.

14 Configure the NTP-Server Settings: • Check the Enable box

• Enter the NTP Server (only one

allowed) from the Installation Template

• Select Time Zone UTC from pull-down

• Click Submit

If NTP is not enabled, rebooting the

PDU resets the date to 1970 January 1.

15 Select the PDU Information tab. • Enter the hostname of PDU from
Installation Template into the Name field
(format similar to EDX52sw-pdua0)

• Enter Exadata X8-2-CC into the

Product Identifier field (case-sensitive).

• Enter the Rack Serial Number (format

similar to AK12345678)

• Optionally set the Location field. This

should refer to where the PDU is located
(e.g.Lab2002-Rack04).

• Click Submit

16 Confirm PDU f/w is at 2.12 Upgrade if necessary - see README

from f/w PDU Firmware

17 Log out from the PDU.

18 PDU Setup is complete Handover to SWAT FE