Home Knowledge Cases RMAs More

[MX] FPC x Major Error - 'TOE Error code: 0x10404'

Article ID KB36025 Created 2020-06-24

Last Updated 2020-07-03

Description
The article explains the meaning of FPC major error with 'TOE Error code: 0x10404' as alarm reason on
MX series devices.

Symptoms
Active Alarm:

user@host> show chassis alarms no-forwarding

1 alarms currently active

Alarm time Class Description
2020-04-29 16:10:11 UTC Major FPC 4 Major Errors - TOE Error code: 0x10404

​Error Logs:

Apr 29 16:10:11 alarmd[19864]: Alarm set: FPC color=RED, class=CHASSIS, reason=FP

Apr 29 16:10:11 craftd[18958]: Major alarm set, FPC 4 Major Errors - TOE Error c

Solution
Check FPC state:

user@host> show chassis fpc detail

Slot 4 information:
State Online
Temperature 53 degrees C / 127 degrees F
Total CPU DRAM 3584 MB
Total XR2 291 MB
Total DDR DRAM 24960 MB
Start time 2019-12-26 06:51:50 UTC

If the FPC is offline, move to the step where you can restart the affected FPC.
Check for core dumps:
user@host> show system core-dumps no-forwarding
/var/tmp/*core*: No such file or directory
--------------------------------------------------------------------------

-rw-r--r-- 1 root wheel 51250246 Apr 29 16:10 /var/crash/core-NPC4.gz.core.0

Core Dump Logs:

Apr 29 16:10:09 fpc4 PFE-0 XM-1 TOE-0 Coredump Scheduled for thread mask 0x1 in
Apr 29 16:10:09 fpc4 ** TOE ERROR 0x1c DETECTED IN PFE 0 TOE host packet transfe
Apr 29 16:10:10 fpc4 Starting Coredump /var/tmp/core-FPC4-PFE0-XM1-TOE0
 If there is a core dump seen as above, then move to the last section where you will have to raise a case with
JTAC for core dump analysis and also collect the logs mentioned for faster root cause analysis. If there is no
core dump, continue with the next steps.
Check detailed logs:

Apr 29 16:10:09 fpc4 toe_interrupt_errors:405 XM TOE chip 4 Thread 0 ifetch err i

Apr 29 16:10:09 fpc4 toe_interrupt_errors:452 XM TOE chip 4 Memory Error Thread 0

Apr 29 16:10:10 fpc4 XMCHIP(1): XMCHIP(1): HOSTIF: Protect: Log Error 0x1, Log Ad
Apr 29 16:10:10 xntpd[19839]: kernel time sync enabled 6001
Apr 29 16:10:11 fpc4 Cmerror Op Set: TOE: TOE XM.0.1.0 : SetErr - thread 0 ifetch
Apr 29 16:10:11 alarmd[19864]: Alarm set: FPC color=RED, class=CHASSIS, reason=FP
Apr 29 16:10:11 fpc4 Cmerror Op Set: TOE: TOE XM.0.1.0 : SetErr - thread 0 ifetch

Jul 25 09:46:48 fpc0 Cmerror Op Set: TOE-MQ-3:0:0: TOE MQ.3.0.0 : SetErr - MQ TOE
Jul 25 09:46:48 fpc0 Cmerror Op Set: TOE-MQ-3:0:0: TOE MQ.3.0.0 : SetErr - MQ TOE
Jul 25 09:46:48 fpc0 Cmerror Op Set: TOE-MQ-3:0:0: TOE MQ.3.0.0 : SetErr - MQ TOE

TOE or Trinity Offload Engine is a mechanism which moves data between ASICs and the FPC microkernel. This
helps in the interaction of the FPC with the internal chips and the hardware on the underlying chip. This TOW
error usually occurs due to memory error in any other chip example: XM chip, MQ chip. An issue with the TOE
functionality will cause the thread that moves data between ASIC and FPC to be blocked/hampered and that
affects the traffic flow from FPC towards the ASIC. The nature of this error is purely stochastic and can recover
post reseat in ideal cases.
Restart FPC:
user@host> request chassis fpc (offline | online | restart) slot slot-number
Here,
user@host> request chassis fpc restart slot 4
If the issue does not resolve after the restart i.e., if errors continue to occur or FPC remains offline, please open
a case with JTAC.
Contact JTAC:
Collect the logs below before contacting JTAC for faster issue analysis.

>request pfe execute command "show syslog messages" target fpc4

>request pfe execute command "show cmerror module" target fpc4
>request pfe execute command "show cmerror module error 0x10404" target fpc4
>request pfe execute command "show hsl2 statistics" target fpc4
>request pfe execute command "show nvram" target fpc4
>request pfe execute command "show heap" target fpc4

AFFECTED PRODUCT SERIES / FEATURES

People also viewed

2022-04 Security Bulletin: Junos OS Evolved: Specific packets reaching the RE lead to a counter overflow
and eventually a crash (CVE-2022-22195)

2022-04 Security Bulletin: Junos OS and Junos OS Evolved: The rpd CPU spikes to 100% after a malformed
ISIS TLV has been received (CVE-2022-22196)

Was this article helpful?

Was this article helpful? 0 0

Choose a general reason

-- Choose a general reason --

Description

Please, leave us some feedback...

Submit