POPE/// 5G #2

NSA KPIs

Sanja Bonevska
Salim AOUDIA
Abdillahi Youssouf
Alvaro Cardenas

DREAM/RAN/PSE 23/04/202
0
Agenda

 Introduction
 Présentation des Kpis NR & 4G NSA dans Osiris
 KPIs majeurs 4G NSA & call flow
 KPIs majeurs 5G NR & call flow
 Limitations actuelles sur les KPI/counters
 Présentation TDB : KPIs Paris vs Lille
 Conclusion
 Annexe

2 Interne Orange
 Introduction

NR NSA Architecture - Overview

 We are implementing the NSA (non-stand-
alone) Option 3x

 Only data services are considered for 5G,

while voice should rely on VoLTE or WCDMA-
CS

 PM counters and PM events for NR NSA

needs to be collected from both eNB and
gNB to get the full picture

 PM for Control Plane  eNodeB

 PM for User Plane  eNodeB & gNodeB

 Event Based Statistics (EBS) to produce
additional counters in ENM based on PM
events

3 Interne Orange
 Introduction
 KPI Observability areas

Availability Cell availability

The ability of a service to be obtained, within specified

Accessibilit
tolerances and other given conditions, when requested by the
y
user

Retainabilit The ability of a service, once obtained, to continue to be

y provided under given conditions for a requested duration

Mobility Performance of available handover types

The degree to which a service, once obtained, is provided

Integrity
without excessive impairments

System Hardware usage, Licenses, Traffic and Load management and

Utilisation Admission and congestion management

Radio
Quality of the radio channel propagation conditions
Quality
5 Interne Orange
Agenda

 Introduction
 Présentation des Kpis NR & 4G NSA dans
Osiris
 KPIs majeurs 4G NSA & call flow
 KPIs majeurs 5G NR & call flow
 Limitations actuelles sur les KPI/counters
 Présentation TDB : KPIs Paris vs. Lille
 Conclusion
 Annexe

6 Interne Orange
NR and 4G NSA KPI availability in OSIRIS (1/3)
 Present KPIs in OSIRIS * In process of implementation
Availability OSIRIS Dashboard OSIRIS OSIRIS Query Formulaire KPI
Groupe
4G NSA (ENDC) Yes Yes Yes Yes
KPI
NR KPI On going Partially* Partially* Yes

7 Interne Orange
NR and 4G NSA KPI availability in OSIRIS (2/3)
 Definition for the KPI available in OSIRIS
 Details for the formulas available in OSIRIS
Type ‘endc’

8 Interne Orange
NR and 4G NSA KPI availability in OSIRIS (3/3)
 NSA filter present in Osiris RAN Dashboard
 Monitoring all 4G ENDC available KPIs

9 Interne Orange
 List of available 4G NSA KPIs
 The related counters are incremented in
eNb KPI 4G NSA E///
Dual Connectivity (EUTRAN-NR) Flex Counters in NSA (no more details in this
Accessibility presentation) :
EnDC_NR_RA_SCG_Failures  A new filter parameter ENDC is defined for NR NSA,
EnDC_SCG_Failure_Ratio with three levels:
EnDC_SR 0 = Counter stepped if the UE is capable of ENDC .
EnDC_END_USER_SR
EnDC_Establishment_ATT 1 = Counter stepped if the UE’s ENDC capability
EnDC_Establishment_SUCC matches the eNB .
EnDC_END_USER_Establishment_SUCC
Mobility 2 = Counter stepped if the UE has user plane
EnDC MN MCG Bearer Relocation SUCC through gNodeB, i.e. NR leg setup.
EnDC MN MCG Bearer Relocation ATT
EnDC MN MCG Bearer Relocation_SR  Using flex counters will help operator distinguish
UE_CTXT_FETCH_SR_During_Endc_Setup legacy LTE UE or ENDC UE from of all types of UE
UE_CTXT_FETCH_SR_During_Endc_Release
EnDC_NB_Received_B1_Reports
EnDC_NB_UE_Config_B1_Reports
EnDC _B1_Report_Rate
Trafic
NB_ENDC_Capables_UE_Setup

10 Interne Orange
List of available NR KPIs
 The related counters of these KPIs are incremented in the gNb
Accessibility Traffic Radio Quality
EnDC_SR Avg_RRC_connected_EnDC_Users SINR_PUCCH_AVG
RACH_Msg2_SR Max_RRC_connected_EnDC_Users SINR_PUSCH_AVG
RACH_Msg3_SR Avg_NB_of_Active_UEs_in_DL DL_MAC_BLER
Retainability Avg_Number_of_Active_UEs_in_UL UL_MAC_BLER
SCG_Radio_Resource_Retainability Max_NB_of_Active_UEs_in_UL QPSK_USAGE_RATIO
Abnormal_NR_Releases_MN_Triggered Max_NB_of_Active_UEs_in_DL 16QAM_USAGE_RATIO
Abnormal_NR_Releases_SN_Triggered NR_DL_Mac_Volume_Go 64QAM_USAGE_RATIO
Normal_NR_Releases NR_UL_Mac_Volume_Go 256QAM_USAGE_RATIO
Integrity System Utilization UL_INTERFERENCE_AVG_PUSCH
Avg_Overall_DL_Latency PDSCH_Slot_Utilization CQI_AVG
Avg_DL_MAC_DRB_Latency_per_QoS_Covering_non_DRX_in_Sync PUSCH_Slot_Utilization Availability
Avg_DL_MAC_DRB_Latency_per_QoS_Covering_DRX_in_Sync Throughput Cell Availability
DL_Packet_Loss Avg_DL_MAC_Cell_Throughput_Mbps
UL_Packet_Loss Avg_DL_MAC_UE_Throughput_Mbps
Mobility Avg_UL_MAC_UE_Throughput_Mbps
EnDC_IntraFreq_Intra_Gnb_HO_SR Avg_UL_MAC_Cell_Throughput_Mbps
EnDC_IntraFreq_Inter_Gnb_HO_SR

11 Interne Orange
Agenda

 Introduction
 Présentation des Kpis NR & 4G NSA dans
Osiris
 KPIs majeurs 4G NSA & call flow
 KPIs majeurs 5G NR & call flow
 Limitations actuelles sur les KPI/counters
 Présentation TDB : KPIs Paris vs. Lille
 Conclusion
 Annexe

12 Interne Orange
 4G NSA
[Accessibility] EnDC_SR in 4G NSA
 NR leg setup can be monitored
from the eNb

Accessibility Formula Description

1
NB_ENDC_Capa pmEndcCapableUe Total number of EN-DC capable
bles_UE_Setup UEs setup to the cell (at initial
1 setup or when UE capability is
fetched or at handover or at re- Indicating the NR leg setup
establishment criteria, add b1 measurement
for the selected NR frequency
EnDC_Establish pmEndcSetupUeAtt Number of EN-DC Setup
ment_ATT attempts initiated towards
SgNB
2 2

EnDC_Establish pmEndcSetupUeSucc Successful number of EN-DC

ment_SUCC Setup attempts.

3
EnDC_SR 100* Measures success rate of
(pmEndcSetupUeSucc / establishing E-UTRA-NR Dual
pmEndcSetupUeAtt) Connectivity (EN-DC) UE
connection with gNodeB for
UEs already connected to E- 3
UTRAN (this KPI exclude NR

13 Interne Orange
 4G NSA
[Accessibility] EnDC_B1_Report_Rate Nb of Ues
requested to
Measurement based NR Leg Setup do IRAT
1
Accessibility Formula Description measurements
on ENDC freq
EnDC_NB_UE_Config pmMeasConfigB1Endc by B1 event
_B1_Reports 1
Nb of received
EnDC_NB_Receive pmB1MeasRepEndcConfig B1 2
d_B1_Reports 2 measurement
EnDC 100*(pmB1MeasRepE The report rate for B1 reports on NR
_B1_Report_Rate ndcConfig / measurements: the freq
pmMeasConfigB1Endc proportion of Event B1
) reports per Event B1
configuration.
EnDC_NR_RA_SCG pmEndcSetupFailNrRa Number of NR RACH
_Failures failure when trying EN-DC
configuration. Stepped at
3 every received RRC
SCGFailureInformationNR
message indicating
reconfiguration with sync
failure for an SCG (CAUSE:
scg-ChangeFailure or
synchReconfigFailureSCG
)

3
14 Interne Orange
 4G NSA
Radio Link Failure
UE MeNB SgNB MME
 UE detected RLF:
 Failed random access (scg-ChangeFailure)
 T304 expiry  radio link failure UE detected RLF
 RLC UL delivery failure (RLC-MaxNumRetx)
 Number of UL RLC retransmissions
exceeds a threshold Suspend SCG
(ulMaxRetxThreshold) pmEndcSetupFailNrR
 Out of synchronization (t310-Expiry) a++
 UE monitors SSB and counts “in-synch” SCG Failure Indication NR
and “out-of-synch” indications. T310
expiry  radio link failure MeNB initiated NR Leg Release

 Network detected RLF

 RLC DL delivery failure
Network detected RLF
 Number of DL RLC retransmissions
exceeds a threshold

SgNB initiated NR Leg Release

15 Interne Orange
 4G NSA
KPI Status

 Difference in KPI EnDC_SR and EnDC_END_User_SR

Accessibility Formula Description
EnDC_SCG_Failure_Ratio 100*( PmEndcSetupFailNrRa / PMENDCSETUPUESUCC) Rach Failures ratio after ENDC
Setup success (excluding FALSE
preambles)
EnDC_END_USER_SR 100*(pmEndcSetupUeSucc – pmEndcSetupFailNrRa) / End user ENDC Success Rate
pmEndcSetupUeAtt taking into account NR Random
Access SCG Faillure
EnDC_END_USER_Establishment_SUCC pmEndcSetupUeSucc-pmEndcSetupFailNrRa Number of end user ENDC
established successes
Endc_SR 100*(pmEndcSetupUeSucc ) / pmEndcSetupUeAtt End user ENDC Success Rate
without considering NR Random
Access SCG Faillure

16 Interne Orange
 4G NSA
[Mobility] EnDC MN MCG Bearer Relocation_SR (1/2)

 After NR Leg Release, all SN terminated split bearers

are converted to MN terminated MCG bearers
 PDCP reallocations to eNb  counters to monitor the
relocations in different scenarios

Mobility Formula Description

EnDC MN MCG Bearer pmEndcRelMnMcgRel Number of attempts to start
Relocation ATT ocAtt master node terminated master
cell group bearer and PDCP
relocation for each UE when NR
leg is released.
EnDC MN MCG Bearer pmEndcRelMnMcgRel Number of successful master node
Relocation SUCC ocSucc terminated master cell group
bearer and PDCP relocations for
each UE when NR leg is released.
EnDC MN MCG Bearer 100 * Rate of successful master node
Relocation_SR (pmEndcRelMnMcgRe terminated master cell group
locSucc / bearer and PDCP relocations for
pmEndcRelMnMcgRel each UE when NR leg is released.
ocAtt)

17 Interne Orange
 4G NSA
[Mobility] EnDC MN MCG Bearer Relocation_SR (2/2)
NR Release event
 Monitoring relocations in the target eNb when NR Leg release takes in place
 EnDC MN MCG Bearer Relocation_SR
= 100 * (pmEndcRelMnMcgRelocSucc /pmEndcRelMnMcgRelocAtt)

pmEndcRelMnMcgRelocA pmEndcRelMnMcgRelocS 2
tt 1 ucc
eNodeB receives an NR eNodeB receives ERAB
1 leg release event and NR Modification Confirm
leg release procedure is message from MME
triggered by event during NR leg release
procedure
eNodeB receives Context eNodeB receives RRC
Fetch Response message Connection
with NR-PDCP Reconfiguration
configuration Complete message for
relocation to E-UTRAN
PDCP after successful
RRC Reestablishment
procedure

2 an intra-cell handover eNodeB receives ERAB

procedure with NR leg Modification Confirm
release is triggered message from MME
18 Interne Orange
during intra-cell
handover procedure with
 4G NSA
[Mobility] UE_CTXT_FETCH_SR_During_Endc_Setup or Release
 RRC Reestablishment procedures during on-going procedures, NR Leg Setup or Release, are supported
 Monitor UE context fetch procedure during ongoing through counters incremented in source eNb, where the
previous UE context is stored:
 pmUeCtxtFetchAttEndcSetup 1 pmUeCtxtFetchAttEndcRel 2

 pmUeCtxtFetchSuccEndcSetup 3 pmUeCtxtFetchSuccEndcRel 4

Mobility Formula Description

UE_CTXT_FETCH_SR_D 100* Rate of UE context

uring_Endc_Setup (pmUeCtxtFetchSucc fetch procedures 1 2
EndcSetup / successfully completed
pmUeCtxtFetchAttE during EN-DC setup
ndcSetup)

UE_CTXT_FETCH_SR_D 100*(pmUeCtxtFetc Rate of UE context

uring_Endc_Release hSuccEndcRel/ fetch procedures
pmUeCtxtFetchAttE successfully completed 3 4
ndcRel) during EN-DC release.

19 Interne Orange
Agenda

 Introduction
 Présentation des Kpis NR & 4G NSA
dans Osiris
 KPIs majeurs 4G NSA & call flow
 KPIs majeurs 5G NR & call flow
 Limitations actuelles sur les KPI/counters
 Présentation TDB : KPIs Paris vs. Lille
 Conclusion
 Annexe

20 Interne Orange
 [Accessibility] EN-DC Setup Success Rate (X2) NR
Accessibility Formula Description UE MeNB SgNB MME

100* Measures success rate of RRC: B1 Measurement

(pmEndcSetupUeSu establishing EN-DC UE connection Report
EnDC_SR
cc / with gNodeB for UEs already
pmEndcSetupUeAtt) connected to E-UTRAN

X2AP:
1
 ENDC setup process is the 1st step towards an NR SGNB ADDITION REQUEST
1
service
 UE attached to 4G network signals the support for X2AP:
SGNB ADDITION REQUEST ACKNOWLEDGE
dual connectivity
 When instructed, the UE starts reporting X2AP:
SN STATUS TRANSFER
measurements on 5G neighbors RRCConnectionReco
 The KPI ENDC SR can be calculated on eNb and nfiguration
gNb level

RRCConnectionRec X2AP: 2
pmEndcSetupUeAtt 1 onfigurationCompl2 SGNB RECONFIGURATION COMPLETE
ete S1AP:
pmEndcSetupUeSucc 2 E-RAB MODIFICATION INDICATION

S1AP:
E-RAB MODIFICATION CONFIRM

21 Interne Orange
[Accessibility ] Random Access Success Rate NR
Accessibility Formula Description
100 * pmRadioRaCbAttMsg2 /
RACH_Msg2_SR Successfully replied Msg1 (include false preambles )
pmRadioRaCbPreambles
100 * (pmRadioRaCbSuccMsg3 / Describes how often transmitted RaMsg2 results in
RACH_Msg3_SR
pmRadioRaCbAttMsg2 ) successfully received RaMsg3

MME MeNB SgNB UE

 UE accesses the 5G network using the
contention based random access Procedure of EN-DC leg setup
procedure
 UE then maintains simultaneous dual Msg1 Random Access Preamble
connectivity to the 4G eNB and the 5G 1
gNB
 Msg2, Random Access Response
The main gNodeB counters: through PDCCH
1 2
 pmRadioRaCbPreambles Msg3, RRC Setup Request
2 through PUSCH
3
 pmRadioRaCbAttMsg2 Contention resolution grant
S1AP E-RAB Modification Indication
3
 pmRadioRaCbSuccMsg3 S1AP E-RAB Modification Confirmation
22 Interne Orange
 KPI Status NR
Degraded KPI Msg3_SR
High #of preambles and msg2
Reason : Detection of a huge number of
false (phantom) cb preambles due to a low
value in the gNb for threshold for preamble RP139
change
detection
Action done : Change of the value of
system constant RP139.
Old value 10
New value 20 (since 20/02)
Result : Improvement of the KPI Msg3_SR
and significant decrease of the detected
preambles and msg2
Note: More significant improvement seen in
Paris than in Lille. In Lille, one cluster
impacts the global KPI of the whole city

23 Interne Orange
[Retainability] SCG_Radio_Resource_Retainability NR
 SCG Radio Resource Retainability measures the percentage of established NR connections
for E-UTRA-NR Dual Connectivity (EN-DC) UEs that are lost with an abnormal release
 EN-DC connection (NR Leg) release can be initiated by:
 eNb
 gNb

Retainability Formula Description

100 * (pmEndcRelUeAbnormalMenb + Measures percentage of established NR connections
SCG_Radio_Resource_Retainability pmEndcRelUeAbnormalSgnb) / (pmEndcRelUeNormal + for E-UTRA-NR Dual Connectivity (EN-DC) UEs that
pmEndcRelUeAbnormalMenb + are lost with an abnormal release. In this case, only
pmEndcRelUeAbnormalSgnb) NR service is lost

Number of abnormal E-UTRA-NR Dual Connectivity

Abnormal_NR_Releases_MN_Triggered pmEndcRelUeAbnormalMenb (EN-DC) NR leg releases for each cell triggered by
master eNodeB.
Number of abnormal E-UTRA-NR Dual Connectivity
Abnormal_NR_Releases_SN_Triggered pmEndcRelUeAbnormalSgnb (EN-DC) NR leg releases for each cell triggered by
secondary gNodeB.

Normal_NR_Releases pmEndcRelUeNormal Number of normal E-UTRA-NR Dual Connectivity

(EN-DC) NR leg releases for each cell.

24 Interne Orange
 EN-DC connection release flow chart NR

Release by eNb
Release by gNb
pmEndcRelUeAbnormalSgnb

Normal release
cause:
pmEndcRelUeNormal pmEndcRelUeAbnormalMenb
• Cell not
Available
• User Inactivity
• MCG Mobility
• SCG Mobility

X2AP: UE Context release

X2AP: UE Context release

25 Interne Orange
KPI status NR

 SCG radio resource retainability status of VO5G IDF

in function of volume of eNb and gNb releases
 pmEndcRelUeAbnormalMenb >>
RP139
pmEndcRelUeAbnormalSgnb change
 pmEndcSetupFailNrRa included in
pmEndcRelUeAbnormalMenb

 No additional counters on Gnodeb available to

investigate the reason of the drops for the moment.

 CSR is open from E/// Side to improve the abnormal

releases counters in coming SW releases
 Actions done with objective to improve SCG Radio
Resource Retainability and RACH SR:
 RP139 from 10 -> 20
 Rachroot sequence change
 B1 threshold optimisation.

26 Interne Orange
[Mobility] ENDC Intra/Inter gNb HO SR NR
Two types of Intra-frequency HO:

 Success rate of intra-sgNodeB Primary

Secondary Cell (PSCell) change in sgNodeB for
EN-DC UE connections
 Success rate of inter-sgNodeB Primary
Secondary Cell (PSCell) change in source Reported NR PCI is
a known neighbor
sgNodeB for ENDC UE connections
cell Reported NR PCI is a
 Incoming HO formula will be added
unknown cell
Mobility Formula

EnDC_IntraFreq_Intra_Gnb_HO_SR 100*(pmEndcPSCellChangeSuccIntraSgnb
pmEndcPSCellChangeAttIntraSgnb)
/

EnDC_IntraFreq_Inter_Gnb_HO_SR 100*(pmEndcPSCellChangeSuccInterSgnb
pmEndcPSCellChangeAttInterSgnb)
/

27 Interne Orange
 Intra-freq intra gNB mobility flow chart NR
EnDC_IntraFreq_Intra_Gnb_HO_SR

The main gNodeB counters for this scenario:

 pmEndcPSCellChangeAttIntraSgnb 1

Number of intra-sgNodeB Primary Secondary Cell

(PSCell) change attempts for EN-DC in source NR cell
 pmEndcPSCellChangeSuccIntraSgnb 2 Evaluation of A3
measurement report
Number of successful intra-sgNodeB Primary
Secondary Cell (PSCell) changes for EN-DC in
source NR cell Sending the modified
SCG configuration to
the UE on SRB1
1

2
UE synchronizes the target
NR PSCell and performs NR
random access

28 Interne Orange
 Intra-freq inter gNB mobility flow chart NR
EnDC_IntraFreq_Inter_Gnb_HO_SR

The main gNodeB counters:

1
1 pmEndcPSCellChangeAttInterSgnb
3
2 pmEndcPSCellChangeSuccInterSgnb
4

2
Synchronous User-data
SCG reconf suspended

User-data
3 pmEndcPSCellChangeResAllocReqInterSgnb resumed

4 pmEndcPSCellChangeResAllocSuccInterSgnb
NR User-data
resumed in target
PSCell
29 Interne Orange
 [Throughput] Average DL MAC Cell Throughput NR
THROUGHPUT Formula Description
Normalized throughput against traffic. Slots where no
Avg_DL_MAC_Cell_Throughput_Mbps 64 * pmMacVolDl / (pmPdschSchedActivity * 1000)
traffic is taking place are excluded

 pmMacVolDl
Aggregated volume of successfully transmitted DL
MAC PDUs for DRB or SRB using cell as PCell or
SCell. Unit: bytes Scheduled time
All successfully transmitted DL MAC PDUs for DRB
or SRB from cell are included.
 pmPdschSchedActivity
Total duration of slots where there is DL PDSCH
scheduling activity. Unit: 125 us Available time
Stepped for each slot where there is at least one
SRB or DRB PDSCH scheduling activity. Other It is a good indicator of transmission performance
scheduling activities (Msg2 or Msg4, paging, SI, and when data is scheduled on DL PDSCH
so on) have to be excluded.

30 Interne Orange
[Throughput] Average UL MAC Cell Throughput NR

THROUGHPUT Formula Description

Normalized throughput against traffic. Slots where no
Avg_UL_MAC_Cell_Throughput_Mbps 64 * pmMacVolUl / (pmPuschSchedActivity * 1000)
traffic is taking place are excluded

 pmMacVolUl
Data volume in MAC SDUs successfully received Scheduled time
in UL. Unit: byte
Stepped when data is successfully received
from DRBs and SRBs when cell is used as PCell
or SCell.
 pmPuschSchedActivity
Total duration of slots where at least one MAC
Available time
SDU is received on PUSCH. Unit: 125 us
Stepped for each slot where at least one MAC
SDU is successfully received It is a good indicator of transmission
performance when data is scheduled on UL
PUSCH

31 Interne Orange
[Throughput] Average DL MAC UE Throughput NR
THROUGHPUT Formula Description
Avg_DL_MAC_UE_Throughput_Mbps 64 * pmMacVolDlDrb / (pmMacTimeDlDrb * Average DL MAC DRB Throughput Captured in gNodeB
1000)

 pmMacVolDlDrb - Aggregated volume of downlink MAC SDUs

transmitted for HARQ initial transmission for each DRB.
 pmMacTimeDlDrb - Aggregated time for HARQ initial
transmission scheduling of downlink DRB since last
scheduled DRB
 The volume counters increment at first transmission
time
 No HARQ retransmission volume is included
 The time counters however includes the time for HARQ
retransmission
 In the case of Carrier Aggregation, increment is done as a
merged view across all carriers used for the DRB

32 Interne Orange
 [Throughput] Average UL MAC UE Throughput NR
THROUGHPUT Formula Description

Avg_UL_MAC_UE_Throughput 64 * pmMacVolUlResUe / UL MAC UE throughput for data bursts

_Mbps (pmMacTimeUlResUe * 1000) restricted by air interface

 pmMacVolUlResUe - Data volume in MAC SDUs

successfully received in UL as part of a burst while being
restricted by air interface, excluding data volume
received in last slot in burst
 pmMacTimeUlResUe - Transfer time of restricted UL
data volume. Unit: 125 us
 The volume is incremented when positive feedback
HARQ-ACK is received for each transmission
 The time counters however includes the time for
HARQ retransmission

Slots between 9 and

16 show when the UE
sends data and
gNodeB decodes its
successful reception
(CRC OK)
33 Interne Orange
[System Utilization] PSDCH/PUSCH Slot Utilization NR

System Utilization Formula Description

The utilization of the Physical Downlink Shared Channel
PDSCH_Slot_Utilization 100 * (pmPdschSchedActivity / pmPdschAvailTime)
(PDSCH)
100 * (pmPuschSchedActivity / The utilization of the Physical Uplink Shared Channel
PUSCH_Slot_Utilization
pmPuschAvailTime) (PUSCH).

UL Scheduled time

DL Scheduled time

DL Available time

UL Available time

34 Interne Orange
Agenda

 Introduction
 Présentation des Kpis NR & 4G NSA
dans Osiris
 KPIs majeurs 4G NSA & call flow
 KPIs majeurs 5G NR & call flow
 Limitations actuelles sur les
KPI/counters
 Présentation TDB : KPIs Paris vs. Lille
 Conclusion
 Annexe

35 Interne Orange
 Limitations on KPIs/counters
Blocking point Replacement/Action
NR PDCP Volume and Not available on cell level. PDCP Counters for PDCP volume on CUUP
should be measured in CUUP  part of gNB, on MO class
throughput  there is no cell knowledge, and RpUserPlaneLink (per F1-U NR Leg
no contact with RLC scheduler  and X2-U LTE Leg).
not possible to calculate time Counters available:
component for the throughput pmPdcpVolRecUl5qi and
pmPdcpVolTransDl5qi

SCG Radio Resouce Not correct representation of real pmEndcSetupFailNrRa should be

NR drop. excluded from the formula – CSR
Retainability  The abnormal releases initiated opened
by eNb includes the
pmEndcSetupFailNrRa (synchro
reconfigurations issues)

B1 report rate  Stepped at receveid B1 report

with repetition and ratio >>100%
correction requested is to take into
account only first B1 report
in some cells

KPI for Mobility  Available KPI only on cell level

and not on cell relation
Future development for this KPI to
be on cell relation level

36
Agenda

 Introduction
 Présentation des Kpis NR & 4G NSA dans
Osiris
 KPIs majeurs 4G NSA & call flow
 KPIs majeurs 5G NR & call flow
 Limitations actuelles sur les KPI/counters
 Présentation TDB : KPIs Paris vs Lille
 Conclusion
 Annexe

37 Interne Orange
 TDB :
PSE@4G_NSA_Daily_Dashbord
(disponible dans Osiris Query / Stats sur 700
ENDC)

38 Interne Orange
39 Interne Orange
LILLE Paris
40 Interne Orange

LILLE Paris
 TDB : PSE@5G_Daily_Dashbord
(disponible dans Osiris Query / Stats sur NR TDD 3.5
Ghz)

41 Interne Orange
42 Interne Orange

LILLE Paris
43 Interne Orange
LILLE Paris
44 Interne Orange
LILLE Paris
 LILLE Paris

45 Interne Orange
 LILLE Paris

46 Interne Orange
47 Interne Orange
LILLE Paris
 Conclusion
 Grille KPIs
-Première Grille de KPIs majeurs NSA (EnB/Gnb) disponibles

(qualifications en cours de finalisation sur
quelques indicateurs NR )

-TDBs dispos dans Osiris Query , à enrichir par la suite.

- Limitations sur quelques compteurs majeures (Volume NR PDCP , NR Drop) , améliorations
attendues
surPerformances
 Suivi

les prochains :paliers Sw.

-En archi NSA , un suivi des indicateurs sur 4G NSA et 5G est requis .
-La première grille de Kpis permet de dégager une première tendance entre les
performances
Paris vs Lille . (meilleur débit moyen en DL à paris..) .
-KPI Débit user moyen en DL est ~ 300mbps , proche du débit moyen observé sur les
drives.
-UL Interferences du 3.5Ghz à vide est ~ -115dbm.

 A venir :

-Flex counters avec filtres ENDC.

48
-Enrichir la grille existante avec les prochains paliers et features.
Interne Orange
-Mise en place des seuils de détection ( trafic , interférences , ENDC-SR , mobilité )
 Annexe

49
 KPI Framework

 In NSA 3x, fundamental to guarantee the system performance are the Dual Connectivity
and 4G-5G interworking
 KPIs:
 Accessibility, Retainability and Mobility will rely on LTE/NR performance
 Integrity (Latency and Throughput DL / UL) are the main KPIs influenced by the NR
Network and should be measured for EN-DC UE using LTE plus NR leg. If NR is not
available, then only LTE will be considered for the KPI.
 Initial tuning phase  drive-test using an EN-DC UE shall be considered as the main
measurement method as the network might not have enough load to be evaluated fully
only based on the network counters
 On final tuning phase (loaded network scenario) – network counters will be mainly used to
evaluate the performance

50
 4G NSA
[Mobility] EnDC MN MCG Bearer Relocation_SR
RRC re-establishment
 Monitoring relocations in the target eNb when NR Leg release takes in place
 EnDC MN MCG Bearer Relocation_SR
= 100 * (pmEndcRelMnMcgRelocSucc /pmEndcRelMnMcgRelocAtt)
pmEndcRelMnMcgRelocAtt pmEndcRelMnMcgRelocSucc

NR leg release event and ERAB Modification Confirm

NR leg release procedure message from MME during
1 is triggered by event NR leg release procedure
1 RRC Connection 2
Context Fetch Response Reconfiguration Complete
message with NR-PDCP message for relocation to E-
configuration (RRC UTRAN PDCP after
reestablishment) successful RRC
Reestablishment procedure
When primary eNodeB
receives ERAB Modification
An intra-cell handover
Confirm message from MME
procedure with NR leg
2 during intra-cell handover
release is triggered
procedure with NR
leg release
51 Interne Orange
[Traffic] Average RRC Connected Users (1/2) NR
(ENDC UE Connections)
TRAFFIC Formula Description

Avg_RRC_connected_EnDC_ pmRrcConnLevelSumEnDc / Average Number of RRC connected Internal event

Users pmRrcConnLevelSamp Users for sampling of
the number of
Max_RRC_connected_EnDC_ Peak number of NR EN-DC UEs in NR ENDC UE in
Users pmRrcConnLevelMaxEnDc RRC_CONNECTED mode in NR CU-CP. RRC connected Each
mode 5sec

 On gNb side, in NR NSA

Internal event maintains
there is no presence of current number of NR EN-DC +1 pmRrcConnLevelSamp
actual RRC connection setup UEs in RRC_CONNECTED mode
 Follow up of #rrc endc in NR CU-CP
connected UEs is based on The aggregated counts of RRC + (sum)
internal event for ENDC UE Connected Users are increased pmRrcConnLevelSumEnDc
by the sum of the RRC
connections
Connected Users present when
 This KPI can contribute to the sample is taken (max)
pmRrcConnLevelMaxEnDc
investigation on
Counter is updated at the end
increase/decrease of traffic of ROP with maximum value
during ROP period
52 Interne Orange
 [Traffic] Average RRC Connected Users (2/2) NR
(ENDC UE Connections)

 Example of counter value in 3

different ROPs

 Availability of the counters on

CellCU level, and on CU-CP level

53 Interne Orange
 NR
[Traffic] Average/Max Number or active UEs in UL/DL
Active UE Each slot
instances in DL (0.5ms)
or UL direction
TRAFFIC Formula Description (UE has
The average value of the number of DRB/SRB data
UEs with data in buffer in downlink
Avg_NB_of_Active_UEs_in_ pmActiveUeDlSum / direction. This is defined by the ratio in his buffer)
UL/DL
DL pmActiveUeDlSamp of UEs with data in buffer to the
number of TTIs in the observation
period. Total number of DL or UL
samples made for
The average value of the number of pmActiveUeDlSum or +1 pmActiveUeDlSamp
UEs with data in buffer in Uplink pmActiveUeUlSum. Max value +1 pmActiveUeUlSamp
Avg_Number_of_Active_UE pmActiveUeUlSum / direction. This is defined by the ratio
s_in_UL pmActiveUeUlSamp of UEs with data in buffer to the reached at the end of the ROP
number of TTIs in the observation 1800000.
period.
+ (sum)
pmActiveUeDlSum
Peak number of UE instances Aggregated number of active
Max_NB_of_Active_UEs_in_ considered active in UL direction. + (sum)
UL pmActiveUeUlMax These UE instances have data in DRB
UE instances
pmActiveUeUlSum
or SRB.
Each sample is an average of
(max) pmActiveUeDlSum /
active UE instance. The
Peak number of UE instances pmActiveUeDlSamp
counter is set to the maximum
Max_NB_of_Active_UEs_in_ considered active in DL direction.
pmActiveUeDlMax of all average samples made
DL These UE instances have data in DRB (max) pmActiveUeUlSum /
only. over ROP pmActiveUeUlSamp

54
 [Integrity] DL MAC Latency NR
Integrity Formula Description
(pmMacLatTimeDlDrxSyncQos +
Avg_Overall_DL_Latency pmMacLatTimeDlNoDrxSyncQos) / (8 * Average Overall DL Latency
(pmMacLatTimeDlDrxSyncSampQos +
pmMacLatTimeDlNoDrxSyncSampQos)) Empty
buffer
Avg_DL_MAC_DRB_Latenc
y_per_QoS_Covering_non_ (pmMacLatTimeDlNoDrxSyncQos / (8 * Average DL MAC DRB Latency per QoS
DRX_in_Sync pmMacLatTimeDlNoDrxSyncSampQos)) Covering non-DRX in-Sync

Avg_DL_MAC_DRB_Latenc Packet arrival

(pmMacLatTimeDlDrxSyncQos / (8 * Average DL MAC DRB Latency per QoS
y_per_QoS_Covering_DRX_
in_Sync pmMacLatTimeDlDrxSyncSampQos)) Covering DRX in-Sync

UE
state

 DL MAC latency measures MAC scheduling latency from the

time when packet arrives in empty DL buffer to the time when
first packet is transmitted DRX_ACTIVE DRX_SLEEP
 The main counters for this : & UL in-sync & UL in-sync
 pmMacLatTimeDlNoDrxSyncQos 1
 pmMacLatTimeDlNoDrxSyncSampQos 1
 +pmMacLatTime +pmMacLatTime
pmMacLatTimeDlDrxSyncQos 2 DlNoDrxSyncQos DlDrxSyncQos
 pmMacLatTimeDlDrxSyncSampQos 2
55 Interne Orange +pmMacLatTimeDlN +pmMacLatTimeDlD
oDrxSyncSampQos rxSyncSampQos
Merci