Scribd
Upload
514 views

E2E KPI Monitoring

This document discusses strategies for monitoring end-to-end (E2E) data performance on mobile networks. It addresses why performance should be measured, how it can be assessed using various tools, and how "Big Data" can help analyze performance metrics. Key points include: measuring performance ensures quality of service and experience for customers; a combination of network KPIs, probes, and crowd-sourced data provides insights; performance should be evaluated against targets, competitors, and evolving customer expectations; and consolidating diverse data sources through "Big Data" tools allows more comprehensive network optimization and marketing strategies.

Uploaded by

Frensel Petrona
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
514 views

E2E KPI Monitoring

This document discusses strategies for monitoring end-to-end (E2E) data performance on mobile networks. It addresses why performance should be measured, how it can be assessed using various tools, and how "Big Data" can help analyze performance metrics. Key points include: measuring performance ensures quality of service and experience for customers; a combination of network KPIs, probes, and crowd-sourced data provides insights; performance should be evaluated against targets, competitors, and evolving customer expectations; and consolidating diverse data sources through "Big Data" tools allows more comprehensive network optimization and marketing strategies.

Uploaded by

Frensel Petrona
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 19

Determining how to

carry out accurate


monitoring of E2E
data performance
Mobile Network Performance
Management
London June 2014
Martin Harris
Orange Corporate Services Ltd
[email protected]

Contents
Why measure data service performance and what
is the benefit?

examining strategies to benchmark and monitor


end to end data performance against competitors
or other countries

ensuring performance is in line with promises

Monitoring end-to-end service performance

understanding the benefits and trade-offs between


the use of network KPIs, probes, robots, drive
tests, network conters

How do we assess performance?

looking at how and where to set performance


targets, what impacts the choice, and how these
evolve for LTE

examining the customer experience by using


speed tests (crowd sourced data) which can give
a specific view of QoS

How can Big Data help us?

or is this just another complexity?


2

Why measure service performance?


To verify performance is in line with promises given
to our customers

business offers and customer expectations

To understand how our networks (and competitor


networks) are performing

best network and meeting customer expectations

identify weaknesses (bottlenecks, slow speed,


packet loss, capacity, interconnect, outages)

ensure we meet SLAs with business partners

optimise expenditure, and radio spectrum use

To provide a good quality of experience

keeping customers, preventing churning

capturing new customers by good performance

ensuring services perform as required

To bridge the gap between network performance


and quality of experience

ensure that good network performance means


good QoE for customers
3

What is service performance?


Traditional data KPI measurements

network KPIs: drop session rate, session setup


success rate, congested cell ratio, cell loading

E2E KPIs (robots): data rates, web page, latency

used to ensure good network performance

insufficient to reflect quality of experience

Services run on top of the networks

monitor the real service performance

what the customers are doing


web browsing, YouTube, video, messaging,gaming

where they are doing it


indoors, at home, in the office, on the tube
challenges to measure in these locations

while respecting privacy and confidentiality

E2E performance depends on the other end and


third part connectivity

often outside the operators network

expectation of network operator responsibility


4

Performance has to be measurable


Network counters, passive probes

large volume of data from the complete network

high level overview or focus on specific areas

modern probes enable us to drill down to:


specific service streams
individuals or identified user groups
device types

What changes for LTE?

basic principles remain the same

upgrade of tools, robots, probes

extended to cover eNodeB, EPC, IMS etc

How to assess network and service performance?

against targets or objectives

against customer expectations?

against local competitors?

against other operators in the same group?

Traditional data KPI measurements


Monitor KPIs or a
regular basis, look at
trends, and keep them
under control

Variable
performance

Coming under
control here

Further investment
may be needed to
reduce congested
cells
Define congested
cells to meet your
own requirements
Track performance
by region to identify
any poorly
performing areas

Ensure good network performance = good quality of experience


6

Traditional data rate measurements


Measure where the customer is located
business areas, transport hubs
hotels, residential, indoors, etc
major challenge

Ensure consistent measurement method


drive tests, robots, stationary or mobility
understand the effect of TCP slow start

Ensure measurement tools up to date


capable of maximum data rates
ensure all tools are LTE capable

Data rates compared by operator, country,


region, vendor

Average data rate at


85% of instantaneous
data rate

TIP: larger file sizes are generally used for measuring LTE data rates, but these can take
excessive time if the terminal connects to 3G or 2G network; to avoid this measure the
data rate over a fixed time, e.g. 10-20 seconds and stop the transfer after this time

Data rate evolution over two years


7

Crowd sourcing an alternative solution


What is crowd sourcing?
a fresh way to look at network performance
from the customers perspective

Collected from apps/agents on handsets


different locations, users, handsets
tens of thousands of measurements

Very consistent data month on month


but do customers only perform these tests
when they have a network problem ?

Currently limited to traditional KPIs


data rate and latency

More advanced applications available


passive (silent) or active (on-click) monitoring
web page and video performance

friendly users
benchmarking - alternative to drive tests
capability to test indoors

How can we assess service performance?


Assessment of performance

What can we measure?

do we assess against targets?


how do we establish targets?
against aspirations, customer
expectations, local competitors,
other operators?
should targets be different for LTE?
can we give performance a number?
e.g. 25% = half target speed
50% = on target
100% = twice as good as target

data rate and latency


video services: access success rate,
speed to start playout, image quality
(pauses, lost frames)
web services: download success
rate / time; DNS response
other services: instant messaging,
specific OTT services
over a mix of locations

Example of weighted service performance


Service

Major service
weighing

Web page download


performance

40%

Download data rate

20%

Upload data rate

10%

Video performance
DNS access time
Latency (Round Trip Time)

20%
5%
5%

Individual services
Reference web page (Kepler)
Local web pages
International web pages
Indoors
Large cities (drive test)
Small cities (drive test)
Interconnecting routes
Indoors
Large cities (drive test)
Small cities (drive test)
Interconnecting routes

25%
37.5%
37.5%
40%
30%
20%
10%
40%
30%
20%
10%
20%
5%
5%

Overall service
weighting
10%
15%
15%
8%
6%
4%
2%
8%
6%
4%
2%
20%
5%
5%
9

Service performance versus customer experience?


We must not forget to measure actual customer experience
regular surveys of randomly selected customers
performed by third party, unbiased, covering all operators
telling us what the customer feels about our network performance
Results of recent (4Q-2013)
customer experience surveys for
five European operators show a
high level of correlation against
weighted service performance
However there is often a lag in
customer perception, with
customers remembering bad
experiences

Can we use this to better effect to improve customer experience?


Can Big Data help us to ?

10

How can Big Data help us?


Big data - a consolidated tool for customer experience management
making better use of all network performance indicators and correlating these with
other information, e.g. from customer services, billing

to improve network performance

data capture

to better understand the customer experience

QoS

internet

opportunities:

to provide customers with new offers

LTE
processing
use
probes
cases
services

storage

KPIs
latency

Big Data

benefits:
new value propositions

tools

data rates
IMS

processes

database

consolidation

social networks

QoE

customer retention and base expansion

analysis

3G

customer service optimization


QoS/QoE management and network optimisation
Portal and presentation layer
Data processing and storage

Mediation layer

Probes

Probes

Mobile usage

CRM data

Service description

Network description

IS sources (customer data, service description, network description)


11

Big Data use cases

Already proven benefits for both network operations and marketing


multiple use cases; each country can focus on those important to the business
examples:
15 complaints in one area highlighted that 600 customers had problems,
corrective action was put in place to improve performance
devices with high signalling volume, again able to take corrective action
12

Big Data network optimization - traffic


Traffic by application
highest traffic volume is web,
followed by streaming, file
download, mail, P2P

Traffic by service
provider
highest single traffic provider is
YouTube, with Facebook,
Apple, Orange portal, Skype
also providing high levels of
traffic

13

Big Data network optimization traffic by app & user

Top 10 users per day


7% of traffic from 10 users
can also identify usage by
application, e.g. web,
download, streaming
used to understand cause of
high load and better manage
the traffic
allows marketing to target high
users with upgrades
14

Big Data traffic by terminal type


Traffic volume by terminal brand
Traffic figures show almost half the traffic is on Samsung
or Apple devices. Further drill down could be performed if
necessary to show the precise terminal types.

Traffic by terminal type (smartphone, USB,mobile,


tablet)
Figures show 78% of mobile data usage is now on
smartphone, with less than 10% on USB modems.
Note the low volume of mobile data traffic on tablets as
these probably use Wi-Fi by preference.

15

Big Data QoS/QoE management IP sessions


IP session QoS
time for session
establishment (between
first user request and
first downlink packet)
where the time for
session establishment is
long, there is a need to
identify any single cause

Video performance
(RTSP)
packet desynchronisations

lost packets

16

Big Data QoS/QoE management - throughput


Throughput
distribution
note that the low data
rate sessions can be
caused by a the number
of low volume transfers

Percentage of
retransmitted packets
monitor performance,
review trends, identify
source of retransmissions

17

Summary
Traditional measurements of QoS obtained
from network performance
still applicable today, but not the whole story

LTE
increased data and service usage
Services not networks
emphasis on service measurement
greater need to focus on customer

New tools available to the operator


better understanding of the customer
experience of our data services

Crowd sourcing
measuring the service performance where the
customer is located

Big data
making better use of the data available
providing a high level overview or drilling down
to target specific issues
18

THANK YOU

Martin Harris
Orange Corporate Services Ltd
[email protected]

19

You might also like