Case Summary of “Help Customer Business Success”:

Churn Improvement Successful Practices

in Operator V

Mikhail Rysev Chernovsky Yury Wangguoxian Kovalchuk Alexander Geliang

445703 278407 406109 741896 644325
Survey: level of service is key reason for churn and might lead to general
deterioration of perception due to social snow-ball effect
What was your main reason With whom did you share your decision to leave the
for leaving the company? company due to its level of customer service?*
I told some friends, family
or colleagues
72,5%
18,7%
I didn’t share it 22,3%

I shared it with my friends

and followers on social media
21,5%

17,2% I shared it on the company’s

offical social media or website
11,5%
64,0%
I shared it in an online forum
not owned by the company
10,2%

I shared it with the media 4,2%

Other 3,1%
I hadn’t been happy with their level of service for some time
I simply found out about a company offering better service without
having trouble with the previous company Most of Subscribers who are not satisfied with service quality
I had a crisis with the company as a result of a specific issue shares their negative perception via different social channels

Source: TechSee | 2019 Customer Churn in the Telecom Industry Survey Results
Customer Challenges and Target Alignment
13,8%
 Customer V has highest 12,0%
12,8%
10,6% 10,8%
12,4%

churn level on R market Churn V

9,5% 9,0%
8,1% 8,1% 8,1% Churn T
7,2%

1Q19

2Q19

3Q19

4Q19

1Q20

2Q20

3Q20

4Q20

1Q21

2Q21

3Q21

4Q21
 Huawei proposed V ML based Churn prediction consultancy that allowed to
reduce churn by 0.8% for operator A in M
Step 1: Step 2: Step 3: Step 4:
CRM and CEM data collection Data Cleaning and preprocessing ML model training on historical data Future churners prediction
13.12.2020
20.12.2020
27.12.2020
03.01.2021
10.01.2021
17.01.2021
24.01.2021
31.01.2021
07.02.2021
14.02.2021
21.02.2021
28.02.2021
07.03.2021
14.03.2021
21.03.2021
28.03.2021
04.04.2021
11.04.2021
18.04.2021
25.04.2021
02.05.2021
09.05.2021
16.05.2021
23.05.2021
30.05.2021
06.06.2021
13.06.2021
20.06.2021
27.06.2021
04.07.2021
11.07.2021
18.07.2021
25.07.2021
01.08.2021
D1: Train group
D5:
User
D3:
T
T
features Churn
1 3
Usage Channel T

Test group fact

2
T
D2: T ML model
Product
D6: D4:
T
T features
Service Value
T

Experience D7:
T
T Existing historical data
Network
experience D9: Future
D8: Up to date data
Competition
Network churn?
Basic cell level
Info Week -N … Week -1 NOW Week K

 V already implemented ML Churn prediction but due to heavy engagement agreed to

check our model efficiency over their in-house model
 Machine Learning based Churn Prediction consultancy plan –
clear defining of Huawei and V responsibilities for E2E project
Project plan
Month 1 Month 2 Month 3
Activity 01 02 03 04 05 06 07 08 09 10 11 12 …
1. Churn insight and solution clarification Huawei
2. Data collection Operator
3. Churn prediction modeling and report Huawei
4. Retention campaign design and execution Operator
5. Campaign efficiency analysis Jointly Jointly
6. Technology migration Jointly

Data collection requirements: CRM and CEM data* per user level with weekly aggregation for 6…8 months
D-1 D-2 D-3 D-4 D-5 D-6 D-7 D-8 D-9
Dimension Service Network Network
User Product Channel Value Usage Competition
Experience Experience (cell level)
Tenantry Plan Online ARPU Voice Complaints Location Visit competitor Coverage
Variables
Social- Life Cycle POS Recharge SMS Query Roaming Call competitor Capacity
categories demographic
examples Package Cash credit Payment Data Satisfaction Experience Site information
Membership Terminal Apps Survey
* Model will be customized according to data availability on Customer side, but more dimensions lead to better churn prediction accuracy
Data collection from V and Sample preparation
 17 Nov 2021 V shared with Huawei sensitive CRM data of full country R Subscriber base
with life time >12 months
58.3 GB 19 mln. Subs 34 weeks 55 features
Initial data Select region: Select samples X for model training and
M + S regions split for train, test verify

• Randomly select X Subscribers id from all

partitions (read only id column for
memory optimization)
…
• Make sample X from selected Subscribers
• From each week select all data for
sample id’s with splitting by chunks for
co,putting resource optimization

…
…
 Data clearing and preprocessing, sliding window datasets generations
for ML model input
Sliding windows for analyzed data set (10 iterations)
• Model is trained based on 12 weeks time windows
13.12.2020
20.12.2020
27.12.2020
03.01.2021
10.01.2021
17.01.2021
24.01.2021
31.01.2021
07.02.2021
14.02.2021
21.02.2021
28.02.2021
07.03.2021
14.03.2021
21.03.2021
28.03.2021
04.04.2021
11.04.2021
18.04.2021
25.04.2021
02.05.2021
09.05.2021
16.05.2021
23.05.2021
30.05.2021
06.06.2021
13.06.2021
20.06.2021
27.06.2021
04.07.2021
11.07.2021
18.07.2021
25.07.2021
01.08.2021
• Current week
• Target event for training: last week with data is earlier than
T 6 weeks after current week:
T
T • Target event segmentation:
T
Before < -3 weeks Dropped
T
T Near before -3 weeks…current week Used for training and
T strong verification
Near after Current week…+6 weeks
T
T Future > +6 weeks Light verification
T
• T – Subscriber should have traffic on this week or after
Sliding window target event statistics
Churn flag quantity Churn flag share in total sample
Current Total
Before Near Before Near
period Subscribers Near after Future Total Near after Future Total
(dropped) before (dropped) before
07.03.2021 217 807 5 098 2 154 5 031 7 045 19 328 2,34% 0,99% 2,31% 3,23% 8,87%
14.03.2021 217 854 5 456 2 253 5 008 6 413 19 130 2,50% 1,03% 2,30% 2,94% 8,78%
21.03.2021 217 784 5 738 2 401 4 932 5 775 18 846 2,63% 1,10% 2,26% 2,65% 8,65%
28.03.2021 217 424 6 071 2 388 4 748 5 165 18 372 2,79% 1,10% 2,18% 2,38% 8,45%
04.04.2021 216 851 6 273 2 404 4 800 4 404 17 881 2,89% 1,11% 2,21% 2,03% 8,25%
11.04.2021 252 185 6 421 2 518 4 888 4 008 17 835 2,55% 1,00% 1,94% 1,59% 7,07%
18.04.2021 252 470 6 600 2 549 4 868 3 176 17 193 2,61% 1,01% 1,93% 1,26% 6,81%
25.04.2021 252 615 6 729 2 577 5 114 2 243 16 663 2,66% 1,02% 2,02% 0,89% 6,60%
02.05.2021 252 539 6 979 2 484 5 494 1 144 16 101 2,76% 0,98% 2,18% 0,45% 6,38%
09.05.2021 252 412 7 092 2 451 5 975 0 15 518 2,81% 0,97% 2,37% 0,00% 6,15%
 Sliding window users journey processing
Categorical values Numeric values correlations check
1. Map first known before to future periods
2. Map first known after to previous periods
W -7 W-6 W -5 W-4 W-3 W-2 W-1
Initial data D1 D1
Iteration 1 D1 D1 D2 D2 D2
Iteration 2 D1 D1 D1 D1 D2 D2 D2

Periodical values (lifetime, days from last_pp_started

1. Map first known before to future periods +7
2. Map first known after to previous periods -7
W -7 W-6 W -5 W-4 W-3 W-2 W-1
Initial data D1 D1
Iteration 1 D1 D1+7 D2 D2+7 D2+14  Some numeric values has high correlations (e.g. tariff
Iteration 2 D1-14 D1-7 D1 D1 D2 D2 D2 bundles with ARPU, recharge with ARPU, total traffic
with 4G traffic, etc.)
Periodical values (last usage, last significant usage)  Still excluding of highly correlated data leads to model
1. Fulfill 1st
week by number of days from full dataset start accuracy degradation, all values kept
2. Map first known before to future periods +7 Aggregation calculation
W -7 W-6 W -5 W-4 W-3 W-2 W-1  Adding of statistical aggregation for usage features
Initial data D1 D1 during window period leads to model accuracy
degradation (sum, number of 0 periods, etc.)
Iteration 1 7*X D1 D2
 Aggregations was not included to final model
Iteration 2 7*X 7*X+7 D1 D1+7 D2 D2+7 D2+14
 Script meta parameters customization and ML model training
Model results 12 before 4 after Model results 8 before 6 after Model results 12 before 6 after

Threshold
Model results 12 before 10 after Model results 20 before 6 after 0,82 0,85 0,87 0,90 0,92 0,92 0,93
Recall 50,2 45,1 39,7 30,6 24,7 22,0 19,4
Precision (strong verificaton) 24,6 27,5 30,0 34,7 37,0 38,6 40,7
Precision (light verification) 37,8 41,3 43,8 48,6 50,7 52,5 54,0

True positives churn segmentation

Near before, % 29,9 30,3 32,5 35,1 36,8 37,6 38,5
Near future, % 34,0 34,9 33,8 32,9 31,9 31,0 32,0
Future, % 36,1 34,8 33,7 32,0 31,3 31,4 29,5

 Models are trained with

 300 thsd. Subscribers sample
 Statistics is shown for 1st 3 weeks of dataset
to check accuracy with longer future period
 Model value verification in retention process proposal

01.11.2020
08.11.2020
15.11.2020
22.11.2020
29.11.2020
06.12.2020
13.12.2020
20.12.2020
27.12.2020
03.01.2021
10.01.2021
17.01.2021
24.01.2021
31.01.2021
07.02.2021
14.02.2021
21.02.2021
28.02.2021
07.03.2021
14.03.2021
21.03.2021
28.03.2021
04.04.2021
11.04.2021
18.04.2021
25.04.2021
02.05.2021
09.05.2021
16.05.2021
23.05.2021
30.05.2021
06.06.2021
13.06.2021
20.06.2021
27.06.2021
04.07.2021
11.07.2021
18.07.2021
25.07.2021
01.08.2021
08.08.2021
15.08.2021
22.08.2021
29.08.2021
05.09.2021
12.09.2021
19.09.2021
26.09.2021
03.10.2021
10.10.2021
17.10.2021
24.10.2021
31.10.2021
07.11.2021
14.11.2021
21.11.2021
28.11.2021
05.12.2021
12.12.2021
19.12.2021
26.12.2021
02.01.2022
09.01.2022
16.01.2022
23.01.2022
30.01.2022
06.02.2022
13.02.2022
20.02.2022
27.02.2022
06.03.2022
13.03.2022
20.03.2022
27.03.2022
03.04.2022
10.04.2022
17.04.2022
24.04.2022
01.05.2022
08.05.2022
15.05.2022
22.05.2022
29.05.2022
05.06.2022
12.06.2022
19.06.2022
26.06.2022
Phase 1. Model
accuracy verification P R

for next periods

Phase 2. Model
R R
value in retention P C
S L
process

Option 1. Model accuracy verification for next periods Option 2. Model value in retention process
P Huawei makes prediction of churners P Huawei makes prediction of churners
R Analysis of model accuracy results and report creation C V makes retention marketing campaign with AB testing
RS Analysis of retention efficiency results within 6 weeks
RL Analysis of retention efficiency results within 3 months

Data requirements
Data to make prediction
Verification of churners in 6 weeks period Customer agreed with proposal and shared additional CRM data in
Verification of churners in 3 months period 2 iterations to verify results in live retention marketing campaign
38.37GB / 28 weeks data 7.3 GB / 4 weeks data
Pattern fill shows periods of additional required data
Shared at 28 Jan Shared at 18 Feb
Retention marketing campaign pilot (phase 2)

 Customer shared additional CRM data in 2 iterations to verify results in live retention
marketing campaign

58.3 GB / 28 weeks data @ 28 Jan 7.3 GB / 4 weeks data @ 18 Feb

 1st week of March V made retention marketing campaign to compare efficiency of in-house
and Huawei ML models results in M and S regions

11K 11K
1K 1K • For each model operator selected samples with 11 thsd.
Subscribers that model predicts as potential churner

B • Each sample was splitted for A/B testing 10/1

10K 10K A • Operator made outgoing calls campaign with CSI* survey
and proposal of retention offer to detractors

* CSI – Customer Satisfaction Index

Operator in- Huawei
house model model
Retention marketing campaign results evaluation by V (phase 2)
 Due to higher accuracy and better focus on high active (high value) Subscribers Huawei AI
model overcomes operator X in-house AI model efficiency by 28% in detractors identification
Marketing campaign Share of identified Number of identified Number of accepted Root cause analysis
Call Success Rate** detractors detractors retention offers of detractors
Other Bad internet quality
+19% +10% High tariff Coverage issues
+28% +22% Bad voice quality

27% 23%
X 25%
19%

54,0% 64,1% 19,8% 21,7% 1.004 699 27%

782 571 20%
14% 18%
13% 13%
Operator Huawei Operator Huawei Operator Huawei Operator Huawei Operator Huawei
in-house model in-house model in-house model in-house model sample sample
model model model model
**
• For model creation Huawei verified with operator retention of what kind of Subscribers is more valuable for them. V told that we need to focus on
Subscribers with higher activity
• To meet this requirement of Customer we developed special model for high activity Subscribers, that is proved by higher CSR in pilot results
• V highly evaluated our flexibility and real focus on their business needs

 Next steps: V is very interested in joint churn model development, especially for adding service
quality parameters to improve accuracy. V plans to discuss internally with CBM director our
SmartCare pilot proposal
Churn prediction methodology development plan for
Smartcare promotion
Joint churn prediction analysis with MKT&CEM team, Churn root cause deep dive

2021 2022 plan

Create churn prediction model based  Duplicate churn analysis model to:
Huawei AI based 9D potential churner on data provided by V CBM: • Other regions
analysis model with accuracy 85% can be
• Subscribers with lifetime >12 months • Additional Subscriber segments
improved with service quality features
• Sample for M and S regions (new sales, lifetime <12 months)
D1:
Value
D5: D3:  Improve churn prediction model
Recharge Life Cycle
D2: accuracy with service quality data
Usage * Model predicts churn in next periods
D6: D4: from Huawei platforms
Terminal

D8:
D7:
Location
Service

D9:
+ based on data that we have for
Subscriber in previous periods
Experience Network
Period (weeks)

Subs K
Basic Info
-N … -2 (weeks)
Period -1

Subs 1
Period (weeks)

Subs 1
Churn
9D data D1 x.x x.x
-N …x.x -2x.x -1 probability in  Make segmentation and correlate
-N … -2 -1
next M
D2 x.x
D1 x.x
x.x x.x
x.x x.x
x.x x.x
weeks
churn by possible reasons
D1 x.x x.x x.x x.x
Information that was used in D3 x.x
D2 x.x
x.x x.x
x.x x.X
x.x x.x
current model creation D2 x.x x.x x.x x.x Sub1
… x.x
D3 x.x
x.x x.x
x.x x.x
x.x x.X
D3 x.x x.x x.x x.X …
DN …x.x x.x
x.x x.x
x.x x.x
x.x x.x
… x.x x.x x.x x.x Subk
DN x.x x.x x.x x.x
DN x.x x.x x.x x.x
Thank you