Predicting QoE Factors with Machine Learning

Vladislav Vasilev, Jérémie Leguay, Stefano Paris, Lorenzo Maggi, Merouane

Debbah

To cite this version:

Vladislav Vasilev, Jérémie Leguay, Stefano Paris, Lorenzo Maggi, Merouane Debbah. Predicting QoE
Factors with Machine Learning. IEEE International Conference on Communications (ICC 2018), May
2018, Kansas, United States. �10.1109/icc.2018.8422609�. �hal-01947195�

HAL Id: hal-01947195

https://centralesupelec.hal.science/hal-01947195
Submitted on 6 Dec 2018

HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est

archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents
entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non,
lished or not. The documents may come from émanant des établissements d’enseignement et de
teaching and research institutions in France or recherche français ou étrangers, des laboratoires
abroad, or from public or private research centers. publics ou privés.
 Predicting QoE Factors with Machine Learning
Vladislav Vasilev, Jérémie Leguay, Stefano Paris, Lorenzo Maggi, Mérouane Debbah
Mathematical and Algorithmic Sciences Lab, Paris Research Center - Huawei Technologies Co. Ltd.
Email: {name.surname}@huawei.com

Abstract—Classic network control techniques have as sole resulting in different file sizes. The availability of multiple
objective the fulfillment of Quality-of-Service (QoS) metrics, representations for the same video segment enables DASH
being quantitative and network-centric. Nowadays, the research clients to scale up or down the video quality by simply
community envisions a paradigm shift that will put the em-
phasis on Quality of Experience (QoE) metrics, which relate selecting the best segment to be downloaded according to
directly to the user satisfaction. Yet, assessing QoE from QoS network status and video player’s buffer.
measurements is a challenging task that powerful Software The way final users perceive the quality of a streamed video
Defined Network controllers are now able to tackle via machine depends on several factors that cannot be all measured. This
learning techniques. In this paper we focus on a few crucial perceived quality is denoted as Quality of Experience (QoE).
QoE factors and we first propose a Bayesian Network model
to predict re-buffering ratio. Then, we derive our own novel According to [7], the user experience highly depends on three
Neural Network search method to prove that the BN correctly crucial factors: (i) the visual quality and its variation, (ii) the
captures the discovered stalling data patterns. Finally, we show frequency and duration of re-buffering events (i.e., stalls or
that hidden variable models based and context information boost interruptions), and (iii) the startup delay. While the visual
performance for all QoE related measures. quality and its variation can be measured using PSNR-based
Index Terms—Software Defined Networking, Quality of Experi-
ence, Bayesian Network, Neural Network Search Method, Graph
metrics when traffic is not encrypted, re-buffering events and
Clustering, Hidden Variable Model start-up delay cannot be directly measured, but only predicted
from classic QoS metrics [4]. This allows to infer QoE factors
I. I NTRODUCTION by still relying on legacy, QoS monitoring systems.
Yet, the mapping between QoS and QoE metrics is highly
According to a recent report [1], video traffic will steadily complex, as they often lay in high dimensional spaces and are
grow in the next years, representing 82% of the whole subject to noise. As a consequence, a closed form modeling
Internet traffic by 2021. Therefore, handling video traffic and its experimental validation are not practical. We therefore
so as to maximize the quality perceived by final users is resort to machine learning techniques to derive the complex
becoming critical both for content and network operators. relationships between QoS and QoE metrics.
To this end, Content Delivery Networks (CDNs) operators On a data set produced with a high-fidelity and fully
have adopted coordinated control planes [2] between routing controllable simulation environment, we show that while lo-
and their streaming systems following the recent trend of cal linear relationships hold for the video quality variation
Software Defined Networks (SDN) [3], which has deeply and network measurements, re-buffering events lay in high
transformed the way network architectures are designed and dimension clusters of QoS metrics. For re-buffering events,
controlled. Nonetheless, Internet Service Providers (ISPs) can we present a Bayesian Network (BN) classifier based on two
also contribute to improve the perceived quality of video traffic Logistic Regressions (LR) which better balances the class
by optimizing network resources according to the user needs. accuracy compared to the state of the art method based
However, ISPs can only exploit coarse-grained information on on random forests [8]. Furthermore, we demonstrate that
video flows due to the end-to-end encryption that many Over- due to the stochastic nature of re-buffering events, clusters
The-Top (OTT) operators like Facebook, Google, and Amazon partially overlap, hence increasing the inaccuracy of standard
employ [4]. ISPs are therefore calling for new methods for predictors. A pattern exploration model that we specifically
handling network resources in order to maximize the perceived design using a novel Neural Network (NN) search method
quality in video services, which directly reflects the opinion confirms our intuition that other predictors incur the same or
customers have on the network infrastructure [5]. worse inaccuracy of BN-based methods. Finally, we turn our
HTTP Adaptive Streaming (HAS), which has been stan- attention to hidden variables, namely metrics that cannot be
dardized into MPEG-Dynamic Adaptive HTTP Streaming directly measured but can be still inferred from QoS metrics.
(DASH) [6], represents nowadays the pillar technology for We show that the use of predicted hidden variables as features
video streaming over the Internet. Indeed, HAS connections can indeed improve accuracy for re-buffering events. Finally,
can easily pass through intermediate services like NATs, we show that if we have access to information about network
gateways and proxies without the need of complex network congestion (e.g., number of competing sessions, QoS measures
configurations. Videos are split into temporal segments whose on bottlenecks) and basic characteristics on video streams
duration lasts from a couple up to hundreds of seconds. Each (e.g., type of device, content provider) all predictions of QoE
segment (also knows as chunk) is encoded at different qualities factors can be further improved.
 The paper is structured as follow. Sec. II discusses relevant
related work. Sec. III describes the problem of predicting
QoE from QoS measurements and the data set we produced. • Average video bitrate of the downloaded segments.
Sec. IV illustrates our method to classify and predict re- • Average video bitrate variation: the standard deviation
buffering events, while Sec. V presents results for the video of the video bitrate. It quantifies quality changes over the
quality and its variation. Finally, Sec. VI concludes our paper. different downloaded segments.
• Re-buffering ratio: freezing (or stalling) time over the
II. R ELATED W ORK duration of the video streaming session.
Quality of Experience (QoE) has recently gained momen- Our aim in this paper is to infer the three aforementioned QoE
tum as a way to assess the user opinion of the network quality factor from the observable QoS metrics described in Tab. I
while watching videos. An additive log-logistic model that using machine learning techniques.
maps video quality, freezing (i.e., stall of the video session),
and image artifacts due to compression and re-buffering events B. Dataset Description
into a QoE score has been firstly proposed in [9] and succes-
To build and evaluate QoS to QoE mapping functions, we
sively adopted by ITU in the Recommendation P.1202.2 as a
have used a high-fidelity and fully controllable simulation
reference model for quantifying QoE [10]. The investigation
environment at both network and streaming levels. The simula-
performed in [11] on how a user perceives the video quality
tion platform is based on the Adaptive Multimedia Streaming
and the main factors that influence this perception resulted in
Simulator Framework (AMust) [17] in ns-3 which implements
the definition of eight mathematical models of QoE. Studies
an HTTP client and server for LibDASH, one of the reference
like the one presented in [12] provide quantitative methods to
software of ISO/IEC MPEG-DASH standard.
measure the distortion of the received bit-stream due to video
As streaming content, we have chosen 3 representative
quality and freezing. While different in the way they compute
open movies1 commonly used for testing video codecs and
a score for measuring QoE, all these works agree on three main
streaming protocols: Big Buck Bunny (BBB), a cartoon with
impairments that affect the QoE, namely re-buffering events,
a mix of low and high motion scenes, Swiss Account (TSA),
the video quality and its variation. Furthermore, due to the
a sport documentary with regular motion scenes and Red Bull
psychological effect known as memory effect, the repetition
Play Street (RBPS), a sport show with high motion scenes.
of the same impairment during the video session such as
We have considered a star network with a bottleneck link
the experience of multiple video stalls due to re-buffering
as shown in Fig. 1, on top of which we have simulated a
strongly affects the quality perceived by the final user [13]. For
large number scenarios varying the number of nodes (from 1
this reason, both client-side and network-side mechanisms [7],
to 100), the bottleneck capacity (from 500 kbps to 10Mbps
[14], [15] have been recently proposed to prevent or at least
per stream), the bottleneck delay (from 10ms to 100ms), the
minimize re-buffering events and video quality variations.
bottleneck packet loss (from 0% to 3%), screen resolutions
Existing client-side DASH adaptation policies base their
and DASH policies (RB, BB and hybrid). After a month
decisions on several network performance and the internal
of simulations, we have obtained statistics for more than
client state. Rate-Based (RB) policies base their decisions
69,000 video sessions with 50 associated variables from 4
on the measured download throughput, whereas Buffer-Based
categories: Context information on network congestion and
(BB) [14] approaches use the level of the buffer containing the
stream characteristics, QoS metrics, Target QoE factors and
downloaded segments to decide the quality of the next chunk.
Hidden QoE variables (see Tab. I for a complete list). The
A number of hybrid approaches also exist, where the explicit
dataset is meant to become public.
formulation of the optimization problem [7] enables the use
of control theoretic methods.
Machine learning has been recently used to predict QoE
from network measurements [4], [16]. Dimopoulos et al. [4]
shows how the rebuffering ratio, and the average video quality
and its variation, can be predicted using random forests. We
consider this work as a starting point for our research and
present two further contributions: 1) a Bayesian Network
model to predict rebuffering events with a better balance in
class accuracies and 2) the evidence that additional context
information on network congestion and basic characteristics Fig. 1: Simulation environment with AMust in ns-3.
of video streams improve predictions for all QoE factors.
III. F ROM Q O S TO Q O E FACTORS Arguably, out of the 3 target variables we want to predict
RebufferingRatio is the most difficult, especially in its raw
A. Problem Statement
continuous form. To simplify our task we take a similar
We consider three main QoE factors which are commonly
used to measure user-perceived video quality [5]: 1 http://concert.itec.aau.at/SVCDataset/
 Index Name Type Description
Class Training Accuracy Validation Accuracy
1 RequestID Context Streaming session identifier
2 NbClients Context Maximum number of streams competing on the bottleneck NoStall 0.96178 0.95525
3 BottleneckBW Context Capacity of the bottleneck
4 BottleneckDelay Context Network delay on the bottleneck MildStall 0.7585 0.73587
5 BottleneckLoss Context Packet loss on the bottleneck
6 DASHPolicy Context DASH policy (e.g, or name of content provider)
SevereStall 0.43874 0.34211
7 ClientResolution Context Client screen resolution or device type (e.g., smartphone)
8 RequestDuration QoS metric Duration of the stream TABLE II: RF class accuracies. Training to validation ratio is 4:1.
[9, 13] TCPOut/InPacket QoS metric Number of TCP packets (In and Out)
[10, 14] TCPOut/InDelay QoS metric Average delay experienced by TCP packets (In and Out)
[11, 15] TCPOut/InJitter QoS metric Average jitter experienced by TCP packets (In and Out)
[12, 16] TCPOut/InPloss QoS metric Packet loss rate experienced by TCP packets (In and Out)
17 TCPInputRetrans QoS metric Packet retransmissions experienced by TCP
18
[19:27]
StdNetworkRate
[0,5,10,25,50,75,90,95,100]
QoS metric
QoS metric
Standard deviation of the network rate
xth quantile for the network rate
IV. S TALLING PREDICTION
NetworkRate (measured in intervals of 2s)
28 StdInterATimesReq QoS metric Std. dev. of inter-arrival times of segment requests
[29:37] [0,5,10,25,50,75,90,95,100] QoS metric xth quantile for the
In this section we use a Bayesian Network (BN) [18] model
38
InterATimesReq
StartUpDelay Hidden
inter-arrival times of segment requests
Initial time at the client to start playing the video
to accurately predict the StallLabel variable from the QoS
39
40
AvgVideoDownloadRate
StdVideoDownloadRate
Hidden
Hidden
Average downloading rate for video segments
Std. dev. of downloading rate for video segments
metrics listed in Tab. I. We then show that StallLabel is formed
41
42
AvgVideoBufferLevel
StdVideoBufferLevel
Hidden
Hidden
Average video buffer length.
Std. dev. of video buffer length
by a mixture of 2 distributions and that if there is a model that
43 StallEvents Hidden Number of stall events predicts accurately the true distribution of a data point we can
44 RebufferingRatio Target Portion of time spent in stall events
45 StallLabel Target Discretization of RebufferingRatio variable get around 97% performance with the proposed BN model.
46 TotalStallingTime Hidden Total duration of stall events
47 AvgTimeStallingEvents Hidden Average duration of stall events Finally, we conjecture through a custom novel neural network
48 AvgQualityIndex Hidden Avg. normalized index of downloaded representations
49 AvgVideoBitRate Target Average video bitrate consumed by the player search method that there is no such model, hence achieving
50 AvgVideoQualityVariation Target Average variation of the video bitrate
51 AvgDownloadBitRate Hidden Average download rate of video segments higher performance with our dataset is unlikely.
TABLE I: Context information, QoS metrics, hidden variables. Target As a benchmark model we use Random Forest (RF) as done
QoE factors, which we want to predict from all other variables, are in [8]. A RF is a bagging of Decision Tree (DT) models, see
highlighted in bold. [19]. At each leaf node, DT greedily selects and splits an input
variable into non-overlapping regions, so that the resulting new
leafs gain predictive power. The bagging procedure essentially
approach as in [8]. The RebufferingRatio values are aggregated tries to minimize the effect of local optimality that stems from
into 3 discrete values in a new variable StallLabel. Firstly, the greedy split procedure. Table II shows the performance of a
RebufferingRatio equals to 0 means that no stalling has RF classifier on the StallLabel variable pruned with minimum
occurred, hence we set StallLabel=NoStall. If it is between leaf size of 50 to prevent over-fitting and training to validation
(0, 0.1) then StallLabel=MildStall. Finally, if RebufferingRatio size ration of 4:1 (the same ratio is used in all the paper).
is above 0.1 then StallLabel is given the value SevereStall. The results show that the RF is making an accurate
Fig. 2 shows the histogram of the 3 target variables. Sim- prediction on the NoStall class of StallLabel while it has
ilarly to the target variables, all other variables’ distribution worse predictions on the MildStall. The performance on the
follow an exponential pattern. For this reason we initially SevereStall class is practically unacceptable. There are 2 most
apply on the input data a logarithmic transformation. commonly occurring problems with RF, i.e., 1) the RF greedy
split procedure result in low quality local optimum; 2) the
In the next sections, we use machine learning techniques to
RF’s rectangular decision regions have boundaries parallel to
derive accurate QoS-QoE mappings given the available data.
the basis of the dimensions, which could fail to capture some
dependencies among features.

10 4
10 For these reasons, we turn our attention to Bayesian Net-
works based on Logistic Regression (LR) predictors. LR is
Count

51155
5
17180 a binary classification model that maximizes the likelihood
794 L(θ) of a target vector of binary values Y given the input
0
NoStall MildStall SevereStall data X for a given prior distribution P (θ) (assumed to be
Re-buffering Ratio

10 4
uniform in our experiments) of the parameter set θ, see Eq.
4 (1). The a-posteriory probability model is assumed to be
P (Yi |X, θ) = (σ[θ, Xi ])Yi (1 − σ[θ, Xi ])1−Yi where σ is the
Count

2
sigmoid function σ[θ, x] = 1+e1−θ·x . In Eq. (2) we report the
0 gradient of the log-likelihood.
0 2 4 6
AvgVideoBitRate [bps]
10 6

10 4
( n
)
5
Y
max L(θ) = P (Yi |X, θ) P (θ) (1)
θ
Count

i=1
( )
∂ log(L(θ)) X ∂ log(P (θ))
0 = Xi (σ[θXi ] − Yi ) + (2)
0 1 2 3 ∂θ i
∂θ
AvgVideoQualityVariation [bps]
10 6
Using the LR probabilistic model we next define the BN
Fig. 2: Histograms for our 3 target variables. we used in Fig. 3 to predict the StallLabel variable.
 Data X θ1 6⊥ θ2 |M ildStall
Input: Data X, initial parameters θ1 = 0, θ2 = 0;
θ1 6⊥ θ2 |SevereStall
Output: Prediction for Y =StallLabel
LR1 : P (Yi |X, θ1 ) θ1
X Set X := log(X); Optimize the parameter θ1 of the 1st
LR by maximizing the log-likelihood via gradient descent
If P (Yi |X, θ1 ) ≥ α ∈ [0, 1] method, i.e.:
Y=NoStall LR2 : P {Yi |X, θ2 , P (Yi |X, θ1 ) ≥ α} θ2 ∂ ln(L(θ1t−1 ))
θ1t (i) = θ1t−1 (i) −
P {Yi |X, θ2 , P (Yi |X, θ1 ) ≥ α} ≥ β ∂θ1t−1 (i)
Y=MildStall Y=SevereStall Optimize the decision boundary α of the 1st LR such that:

P (Y, X, θ1 , θ2 ) = α = argminα |T P R(LR1 ) − T N R(LR1 )|

= [P (Yi |X, θ1 )P (θ1 )] . [P {Yi |X, θ2 , P (Yi |X, θ1 ) > α} P (θ2 )] P (X) if (P (Yi |X, θ1 ) ≥ α) then
Fig. 3: Bayesian Network using LR models. Yi = NoStall
else
Class Training Accuracy Validation Accuracy Yi = SomeStall
NoStall 0.8681 0.8684 end if
MildStall 0.7929 0.8048 Select X := X(Yi = SomeStall) and Y := Y (Yi =
SevereStall 0.9338 0.9368 SomeStall)
TABLE III: BN class accuracy. Training vs. validation size ratio 4:1. Optimize θ2 via gradient descent:
∂ ln(L(θ2t−1 ))
θ2t (i) = θ2t−1 (i) −
∂θ2t−1 (i)
Optimize the decision boundary β of the 2nd LR:
β = argminβ |T P R(LR2 ) − T N R(LR2 )|
In the prediction step each data point is first classified by the
1st LR. If the 1st LR predicts the data to be in the SomeStall if (P {Yi |X, θ2 , P (Yi |X, θ1 ) > α} ≥ β) then
class then the data is further classed by the 2nd LR. Observe Yi = SevereStall
that the 2nd LR is dependent on the prediction of the 1st LR. else
In the training phase, we first optimize the θ1 parameters Yi = MildStall
by standard gradient descent, hence obtaining the 1st LR and end if
a prediction for the NoStall class. Then, we select from Y
only the data that was predicted to have SomeStall by the 1st Fig. 4: Training algorithm for our Bayesian Network model.
LR and we optimize for θ2 to train the 2nd LR and gain the
prediction for mild and severe stall classes. The pseudo-code 1st LR 97% CA
of the training phase of our BN is shown in Fig. 4. We report
QoS,StallLabel
in Table III the performance of the proposed BN model, that Use the misclassified only
outperforms the SoA Random Forest approach (cfr. Tab. II). Data
We point out that we optimized the decision threshold of 2nd LR for wrong 1st LR
each LR, usually set to 0.5, so that the True Positive Rate Experiment (top)
(TPR) and the True Negative Rate (TNR) are equal. We do
Only use 1st LR
this in order to achieve similar final Class Accuracies (CA). Data Switching model 97% CA
After deriving our BN model that makes a better trade-off Only use 2nd LR
between the class accuracies than the RF we would like to Switching model (bottom)
know if the data classes are linearly separable. Fig. 5: Exploring how to improved the 1st LR of the BN: the
experiment and the model that should yield the improvement.
A. Is our data linearly separable?
In this section we seek to find if there is a more accurate, a Neural Network (NN). Being able to extract the relevant
but also more complex mapping between the input data and features and make accurate predictions, NN became the state
the target variable StallLabel. We discovered through the of the art technique to map and search highly complex
experiment shown in Fig. 5 (top) that if we select only the relations [20].
misclassified data by the 1st LR in Fig. 3 to train another (2nd )
LR then we can reach a 97% CA. Thus, if we find a switching B. Neural Network Search Using an Index Invariant Graph
model as in Fig. 5 (bottom) that assigns the correct LR to each Generally, when applying a NN it is best to derive its setting
data point, then we will improve the overall CA of our BN. based on the underlying characteristics of the problem because
In the next section we train the switching model by deriving this reduces the parameter set that needs to be hand-picked.
 Input variables X IIT/Neuron Output variables Input X Input X Input H

F eature1 V1 Training ONLY

X1
V1
Use 1st LR H = FH (X)
F eature2 V2
X2 Y = F (X)
Predict H|X
F eature3 V3
X3 Y = F (X, Hpredicted )

F eature4 V1 Output Y Output Y

X4
V2
Use 2nd LR Fig. 7: Our approach using hidden variables. (Left) the standard pre-
F eature5 V2
X5 diction task. (Right) prediction procedure that includes intermediate
hidden variables prediction.
F eature6 V3
X6

D. Results from the IIT and the StallLabel variable

Fig. 6: The Index Invariant Tree, see [21], [22]. Tab. IV presents the results obtained using the BN predic-
tor for five in different cases: when only QoS metrics are
available, when additional context information is predicted or
known to the network controller, and when hidden variables
In order to derive the appropriate NN search method let us are obtained from prediction or perfectly known to the network
consider the example in Fig. 6. There we map between the controller. The table shows results only with four different
input and output variables using a feature set and a single validation splits, since the analysis of the standard deviation
neuron. Notice that we do not use the classical NN model (STD) suggests that we cannot gain in accuracy.
with sigmoid activation function. Instead, we resort to the tree We observe that the usage of the hidden variable procedure
structure of a neuron in order to reformulate it using an Index always increases accuracy and that if good model can be
Invariant Tree (IIT). For the details of ITT, we refer to [21], available for them the expected gain is high. Indeed, accuracies
[22]. Intuitively, the IIT states that each feature should map in the case where actual hidden variables are used (perfect
only to a single output and all pairs of features should not prediction) plots a significant improvement compared to the
overlap. This enables to gain good prediction power while one with QoS metric only . Also, the addition of context
encoding large part of the data characteristics. variables, which can be retrieved in practice from the network
Following this analysis the training of the NN reduces to a controller, improves prediction accuracy.
stable set search on an appropriate graph whose vertices are In term of the search for improvements, the IIT method
features and each edge specifies if two features overlap. Here, did not find any areas where there are consistent misclassi-
a feature is any subset of the whole domain of the data, i.e., fication between the two LRs in Fig. 5 for all classes (NoS-
F eature ⊂ R#variables . As a result, the training of the NN tall,SomeStall,MildStall,SevereStall). This is true regardless of
becomes a combination of simulated annealing [18] and graph whether the context parameters and the hidden variables are
clustering algorithms as in [22]. It is worth noting at this point added as predicted or actual values. This finding suggests that
that other clustering methods could be used based on available the proposed BN based on LR is likely the best performing
time and complexity constraints. model on the considered dataset.
Case Tr. NS Tr. MS Tr. SS Val. NS Val. MS Val. SS
C. Using Hidden and Simulation Variables QoS metric Mean 0.8685 0.7946 0.9376 0.8665 0.7982 0.9267
only STD 0.0004 0.0014 0.0031 0.0018 0.0045 0.0122
It is worth examining another possibility to improve the Added predicted Mean 0.8691 0.7963 0.9409 0.8691 0.7996 0.9316
context STD 0.0004 0.0008 0.0031 0.0021 0.0036 0.0017
prediction performance, namely the use of additional vari- Added actual Mean 0.8689 0.7975 0.9396 0.8676 0.8023 0.9304
ables H. The additional variables are given in Tab. I as context STD 0.0001 0.0010 0.0027 0.0023 0.0033 0.0018
Added predicted Mean 0.8758 0.8012 0.9434 0.8735 0.7983 0.9366
context and hidden variables. It is possible to train a separate hidden STD 0.0018 0.0033 0.0019 0.0034 0.0101 0.0079
model H = FH (X) for each of the simulation and hidden Added actual Mean 0.9000 0.8399 0.9543 0.8990 0.8466 0.9530
hidden STD 0.0012 0.0023 0.0026 0.0020 0.0091 0.0113
variables and then use their prediction in the final model
Y = F (X, Hpredicted ). In this way the final model still TABLE IV: BN StallLabel training (Tr.) and validation (Val.) class
accuracies for NoStall (NS), MildStall (MS) and SevereStall (SS)
uses only the input variables, but it makes an intermediate using a 4:1 training to validation size ratio and using hidden variables
prediction on the additional variables which are then used [38:42,48:51] (from Tab. I).
along with the input to get the final target prediction. This
process is shown in Fig. 7.
Bagged [19] Regression Tree and Bagged Random Forest V. V IDEO B IT-R ATE AND Q UALITY VARIATION
are used for the additional variable prediction, depending on We finally focus on the prediction of the two remaining
whether they are continuous or discrete. QoE factors, namely average video bitrate (AvgVideoBitRate)
 Case Training Validation
QoS metric only 43.33 (0.23) 51.73 (0.69)
and its variation (AvgVideoQualityVariation). We remark that Context Pred. 37.51 (0.25) 46.41 (0.59)
this turned out to be a much easier task, as those factors Context Actual 28.88 (0.17) 41.24 (0.56)
Hidden + Context Pred. 36.78 (0.26) 46.09 (0.61)
are linearly dependent on quantitative QoS metrics which are
Hidden + Context Actual 26.23 (0.13) 37.78 (0.38)
already observable. For this reason, a classic Regression Tree
(RT) model was found to be suitable for both QoE factors. TABLE VI: Same as Tab. V but for the average video quality variation
AvgVideoBitRate can be easily predicted with high accuracy (in Kbps). Same training to validation ratio and same variables.
through the use of a bagged RT with minimum leaf size of
10. Tab. V shows the RT results. Similarly to AvgVideoBi- R EFERENCES
tRate, AvgVideoQualityVariation is efficiently predicted with a [1] Cisco, “Cisco Visual Networking Index: Forecast and Methodology,
bagged RT with minimum leaf size of 10 as shown in Tab. VI. 20162021,” June 2017.
[2] X. Liu, F. Dobrian, H. Milner, J. Jiang, V. Sekar, I. Stoica, and H. Zhang,
We observe that the prediction of both the video bitrate and “A case for a coordinated internet video control plane,” in Proc. ACM
the quality variation is very accurate. Indeed, as results show, SIGCOMM, 2012.
the mean error for both variables is only of a few kbps while [3] B. A. A. Nunes, M. Mendonca, X.-N. Nguyen, K. Obraczka, and
T. Turletti, “A survey of software-defined networking: Past, present, and
the video bitrate can be up to 8 Mbps in the scenarios we future of programmable networks,” IEEE Communications Surveys &
considered. As for the case of StallLabel prediction discussed Tutorials, vol. 16, no. 3, pp. 1617–1634, 2014.
in the previous Section, these results also demonstrate that [4] G. Dimopoulos, I. Leontiadis, P. Barlet-Ros, and K. Papagiannaki,
“Measuring Video QoE from Encrypted Traffic,” in Proc. ACM IMC,
the accuracy can further improves when context information 2016.
is know to the network controller and good prediction models [5] H. Nam, K.-H. Kim, and H. Schulzrinne, “Qoe matters more than qos:
are available for hidden variables. Our simple predictors for Why people stop watching cat videos,” in Proc. IEEE INFOCOM, 2016.
[6] T. Stockhammer, “Dynamic Adaptive Streaming over HTTP: Standards
hidden variables help reducing a little bit the error but higher and Design Principles,” in Proc. ACM MMSys, 2011.
gains can be obtained with better models. [7] X. Yin, A. Jindal, V. Sekar, and B. Sinopoli, “A Control-Theoretic
Approach for Dynamic Adaptive Video Streaming over HTTP,” in ACM
SIGCOMM, 2015, pp. 325–338.
Case Training Validation [8] G. Dimopoulos, I. Leontiadis, P. Barlet-Ros, and K. Papagiannaki,
QoS metric only 58.13 (0.59) 68.78 (0.17) “Measuring video qoe from encrypted traffic,” in Proc. ACM IMC, 2016.
Context Pred. 55.99 (0.52) 66.19 (0.10) [9] F. Zhang, W. Lin, Z. Chen, and K. N. Ngan, “Additive log-logistic model
for networked video quality assessment,” IEEE Trans. on Image Proc.,
Context Actual 47.18 (0.75) 63.47 (0.23) vol. 22, no. 4, pp. 1536–1547, April 2013.
Hidden + Context Pred. 53.01 (0.75) 64.34 (0.16) [10] ITU-T, “Parametric non-intrusive bitstream assessment of video media
Hidden + Context. Actual 34.87 (0.60) 46.00 (0.12) streaming quality - Higher resolution application area ,” 2013.
[11] W. Song and D. W. Tjondronegoro, “Acceptability-based qoe models
TABLE V: Mean and standard deviation (in parenthesis) for the for mobile video,” IEEE Transactions on Multimedia, vol. 16, no. 3, pp.
prediction error of the average video quality (in Kbps). We used a 4:1 738–750, 2014.
training to validation size ratio and hidden variables [38:42,44,45,51] [12] Z. Chen, N. Liao, X. Gu, F. Wu, and G. Shi, “Hybrid distortion ranking
(from Tab. I). tuned bitstream-layer video quality assessment,” IEEE Trans. on Circuits
and Systems for Video Technology, vol. 26, no. 6, pp. 1029–1043, 2016.
[13] T. Hoßfeld, S. Biedermann, R. Schatz, A. Platzer, S. Egger, and
M. Fiedler, “The memory effect and its implications on web qoe
VI. C ONCLUSION modeling,” in Proc. IEEE ITC, 2011.
[14] T.-Y. Huang, R. Johari, N. McKeown, M. Trunnell, and M. Watson, “A
In this paper we utilize machine learning techniques to buffer-based approach to rate adaptation: Evidence from a large video
demonstrate how QoS metrics can be exploited to accurately streaming service,” ACM SIGCOMM CCR, vol. 44, no. 4, 2015.
[15] P. T. A. Quang, K. Piamrat, K. D. Singh, and C. Viho, “Video streaming
estimate and predict key QoE factors. We mostly focus on the over ad hoc networks: A qoe-based optimal routing solution,” IEEE
StallLabel QoE factor as it is the hardest to predict. We im- Tran. on Veh. Tech., vol. 66, no. 2, pp. 1533–1546, Feb 2017.
prove on the RF performance [8] by building a more balanced [16] Y.-T. Lin, E. M. R. Oliveira, S. B. Jemaa, and S. E. Elayoubi, “Machine
learning for predicting qoe of video streaming in mobile networks,” in
model in term of class accuracies. We discovered simple but Proc. IEEE ICC, 2017.
important patterns in the StallLabel variable and defined the [17] C. Kreuzberger, D. Posch, and H. Hellwagner, “AMuSt Framework -
BN model to envelop them. Using a custom NN based search Adaptive Multimedia Streaming Simulation Framework for ns-3 and
ndnSIM,” 2016.
method we showed that any other StallLabel model is unlikely [18] D. Koller and N. Friedman, Probabilistic graphical models: principles
to outperform our proposed Bayesian approach. and techniques. MIT press, 2009.
For all the crucial QoE factors we considered, we show [19] C. M. Bishop, Pattern recognition and machine learning. Springer,
2006.
that making intermediate predictions for hidden variables can [20] H. Lee, R. Grosse, R. Ranganath, and A. Y. Ng, “Convolutional
boost the predictive performance of our approach, compared deep belief networks for scalable unsupervised learning of hierarchical
to the case where only observable variables are used. We also representations,” in Proc. ACM ICML, 2009.
[21] V. Vasilev, “Chromatic polynomial heuristics for connectivity prediction
show that context information on network congestion and basic in wireless sensor networks,” in ICEST 2016, Ohrid, Macedonia, 28-30
characteristics on video streams further improves predictions. June 2016.
In the future, we plan to design new features, specific to [22] V. G. Vasilev, Algorithms and Heuristics for Data Mining in Sensor
Networks. LAP LAMBERT Academic Publishing, December 2016.
video profiling (such as in [23]), that can be measured by [23] D. Tsilimantos, T. Karagkioules, A. Nogales-Gómez, and S. Valentin,
QoS monitoring systems and improve QoE predictions. “Traffic profiling for mobile video streaming,” in Proc. IEEE ICC, 2017.