Customer Churn Prediction in the Software by Subscription

models IT business using machine learning methods

Anna Kolomiiets a , Olga Mezentseva,b and Kateryna Kolesnikova c
a b, c
Taras Shevchenko National University of Kyiv, Bogdan Gavrilishin Street, 24, Kyiv, 04116, Ukraine

Abstract
The article presents the results of research related to the problems of development of IT
products based on the Software by Subscription model in the IT sphere. Operating in market
conditions, such enterprises are forced to develop modern IT products to support small and
medium-sized businesses based on the interaction of many potential external customers (users
of the IT system), who later, under favorable conditions, become customers of these
enterprises. At the same time, the very nature of the Software by Subscription company is
largely determined by its average client, and is a topic played out in many empirical rules of
Software by Subscription metrics. To improve the efficiency of customer interaction with SaaS
companies, the authors proposed a hypothesis about the possibility of using various forecasting
methods in machine learning. A comparative characteristic of the use of different models and
algorithms for predicting the outflow of customers for an IT company is carried out. At the
same time, the development of innovative IT products should be carried out with the fullest
satisfaction of the interests and needs of all major stakeholders. The article offers a
mathematical description of the model and method of modeling these interactions.. To conduct
chain training, the Python functionality is used, with the processing of user activity data sets.
The analysis is carried out and conclusions are made about the effectiveness of the proposed
approach.

Keywords 1
Software by Subscription, random forest algorithm, IT company, Customer Churn Prediction

1. Introduction

Customer Churn Prediction is one of the classic problems in deep data analysis (data mining). IT
companies have long analyzed customer usage patterns to predict customer churn. Many other
industries, such as banking, regularly analyze customer behavior to predict their satisfaction and level
of renewal [4].
The software by Subscription model allows software vendors to collect data about customer usage
that is not available to traditional software vendors [1,2].
Although the market for Software by Subscription and cloud computing in general is growing
rapidly, however, as far as we know, from a scientific and practical point of view, the problem of
outflow in B2B Software by subscription projects has not been sufficiently studied and the application
of existing methods for analyzing customer outflow in the field of B2B Software by Subscription has
not been fully done to resolve this issue. Despite significant international experience, domestic IT
companies have their own peculiarities of formation and development, which is due, on the one hand,
to globalization processes, and on the other-to the peculiarities of the National Economy [7]. The low
speed of testing IT products, insufficient development of the technological base, and a high level of

ITTAP’2021: 1nd International Workshop on Information Technologies: Theoretical and Applied Problems, November 16–18, 2021,
Ternopil, Ukraine
EMAIL: [email protected] (A. 1); [email protected] (A. 2); [email protected] (A. 3)
ORCID: 0000-0003-4252-5975 (A. 1); 0000-0002-8430-4022 (A. 2); 0000-0002-9160-5982 (A. 3)
©️ 2021 Copyright for this paper by its authors.
Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR Workshop Proceedings (CEUR-WS.org)
competition in the industry cause the main problems of domestic IT startups, namely, the slow pace of
their creation and entry into the market with ready-made products [3].
The software by Subscription business model depends on strong and long-term customer
relationships. If customers are not satisfied or are not reminded of the price that the IT company's service
regularly provides, they cancel their subscription. The company, in turn, should focus on retaining
customers for as long as possible, developing these relationships. The range of predictive capabilities
of machine learning should help it do this [11].
Content streaming services are by far the most well-known examples of the software by Subscription
business model. Companies like Netflix and Spotify have built incredibly successful businesses using
their subscription growth potential. However, not everyone was able to repeat it, even Adobe in 2018
had some financial losses, not correctly calculating its subscription offers for software such as
Photoshop [8].
While some companies do this by monetizing content and demonstrating their value through the
entertainment experience they offer, others compete in narrower niches of industrial and desktop
software, which requires companies to put much more effort into promoting, advertising, and retaining
customers than just good content. Accurate forecasting of customer outflows and building a balanced
marketing and financial policy of an IT company based on this will help an IT company solve this
complex and vital task [15].

2. Related Works

Euler [16] developed a decision tree to identify the types of telecommunications customers that are
most likely to be solved. Euler used the capabilities of the KDD MiningMart data preprocessing system
to obtain predictable characteristics that were not present in the original data. Kusment and Van den-
Paul used support vector machines to improve churn prediction performance for the newspaper
subscription service [17]. The results of this work show that the interaction between customers and the
supplier is important for outflow analysis. Kusment and Van-den-Paul continued to study customer-
provider interaction by adding emotions from customer email to their model [18]. [4] determined the
predictive characteristics of customer churn and found that decision trees outperformed neural networks
and regression in terms of overall accuracy.
To expand the analysis, numerous studies have explored various machine learning algorithms and
their potential for outflow modeling. Since predicting whether a client will be lost is not a binary
classification problem, several models have been tested, such as logistic regression [19], decision trees
[20], random forest, supporting vector machines, and neural networks [21].
Considering the development processes of IT project products, it is necessary to consider the
development directions of companies themselves with Software by Subscription [22] and B2B [23]
business models. Although our review points to numerous studies on software by Subscription, most of
the work tends to focus on multiple programs. In particular, previous subscriber subscription work is
usually based on subscriber data in mobile phone areas [24], credit cards [25], and internet service
delivery [26].

3. Proposed methodology

The purpose of this article is to analyze modern models and methods for predicting customer churn
using machine learning, which can be applied to the activities of B2B Software by Subscription IT
companies. This will help companies and their teams to create, market and scale IT products more
efficiently, which can not only increase the efficiency of the enterprise due to the competence of
personnel, but also generally strengthen Ukraine's position in the global innovation market.
In this article, we used the analysis of customer outflow in telecommunications as a basis for
studying the software by Subscription industry. Although there are differences between these two
industries, they actually have a lot in common. The paper presents a literature review, a detailed
description of the context of the problem, a comparison of software by Subscription and
telecommunications providers, an example of an experiment, and outlines opportunities for future
research in the dissertation using neural networks to build effective algorithms for solving further
problems.
Software by Subscription models usually mean that the customer installs the product, as when selling
software on the spot, but pays the supplier in stages, rather than in full upfront. The provider does not
bear the costs of hosting or integration with other applications. The transition to the subscription model
is often the first stage for traditional software development companies moving to the Software by
Subscription model [9].
The validity period of the licenses is similar, since the client pays for the installed software for a
certain period, usually one, two or three years. Subscription models usually include maintenance and
updates, as in SaaS, but urgent licenses do not. However, there is no rigid generally accepted definition
of Subscriptions compared to fixed-term licenses, as it is clear that IT companies use these terms
interchangeably [13].
Both subscription and fixed-term (and SaaS) payment contracts may or may not include the right to
cancel and refund fees already paid. These conditions affect revenue recognition and customer
retention.
In local software, the supplier does not bear the costs of hosting the software, but must send the
software to the customer, so the distribution costs are different. Local companies are often more likely
to use an affiliate channel to distribute, install, and support their sales. While SaaS vendors are also
expanding the use of channel sales, the channel still typically accounts for a smaller percentage of total
revenue [6].
According to experts in this field, SaaS and similar models do not simplify licensing metrics. If we
look at "pure SaaS solutions" like Salesforce, there are a few things to consider. An article published
by the BSA (navigating the cloud :why managing software assets is more important than ever)
highlights the risks and pitfalls of SaaS in terms of license management:
• Software components: some SaaS solutions include plugins or other user-side software that
requires proper licensing and management
• Unauthorized use: violation of geographical restrictions, shared logins, incorrect logins (for
example, administrator accounts, not regular accounts), providing logins to third parties, generating
value from the SaaS system, and sharing with others who do not have access
• Warehouse (paid but not used software): the use of licenses should be constantly monitored to
identify unused licenses
• User profile definition: given the "multi-user rental" architecture, only one version of the
product is deployed for the entire organization (depending on the company size, product
characteristics, and user requirements) and cannot be changed until the end of the contract.
In another case, there may be a case of a SaaS architecture “for one tenant”. In this case, different
configurations are possible, according to the requirements of each user. In the case of software by
Subscription or “quasi-SaaS”, such as Adobe Creative Cloud, the license rate does not change compared
to the old perpetual model (for example, by installation). This means that if these licenses are not
properly managed and controlled, the risk of compliance may still be high [7].
Contrary to popular belief about software by Subscription licensing, software asset management
practices are still very important in IT departments and should be reviewed to properly handle new
license models [28].
Software as a Service (SaaS) defines the financial/contractual, as well as the delivery aspects of the"
product "and the" product " of the supplier/customer relationship. SaaS revenue is based on customer
subscriptions to access a "service" hosted in the cloud. The SaaS business model for a supplier affects
the entire business and every department, from strategy to finance, sales and marketing, research and
development, customer management and business systems needed to manage and track a variety of
activities (Fig.1).
Fixed-term licenses, sometimes called subscriptions, refer to the financial terms of the relationship
between the supplier and the customer. According to the fixed-term license, the customer pays the
supplier in increments over time, but installs the product, as when selling software on the spot. The
provider does not bear the hosting costs. The transition to a fixed-term license or subscription model is
often the first stage for traditional software companies moving to the SaaS business.
 It is observed that companies use the terms "fixed-term license" and” subscription license
"interchangeably, but" Fixed-term license " is usually applied to a specific term, such as a license for
two or three years, which then expires.
The local software is delivered to the customer and installed on the customer's website. Payment is
usually made in advance with annual maintenance and maintenance fees. The supplier is responsible
for problems with the product and may be responsible for interacting with other customer systems. The
client is responsible for paying for and managing everything necessary for the use of the software.

Figure 1: Revenue to Market Cap Multiple [4]

One of the most powerful ways to grow software by Subscription is through "net negative MRR
churn". With a negative outflow, the company's it revenue generated from existing customers month
after month outstrips the revenue lost as a result of cancellations and downgrades. The lower the user
outflow, the easier it is to achieve a net negative MRR outflow.A negative outflow is observed when
some of your users start paying more: they switch to a more expensive tariff, buy additional modules,
extensions, and so on.
When analyzing B2B companies, attention was drawn to this impressive difference in revenue. A
business with a negative outflow is almost 3 times larger than the same one with a standard outflow of
2.5% (note that this indicator is considered very good). And a business that users do not leave at all has
only 60% more revenue than the same one with an outflow of 2.5%. Obviously, negative outflow is the
most powerful growth accelerator [7].
When calculating the outflow rate, this was done in order to gain a deeper understanding of your
customers and why they are leaving your product. Improve this, we get improve customer retention
rates (from English Customer Retention) in order to strengthen your business in the long run.
Customer Retention is a company's ability to maintain a long-term relationship with a customer. A
high rate means that your customers are happy to return for a new purchase and recommend you to their
friends. Customer Retention rate formula:
𝐶𝑢𝑠𝑡𝑜𝑚𝑒𝑟 𝑎𝑡 𝑒𝑛𝑑 𝑜𝑓 𝑝𝑒𝑟𝑖𝑜𝑑 − 𝐶𝑢𝑠𝑡𝑜𝑚𝑒𝑟 𝑎𝑐𝑞𝑢𝑖𝑟𝑒𝑑 𝑑𝑢𝑟𝑖𝑛𝑔 𝑝𝑒𝑟𝑖𝑜𝑑 (1)
𝐶𝑟𝑟 = 100,
𝐶𝑢𝑠𝑡𝑜𝑚𝑒𝑟 𝑎𝑡 𝑡ℎ𝑒 𝑠𝑡𝑎𝑟𝑡 𝑜𝑓 𝑡ℎ𝑒 𝑝𝑒𝑟𝑖𝑜𝑑

where Crr – Customer retention rate.

Customer churn can result from low levels of customer satisfaction, aggressive competitive
strategies, rejection of new products, etc.churn models are designed to detect early signals of customer
churn and recognize those who are highly likely not to use services.
For machine learning methods, it is necessary to collect training data, the main task of which is to
describe the client in as much detail as possible. The following indicators can be used for this purpose:
• Socio-demographic data
• Transactional data (the number and amount of transactions for the period, grouped by
different criteria, etc.)
• Product and segmentation data (changes in the number of contracts, belonging to the bank's
internal segmentation group, etc.).
 The target variable showing the probability of outflow obviously directly depends on changes in the
client's transaction activity, as well as on the client's category and subscription type. After identifying
a group of customers with an increased risk of outflow, methods of retaining customers are applied by
providing profitable promotions, offers, and so on.
From the analysis of machine learning methods, the optimal one for predicting customer outflows
will be the one that most accurately corresponds to the relationships between the data in the sample that
characterizes the type of business [29].
The experiments were conducted using methods implemented in the Python programming language.
The effectiveness of using methods is determined by the accuracy of classification, i.e. the proportion
of objects that actually belong to the class relative to all objects that are assigned to the class based on
the results [12].
To solve this problem, we used a deep neural network. It is defined as a neural network with several
layers. In deep neural networks, each layer of neurons is trained on the properties / outputs of the
previous layer. Thus, we were able to create a hierarchy of objects to increase abstraction and test
complex hypotheses.
Deep learning based on libraries TensorFlow, PyTorch, MXNet, and Chainer are usually used to
predict customer churn in B2C. In our B2B study, we used scikit-learn and TensorFlow. Code snippets
for initial data processing and Tensor creation are shown in Fig. 8.
The tocategoric () function was used to convert categorical properties to one. This made it possible
to convert String values to numbers and used tf.onehot () for creating vectors.

Figure. 2: Code snippets for initial data processing and tensor creation

We created a two-dimensional tensor from our functions (categorical and numeric) and performed
normalization. Then we broke the data down into training and testing datasets. The missing values were
encoded as 0. The resulting deep neural network has two hidden layers with 32 and 64 neurons,
respectively. It is Fully Connected Linear Neural Network. Each layer has a ReLU activation function
[5]:
𝑓(𝑥) = 𝑥 + = 𝑚𝑎𝑥(0, 𝑥), (2)
where x - neuron input.
This function is defined as the positive part of the argument. This allows for better data preparation
for deep networks compared to the logistic Sigmoid and hyperbolic tangent function.
The model was trained using the Adam optimizer [31, 32], and the binary crossentropy function was
used to measure the error [10]. The model learns for the first ten epochs, after which it reaches a
plateau(fig. 3).
Figure 3: Visualization of model training up to 100 epochs

The model has an accuracy of 79.2% according to test data (based on the results of checking 10%
of mixed data).
As a result (see Table 1), we found that linear regression, logistic regression, and Bayesian classifier
have a low but stable evaluation quality. The decision tree and the reference vector method were forced
to retrain, having a 100% result in the training sample, but a very low result in the test sample [30].
The optimal result was shown by using a random forest. This model has a stable and high evaluation
quality. But it is quite expensive to implement in the IT infrastructure.

Table 1
Results of comparative analysis of machine learning methods for predicting customer churn
Method Training sample Test sample
Linear regression 0.76 0.75
Logistic regression 0.73 0.71
Neural networks 0.85 0.82
Decision tree 1.00 0.76
Random forest 0.88 0.87
Reference vector method 1.00 0.63
Naive Bayesian classifier 0.71 0.69

The principle of random lea prediction is that each tree in a random forest returns a class forecast.
Trees have a very weak correlation between themselves. As a result, the class with the most votes
becomes the forecast of the forest.
The forest is resistant to noise signs, and therefore the validation curve for the number of initial signs
involved will reach the asymptote. That is, if the initial features are, say, 150, you can evaluate the
importance of the features, build validation curves for models trained on 50, 60, .. 150 features and
catch the moment when adding new, less important features does not improve the quality so much. The
tree is built, as a rule, until the selection is exhausted (until only representatives of one class remain in
the leaves), but in modern implementations there are parameters that limit the height of the tree, the
number of objects in the leaves and the number of objects in the subsample at which splitting is
performed.
On the other hand, such an analysis applies to large companies that have a heterogeneous package
of services and large capital. From our point of view, among the methods considered, we should also
analyze the approach to comparing machine learning methods used in similar businesses operating on
the SaaS model, but in the field of B2B [14].
The dataset includes information about:
• Customers who disabled their subscription during the last quarter of Churn
• Services that each client subscribes to
• Customer account information-how long they have been a customer, contract, payment
method, monthly payments and total expenses.
 Decision trees are very sensitive to data. For this reason, we made small changes in the data set on
which the model was trained. This led to significantly different tree structures. We used this advantage
by allowing each individual tree to arbitrarily select data with replacement, which led to different trees.
The concept of a training sample is a key one in pattern recognition. A training sample is an
independent sample 𝑫 = {𝒙𝒊 , 𝒚𝒊 }𝒍𝒊=𝟏 rom some distribution P(x, y) = P(x)P(y|x). Here xi , i = 1, 2, . . . ,
l — are feature vectors (called precedents), whose coordinates represent the values of n features
(independent variables) measured on some object [10].
The corresponding yi represent the values of the dependent variable. If yi can take only a finite
number of values, i.e. yi ∈ {ω1, ω2, . . . , ωc}, c > 2, then we have a classification problem. In this case,
yi is called a class label and determines whether the corresponding k object belongs to one of the c
lasses, and the attribute itself is called class; if yi is measured on a numerical scale, then we have a
regression problem; in this case, the attribute is called a response; A decision tree [31] is a tree with
each vertex t are associated: 1. the entire image space of χt ⊂ χ ; is associated with the root vertex χ. A
subsample Dt ⊂ D of the training sample D, such that Dt = {(x, y) ∈ D : x ∈ Xt}; thus, the entire sample
D is associated with the root vertex. Some function (rule) ft : χ → {0, 1, . . . , kt − 1} (here kt > 2 — is
the number of descendants of vertex t), which determines the partition of the set χ into k disjoint subsets.
No function is associated with terminal vertices.
Let's denote ti(t) , i = 0, 1, . . . , kt − 1 he vertex that is the i–th descendants of vertex t. Subset χt and
rool ft defines subsets χti(t) as follows [5]:
𝜒𝑡 𝑖(𝑡)=𝜒𝑡 ∩ {𝑥 ∈ 𝜒: 𝑓𝑡 (𝑥) = 𝑖 }, (3)
The purpose of constructing a decision tree is to classify vectors x from a distribution P(x).
In general, the reduction of pollution is defined as [32]
∆𝑖(𝑡) (4)
∆𝑖𝐵 (𝑡) = ,
− ∑𝐵𝑘=1 𝑃𝑘 𝑙𝑜𝑔2 𝑃𝑘
where B — the number of descendants of the vertex t, Pk — percentage of subsample examples D(t),
corresponding to the vertex tk and ∑𝑩 𝒌=𝟏 𝑷𝒌 = 𝟏. And the splitting that maximizes the value is chosen
∆iB(t).
In the original dataset, we have 7,044 examples of B2B SaaS subscriptions and 21 variables
(characteristics of companies that purchase services using the SaaS method).We have a very unbalanced
data set. About 20% of customers are businesses that have been around for less than 3 years, and they
are much more likely to have an outflow of customers compared to a long-term business. Answering
the question of how long customers stay in the company, at first glance it seems that the longer the
company remains a customer, the more likely it is that it will continue to be loyal in the future. But this
is only visually and at first glance.
When building the model, we took into account the features of the model. The more trees, the better
the quality, but the setup time and operation of a random forest also increase proportionally. To speed
up the construction, n_jobs=-1 was used (build on the maximum possible number of processors). To
construct reproducible experiments, a preset of a pseudorandom number generator was used:
random_state [29].

4. Results

As the number of n_estimators trees increases, the quality in the training sample increases, and the
quality in the test reaches the asymptote (Fig.4 ). According to the schedule, it was decided to build
about 40 trees. The model was built at full depth, since the data is mostly not noisy.

Figure 4: Quality when varying the value of the number of trees n_estimators

Having determined a sufficient number of trees in the forest, the number of features was selected for
the choice of splitting. The quality graph on the test is unimodal from the value of this parameter, it
strictly increases during training. When max_features increases, the time for building the forest
increases, and the trees become "more monotonous".

Figure 5: Quality when varying the value of the number of features for the choice of splitting
max_features

Obviously, buyers with low monthly payments (<$230). Most likely, they will remain customers.
However, the higher the total amount charged by the company, the higher the probability of retaining
that customer.
The expected value of the expected outflow from each forecast in relation to total income is the
same. Therefore, it is important for an IT company to understand what scale of customer inflow implies
the projected volume of outflow.
The reason why the random forest model works so well is that a large number of relatively
uncorrelated trees working together will outperform any of their individual components.
The key factor is the weak correlation between the trees. Due to this, just as assumed financial
investments with low correlations (for example, investments in stocks and bonds) are combined to form
a portfolio larger than the sum of its parts,uncorrelated models can predict more accurately than any of
the individual forecasts. The reason for this effect is that trees protect each other from their individual
mistakes until they constantly make mistakes in the same direction.
5. Conclusion
The paper analyzes modern models and methods for predicting customer churn using machine
learning, which can be applied to B2B Software by Subscription of IT companies. Among them, the
random forest method is highlighted and a separate analysis of the deep neural network method is
performed. The latter is due to the fact that this method is widely used in the field of B2C
telecommunications. the industry experience turned out to be acceptable for a different audience of
consumers. A deep neural network with 32 hidden layers has shown fairly stable accuracy and allows
you to identify the main indicators of early customer outflow signals. This will help it founders and
their teams implement customer retention policies in advance through various marketing activities and
additional services.

6. References
[1] S. Blank, The Startup Owner's Manual: The Step-By-Step Guide for Building a Great Company
Hardcover, 2012.
[2] Startup statistics, The Numbers You Need to Know. URL:
https://smallbiztrends.com/2019/03/startup-statistics-small-business.html.
[3] E. Ries, The Lean Startup: How Today's Entrepreneurs Use Continuous Innovation to Create
Radically Successful Businesses, 2011.
[4] Small Business Statistics: 19 Essential Numbers to Know, 2020. URL:
https://www.fundera.com/blog/small-business-statistics.
[5] S.P. Chistyakov. Random forests: review. Res. Karel. Res. Centre RAS 1, 2013, 117–136.
URL: http://resources.krc.karelia.ru/transactions/doc/trudy2013/trudy_2013_1_117-136.pdf.
[6] E. Loukis, M. Janssen, I. Mintchev, Determinants of software-as-a-service benefits and impact on
firm performance, Decision Support Systems, volume 117 (2019) 38-47.
[7] Kunal Swani, Brian P. Brown, Susan M. Mudambi, The untapped potential of B2B advertising: A
literature review and future agenda, Industrial Marketing Management, 2019.
[8] V. Morozov, O. Tsesliv, A. Kolomiiets, S. Kolomiiets, Construction of a mathematical model
for analyzing the effectiveness of IT startups, in: CEUR 2021 Workshop Proceedings, pp. 283–
292. URL: http://ceur-ws.org/Vol-2851/paper26.pdf
[9] D. Edelman, M. Singer, Competing on customer journeys: you have to create new value at every
step. Harvard Business Review, 93(11) (2017) 88-98.
[10] E. C. Conforto, D. C. Amaral : Evaluating an Agile Method for Planning and Controlling
Innovative Projects. Project Management Journal, 2010. URL:
https://journals.sagepub.com/doi/abs/10.1002/pmj.20089.
[11] L. Chernova, S. Titov, S. Chernov, K. Kolesnikova, L. Chernova, and V. Gogunskii, Development
of a formal algorithm for the formulation of a dual linear optimization problem. Eastern-European
Journal of Enterprise Technologies 4 (100) (2019) 28–36. doi: 10.15587/1729-4061.2019.175105.
[12] V. Morozov, O. Mezentseva, M. Proskurin, Trainable neural networks modelling for a forecasting
of start-up product development, in: Proceedings of the 2020 IEEE 3rd International Conference
on Data Stream Mining and Processing, DSMP (2020), pp. 55–60. doi:
10.1109/DSMP47368.2020.9204264.
[13] A. Timinsky, O.Voitenko, I.Achkasov, Competence-based knowledge management in project
oriented organisations in bi-adaptive context, in: Proceedings of the IEEE 14th International
Scientific and Technical Conference on Computer Sciences and Information Technologies (CSIT-
2019). Lviv, pp. 17-20. doi: 10.1109/STC-CSIT.2019.8929806.
[14] K. Kolesnikova, O. Mezentseva and O. Savielieva, Modeling of Decision Making Strategies In
Management of Steelmaking Processes, in: IEEE International Conference on Advanced Trends
in Information Theory (ATIT 2019). pp. 455-460. doi: 10.1109/ATIT49449.2019.9030524.
[15] A. Kolomiiets, V. Morozov, Investigation of optimization models in decisions making on
integration of innovative projects. Advances in Intelligent Systems and Computing, 1246 (2021)
51–64. doi:10.1007/978-3-030-54215-3_4.
[16] P. Campbell. The saas business model: stages, pros and cons, and tools to get ahead. December
04, 2019. URL: https://www.profitwell.com/blog/saas-business-model.
[17] J. Böringer, B. Grehan, D. Kiewell, S. Lehmitz, and P. Moser: Four pathways to digital growth
that work for B2B companies. October 2019. URL: https://www.mckinsey.com/business-
functions/marketing-and-sales/our-insights/four-pathways-to-digital-growth-that-work-for-b2b-
companies#.
[18] J. Neumann, O. Morgenstern, Theory of Games and Economic Behavior: 60th Anniversary
Commemorative Edition, Princeton Classic Editions, 2007.
[19] C. Akenbor, L. Nwaeke: Application of game theory in organisational conflict resolution: the case
of negotiation. URL: https://www.researchgate.net/publication/275028170.
[20] T. A. B. Galvao, F. M. M. Neto, M. F. Bonates, and M. T. Campos, A Serious Game for Supporting
Training in Risk Management through Project-Based Learning. in Virtual and Networked
Organizations, Emergent Technologies, and Tools. volume 248 (2012) 52–61.
[21] J. Linowski: Agile A/B Testing: Using Stop Rules To Minimize Losses & Time Wasted, 2016.
URL: https://goodui.org/blog/agile-ab-testing-using-stop-rules-to-minimize-losses-time-wasted/.
[22] J. C. Harsanyi, R. Selten : A General Theory of Equilibrium Selection in Games. URL:
https://www.researchgate.net/publication/23573567_A_General_Theory_of_Equilibrium_in_Ga
mes.
[23] C. Hahn, D. Röher and R. Zarnekow. A value proposition oriented typology of electronic
marketplaces for B2B SaaS applications. AMCIS (2015). URL:
https://aisel.aisnet.org/amcis2015/e-Biz/GeneralPresentations/22/.
[24] H. Junling, M. P.Wellman: Multiagent reinforcement learning: Theoretical framework and an
algorithm. URL: http://ai.eecs.umich.edu/people/wellman/index.html.
[25] G. Tesauro. Extending Q-learning to general adaptive multi agent reinforcement learning:
Theoretical framework and an algorithm. URL:
http://ai.eecs.umich.edu/people/wellman/index.html.
[26] The Essential John Nash. Ed. by Harold W. Kuhn and Sylvia Nasar. Princeton Univ. Press, 2007.
[27] Best Project Management Software from Basecamp, 2020. URL:
https://www.g2.com/products/basecamp/reviews/basecamp-review-3538015.
[28] Designing experiences with information from Atlassian, 2020. URL:
https://medium.com/designing-atlassian/designing-experiences-with-information-e97d191d380e.
[29] URL: https://js.tensorflow.org/api/latest/#metrics.binaryCrossentropy.
[30] URL: https://js.tensorflow.org/api/latest/#train.adam.
[31] Z. Chen, X. Sun and L. Shen, An Effective Tumor Classification With Deep Forest and Self-
Training, in IEEE Access, volume 9, pp. 100944-100950, 2021, doi:
10.1109/ACCESS.2021.3096241.
[32] F. Liu, T. Dong, T. Hou and Y. Liu, A Hybrid Short-Term Load Forecasting Model Based on
Improved Fuzzy C-Means Clustering, Random Forest and Deep Neural Networks, in IEEE Access,
vol. 9, pp. 59754-59765, 2021, doi: 10.1109/ACCESS.2021.3063123.