See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/326928417

Towards Massive Machine Type Communications in Ultra-Dense Cellular IoT

Networks: Current Issues and Machine Learning-Assisted Solutions

Preprint · August 2018

CITATIONS READS

0 2,299

2 authors:

Shree Krishna Sharma Xianbin Wang

University of Luxembourg The University of Western Ontario
110 PUBLICATIONS   1,980 CITATIONS    432 PUBLICATIONS   6,733 CITATIONS   

SEE PROFILE SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Shared Access terrestrial-satellite backhaul Network enabled by Smart Antennas (SANSA) View project

Physical layer security View project

All content following this page was uploaded by Xianbin Wang on 30 September 2018.

The user has requested enhancement of the downloaded file.

 IEEE COMMUNICATIONS SURVEYS & TUTORIALS (DRAFT) 1

Towards Massive Machine Type Communications in

Ultra-Dense Cellular IoT Networks: Current Issues
and Machine Learning-Assisted Solutions
Shree Krishna Sharma, Senior Member, IEEE, and Xianbin Wang, Fellow, IEEE

Abstract— The ever-increasing number of resource-constrained the ever-increasing number of smart devices (forecasted by
arXiv:1808.02924v1 [eess.SP] 8 Aug 2018

Machine-Type Communication (MTC) devices is leading to the IHS Markit to be around 125 billion by 2030), intelligent
critical challenge of fulfilling diverse communication require- industry processes, people and societies together to enhance
ments in dynamic and ultra-dense wireless environments. Among
different application scenarios that the upcoming 5G and beyond the overall quality of our daily life [1]. Towards supporting
cellular networks are expected to support, such as enhanced Mo- connected IoT devices, there are several recent developments
bile Broadband (eMBB), massive Machine Type Communications in the area of licensed cellular technologies such as Long
(mMTC) and Ultra-Reliable and Low Latency Communications Term Evolution (LTE) for Machine-Type Communications
(URLLC), the mMTC brings the unique technical challenge of (LTE-M) and Narrow-Band IoT (NB-IoT), and unlicensed
supporting a huge number of MTC devices in cellular networks,
which is the main focus of this paper. The related challenges technologies such as WiFi, ZigBee and LoRa [2]. Out of these,
include Quality of Service (QoS) provisioning, handling highly cellular technologies are considered to be promising due to
dynamic and sporadic MTC traffic, huge signalling overhead and their several advantages including Quality of Service (QoS)
Radio Access Network (RAN) congestion. In this regard, this provisioning, wide coverage area and tight coordination, and
paper aims to identify and analyze the involved technical issues, therefore, cellular IoT is of the main focus in this paper.
to review recent advances, to highlight potential solutions and to
propose new research directions. First, starting with an overview
of mMTC features and QoS provisioning issues, we present A. Recent Developments in Cellular IoT
the key enablers for mMTC in cellular networks. Along with In recent years, cellular IoT has gained significant impor-
the highlights on the inefficiency of the legacy Random Access tance from academia, industries, regulators and standardization
(RA) procedure in the mMTC scenario, we then present the key
bodies to enable the incorporation of IoT devices in the
features and channel access mechanisms in the emerging cellular
IoT standards, namely, LTE-M and Narrowband IoT (NB-IoT). existing cellular infrastructures. The ITU-R has categorized
Subsequently, we present a framework for the performance the emerging diversified telecommunication services in the
analysis of transmission scheduling with the QoS support along upcoming 5G and beyond cellular networks into the following
with the issues involved in short data packet transmission. Next, three classes [3]: (i) enhanced Mobile Broadband (eMBB),(ii)
we provide a detailed overview of the existing and emerging
massive Machine Type Communications (mMTC), and (iii)
solutions towards addressing RAN congestion problem, and then
identify potential advantages, challenges and use cases for the Ultra-Reliable and Low Latency Communications (URLLC).
applications of emerging Machine Learning (ML) techniques in Out of the above-mentioned categories, the eMBB com-
ultra-dense cellular networks. Out of several ML techniques, we prises of high data rate services of 5G systems while the
focus on the application of low-complexity Q-learning approach mMTC deals with the scalable connectivity to a massive
in the mMTC scenario along with the recent advances towards number of devices in the order of 106 devices per square
enhancing its learning performance and convergence. Finally,
we discuss some open research challenges and promising future kilometers with diverse QoS requirements [4]. On the other
research directions. hand, URLLC aims to provide robust connectivity with very
low latency. The main challenge in the mMTC case is to
Index Terms— Cellular IoT, mMTC, 5G and beyond wireless,
RAN congestion, Machine learning, Q-learning, LTE-M, NB-IoT. support a huge number of devices with the limited radio
resources whereas the key challenge for the URLLC scenario
is to provide extremely high reliability in the order of 99.999%
I. I NTRODUCTION
within a very short duration in the order of 1 ms [5]. Among
The convergence of emerging wireless communication tech- these usage scenarios, the mMTC has to deal with various
nologies, ubiquitous wireless infrastructure and vertical In- non-conventional challenges including QoS provisioning, Ran-
ternet of Things (IoT) applications such as industrial au- dom Access Network (RAN) congestion, highly dynamic and
tomation, connected cars and smart-grid is leading to an sporadic traffic, and large signalling overhead. To this end,
integrated enabling platform for future smart and connected this paper focuses on the involved issues and the potential
societies. This platform envisions to synergistically integrate enablers of the mMTC scenario with a particular emphasis on
This work was supported in part by NSERC Discovery and CREATE the RAN congestion problem and emerging Machine Learning
programs under project number RGPIN-2018-06254 and 432280-2013. (Cor- (ML)-based solutions.
responding author: Xianbin Wang). In terms of ongoing standardization efforts, the main cellular
The authors are with the Department of Electrical and Computer En-
gineering, Western University, London, ON, N6A 3K7, Canada, Email: IoT standards introduced by the 3GPP are LTE-M and NB-
[email protected], [email protected]. IoT. Out of these, LTE-M is intended for mid-range IoT
IEEE COMMUNICATIONS SURVEYS & TUTORIALS (DRAFT) 2

applications which can support voice and video services, while erations since the number of configurable parameters of a
NB-IoT systems target to provide very large coverage and cellular network has been rapidly increasing from one cel-
support for ultra-low cost devices [6]. Since MTC devices lular generation to the next one [15, 16]. The widely-used
usually do not require high channel throughput, the existing link adaptation techniques in the existing wireless systems,
LTE-M and NB-IoT standards allocate a small bandwidth for which adapt different physical layer parameters including
IoT devices, i.e., LTE-M assigns 1.4 MHz bandwidth while the transmission power and modulation and coding scheme based
NB-IoT allocates a significantly lower bandwidth of 180 kHz on the reliability/link of a communication link, may not be
[7]. Despite these recent developments, there are still several efficient in ultra-dense cellular IoT networks. This adaptation
challenges to be addressed while supporting MTC devices in is based on the prediction of reliability of a wireless link in
cellular systems. the form of some metrics such as Packet Error Rate (PER)
and this prediction process becomes extremely complex due
B. Challenges in Cellular IoT to the increasing trend of using multiple antennas, wideband
Although centralized cellular systems provide several ad- signals and a number of advanced signal processing algorithms
vantages in terms of providing large coverage, tight time [17]. Furthermore, the prediction of PER with good accuracy
synchronization and handover operations for mobile users, becomes difficult in practice by using the conventional signal
they are sluggish in terms of handling low-end devices and processing tools. Moreover, due to a significantly large number
face several challenges in supporting a large number of MTC of environmental parameters such as channel state information,
devices with diverse QoS requirements. While incorporating signal power, noise variance, non-Gaussian noise effect and
MTC devices in the existing LTE/LTE-A based cellular sys- transceiver hardware impairments, it becomes complicated to
tems, cellular operators have to face a lot of challenges both provide the near-optimal/optimal tuning of the transmission
at the operational and planning levels. More specifically, there parameters to achieve the efficient link adaptation [18]. The
arise various issues related to the MTC device deployment, severity of this problem greatly increases in ultra-dense net-
mMTC traffic, energy efficiency of low-cost MTC devices works due to the involvement of massive number of devices
and the network protocol aspects such as signalling overhead and system parameters.
[8]. Furthermore, network congestion may occur in different Understanding the context of the surrounding wireless
segments of LTE/LTE-A based cellular network including environment significantly facilitates in developing context-
RAN, core network and signalling network [9]. Out of these, aware adaptive communication protocols and in taking op-
RAN congestion problem is crucial in ultra-dense cellular timized decisions. Nevertheless, handling self-configuration,
IoT networks due to the limited available radio resources at self-optimization and self-healing operations in the ultra-dense
the access-side and the massive number of sporadic access cellular networks becomes challenging since the networks
attempts from heterogeneous MTC devices. need to observe dynamic environmental variations, learn un-
Existing contention-based protocols are effective to sup- certainties, plan response actions and configure the associ-
port the conventional Human-Type Communications (HTC), ated network parameters effectively. To this end, emerging
however, their performance significantly degrades in mMTC ML-assisted techniques seem promising since they can play
scenarios due to infrequent and massive number of access significant roles in learning the system variations/parameter
requests [10]. Also, due to limited available preambles in the uncertainties, classifying the involved cases/issues, predicting
existing LTE-based systems, several MTC devices may need the future results/challenges and investigating potential solu-
to select the same preambles at the same time, resulting in tions/actions [16]. Moreover, the conventional link adaptation
a significantly high probability of collision in the access net- techniques are more localized to a particular network and
work. Furthermore, the number of transmission attempts from a geographical region, and do not usually consider their
the massive number of heterogeneous IoT devices could be impacts on the other systems. However, future ultra-dense
significantly large [11], and their activation periods and frame cellular networks will need to handle mutual impact among the
sizes could be very different [12]. This sporadic and dynamic involved entities to maximize the overall system performance.
nature of mMTC access attempts and data transmissions may To this end, by utilizing the emerging collaborative edge-cloud
result in the peak traffic in both the access and traffic channels processing platform [19], ML-assisted solutions can enable
well beyond the capacity of the IoT access network, thus the utilization of global network knowledge at the edge-side,
leading to the inevitable congestion in an IoT access network and also facilitate the coordination among different distributed
[13]. Moreover, although data packets transmitted by IoT systems. In this direction, the application of ML techniques
devices are relatively short, very high signalling overhead per to address various issues in dynamic wireless environments
data packet becomes another critical issue [8, 14]. To this end, has recently received an important attention [16, 20] and in
it is significantly important to investigate suitable transmission the context of MTC environments, some existing works have
scheduling and efficient signalling reduction techniques in already studied the applications of different ML techniques in
ultra-dense scenarios by utilizing emerging tools such as ML. learning various system parameters [10, 21–25].
However, the direct application of conventional ML tech-
C. Need of Machine Learning and Associated Challenges in niques to complex and dynamic wireless IoT environments is
IoT/mMTC Networks not straight-forward due to several underlying constraints such
Optimizing the operation of cellular networks in dynamic as low computational capability of MTC devices, distributed
wireless environments has been challenging over the gen- nature and heterogeneous QoS requirements of IoT devices,
IEEE COMMUNICATIONS SURVEYS & TUTORIALS (DRAFT) 3

TABLE I
C LASSIFICATION OF SURVEY / OVERVIEW WORKS IN THE AREAS OF I OT/ M MTC, ML AND UDN S .

Main domain Sub-topics References

Enabling technologies/protocols, challenges and applications [1, 2, 6, 11, 14, 28–32]
Random access schemes [22, 33–35]
IoT/mMTC Traffic characterization and issues [36]
Transmission scheduling [37, 38]
IoT big data analytics [19, 39, 40]
QoS provisioning [41]
Intelligence in 5G networks [16, 20]
Machine Learning (ML) Learning in IoT/sensor networks [26, 42, 43]
Reinforcement learning [44, 45]
Ultra-Dense Networks (UDNs) [46–48]

and the distinct features of mMTC traffic as compared to (eMTC), and MAC and higher layer enhancements brought by
the conventional HTC traffic [26]. Furthermore, due to the the extended Discontinuous Reception (eDRX). Furthermore,
limited computed power and low memory size of IoT devices, the survey article [2] provided a comprehensive survey of
implementing sophisticated learning techniques in IoT devices three main low power and long range M2M solutions, namely,
becomes challenging [27]. In this regard, this paper identifies Low Power Wide Area Network (LPWAN), IEEE 802.11ah-
the implementation issues of the ML techniques in ultra-dense based network and cellular M2M including LTE-M and NB-
IoT scenarios and provides an emphasis on computationally IoT. Besides, the overview article [29] presented the new
simpler Q-learning based solutions. requirements and challenges in large-scale MTC applications,
and discussed some enabling techniques including efficient
overhead signalling protocols, data aggregation and in-device
D. Review of Related Overview/Survey Articles
intelligent processing. Also, the survey article [31] provided a
In this subsection, we provide a brief overview of the comprehensive tutorial on the development of MTC design
existing survey works in the main domains related to this over different releases of LTE and recent user equipments
paper, namely, IoT/mMTC, ML and Ultra-Dense Networks belonging to the MTC and the NB-IoT categories, called
(UDNs). Also, we present the classification of the existing CAT-M and CAT-N, respectively. Moreover, another overview
references related to these domains into different sub-topics article [32] provided a review of various features of NB-IoT
which are listed in Table I. introduced in LTE Release 14 including the increased posi-
Several existing papers have provided the review of en- tioning accuracy, multi-casting, enhanced non-anchor carrier
abling technologies, protocols, challenges and applications operation and lower device power class, and the applicability
of IoT/mMTC in different contexts [1, 2, 6, 11, 14, 28–32]. of these features for NB-IoT systems.
Authors in [1] provided a comprehensive overview of ex- The design of effective Random Access (RA) schemes in
isting IoT protocols including application protocols, service the mMTC environment is an important challenge due to
discovery protocols, infrastructure protocols, and discussed massive access requests and sporadic device transmissions
some enabling technologies including cloud computing, edge from a huge number of resource-constrained MTC devices. In
computing and big data analytics for IoT systems. Further- this regard, authors in [22] provided an overview of different
more, the authors in [6] presented a detailed survey of MTC RA overload control mechanisms to avoid the RAN congestion
systems including its features, requirements and the required caused by the random channel access from the MTC devices.
architectural enhancements in LTE/LTE-A based networks. Furthermore, the article [33] presented a comprehensive survey
In the context of short packet transmissions in mMTC/IoT of various RA solutions attempting to enhance the RACH
environment, the contribution in [11] provided a review of the operation of LTE/LTE-A based cellular networks, and carried
recent advances in information theoretic principles governing out the performance evaluation of LTE RACH from the
the transmissions of short data packets and discussed the energy efficiency perspective. Moreover, the authors in [34]
applications of these principles to different scenarios including provided a review of emerging LPWAN technologies both in
a two-way channel, a downlink broadcast channel and an the unlicensed band (LoRa and SIGFOX) and in the licensed
uplink Random Access Channel (RACH). Besides, the article band (LTE-M and NB-IoT) while considering three common
[14] highlighted the requirements and design challenges for fundamental objectives of these access mechanisms, namely,
mMTC systems and discussed various physical and Medium high system capacity, wide coverage and long battery life. In
Access Control (MAC) layer solutions for energy-efficient addition, the article [35] provided an overview of the existing
and massive access. Another overview paper [28] discussed RA solutions towards supporting MTC devices in LTE/LTE-
the physical limitations of MTC devices while operating in A based networks and these solutions are compared in terms
cellular networks, and then analyzed the impact of these device of five key metrics, namely, access success rate, access delay,
limitations on the link performance and the link budget design. QoS guarantee, energy efficiency and the impact on the HTC.
In addition, the article [30] presented a review on var- The nature of MTC traffic is significantly different from the
ious features defined by the 3GPP to support Machine-to- HTC traffic as detailed later in Section II-D, however, existing
Machine (M2M) communications in LTE-based cellular sys- cellular networks are mainly optimized to support the HTC
tems and discussed recent advances in different layers includ- traffic. Therefore, it is important to understand and characterize
ing the physical layer improvements under the enhanced MTC MTC traffic to facilitate the incorporation of MTC devices in
IEEE COMMUNICATIONS SURVEYS & TUTORIALS (DRAFT) 4

cellular networks. To this end, the authors in [36] provided authors in [16] recently introduced the fundamental concepts
a discussion on the traffic issues of MTC and the associated of AI and its relation with 5G candidate technologies. Also,
congestion problems on the access channels, traffic channels the challenges and opportunities for the application of AI in
and core network, and presented a comprehensive review of managing network resources in intelligent 5G networks were
the existing solutions towards addressing these problems along discussed.
with their advantages and disadvantages. In the IoT/mMTC environment, learning techniques need
One of the promising solutions to handle massive access to consider the unique features such as heterogeneity, re-
requests in the mMTC scenarios is to employ suitable trans- source constraints and QoS requirements. In this direction,
mission scheduling techniques at the distributed MTC devices. the article [26] discussed the applicability of different types
In this context, the authors in [37] identified limitations for of learning techniques in the IoT scenarios by taking their
signalling and scheduling of M2M devices over the existing learning performance, computational complexity and required
LTE-based cellular infrastructures and discussed some of the input information into account. Furthermore, authors in [42]
existing proposals. Furthermore, the article [38] provided discussed various aspects of deep Reinforcement Learning
a detailed survey on the uplink scheduling techniques for (RL) and its application in building cognitive smart cities
M2M devices over LTE/LTE-A based cellular networks by while considering the use cases in the areas of water consump-
considering various aspects of M2M communications such tion, energy and agriculture. Moreover, the recent article [43]
as scalability, energy efficiency, QoS support and multi-hop provided an overview of existing context-aware computing
connectivity. studies along with the learning and big data related works
Another important aspect in ultra-dense IoT networks is in the direction of intelligent IoT systems. In the context of
how to handle the massive amount of data generated from the RL techniques, a comprehensive survey of multi-agent RL is
resource-constrained sensors and MTC devices. In this regard, provided in [44] by considering the aspects of stability of the
authors in [39, 40] discussed the connection between IoT and learning dynamics of the agents and adaptation to the varying
big data analytics, and provided a survey on the existing behavior of other learning agents. Also, the recent article [45]
research attempts in the domain of big IoT data analytics. Also, provided a brief survey of the existing deep RL algorithms
authors in [40] provided an overview of the existing network along with the highlights on current research areas and the
methodologies suitable for real-time IoT data analytics along associated challenges.
with the fundamentals of real-time IoT analytics, software plat- Additionally, there exist a few survey and overview papers
forms and use cases, and highlighted real-time IoT analytics in the area of UDNs [46–48]. The authors in [46] provided
issues related to network scalability, network fault tolerance, an overview of the operation of UDNs in the millimeter-wave
spectral efficiency and network delay. Moreover, the article band and presented wireless self-backhauling across multiple
[19] presented basic features, challenges and enablers for big hops to improve the deployment flexibility. Furthermore, the
data analytics in wireless IoT networks, and discussed the article [47] highlighted the key issues in incorporating M2M
importance of collaborative cloud-edge processing for live data communications in the emerging UDNs and also identified
analytics along with the associated challenges and potential different ways to support M2M communications the UDNs
enablers. from the perspective of different protocol layers including
Besides, providing QoS support in ultra-dense IoT networks physical, MAC, network and application. Moreover, authors in
is challenging due to massive connectivity, heterogeneity and [48] provided a comprehensive review on the recent advances
the resource constraints of the MTC devices as detailed later and enabling technologies for UDNs along with a discussion
in Section II. While analyzing from the energy efficiency on the widely-used performance metrics and modeling tech-
perspective, maximizing QoS usually becomes energy costly niques.
and higher energy efficiency can be achieved by considering
satisfactory QoS levels. Motivated by this, authors in [41]
provided a discussion on the need for QoS satisfaction and E. Contributions
the methods to achieve QoS satisfaction efficiently. Also, game Although several existing survey/overview articles reviewed
theory-based fully distributed algorithms were presented to en- in Section I-D have considered different aspects of mMTC
hance the energy efficiency of IoT systems while maintaining systems, ML techniques and UDNs, a comprehensive analysis
a desired QoS threshold. of the research issues involved in supporting the massive
In the direction of incorporating intelligence in 5G and number of MTC devices in ultra-dense cellular IoT networks
beyond networks, there have been some recent attempts in and a detailed review of the recent advances including ML-
applying Artificial Intelligence (AI)-based techniques to ad- assisted solutions attempting to address these challenges are
dress various issues in wireless communications. The AI missing in the literature. As highlighted earlier in Section I-
techniques can provide significant benefits in achieving ef- B, there arise several challenges while incorporating MTC
ficient management, organization and optimization of various devices in the existing LTE/LTE-A based cellular systems. The
system resources in emerging ultra-dense 5G Heterogeneous main issues include QoS provisioning to heterogeneous MTC
Networks (HetNets). In this regard, the article [20] discussed devices, addressing random and dynamic MTC traffic, trans-
the state-of-the-art AI-based techniques for intelligent Het- mission scheduling with QoS support and RAN congestion.
Net systems by considering the objective of achieving self- To this end, the overall aim of this paper is to analyze these
configuration, self-optimization and self-healing. Furthermore, different issues, to review the existing works attempting to
IEEE COMMUNICATIONS SURVEYS & TUTORIALS (DRAFT) 5

I. Introduction A. Machine-Type
Communications

B. Challenges for QoS

II. QoS Provisioning in Provisioning
Ultra-Dense Cellular IoT C. Potential Enablers for
Networks mMTC in Cellular
Networks
D. Traffic Characterization
and Modeling for MTC

A. RA Procedure in Legacy
LTE Systems

B. Failure of RA Procedure
III. Random Access and Inefficiency in mMTC
Procedure in Cellular IoT
Networks C. LTE-M: Key Features
and Channel Access
Mechanisms
D. NB-IoT: Key Features
and Channel Access
Mechanisms
A. Framework for
Performance Analysis
IV. Transmission
with QoS support
Scheduling for mMTC
with QoS Support B. Short Data Packet
Transmission and
Associated Issues

Organization A. Existing Techniques

V. Solutions for RAN
Congestion Problem in
Cellular IoT Networks B. Emerging Solutions

A. Advantages of Learning
in Wireless
Communications

B. Learning Techniques
VI. Learning-Assisted for IoT/mMTC
Solutions for RAN
Congestion Problem C. Overview of Existing
1. Exploration
ML Techniques
Strategies for Q-
Learning
D. Q-Learning for RACH
Congestion Problem 2. Performance
Enhancement of Q-
Learning
VII. Research Challenges
and Future Directions

VIII. Conclusions

Fig. 1. Structure of the Paper

overcome these challenges, to highlight potential enablers, and 4) Existing solutions towards addressing the RAN conges-
to propose ML-assisted solutions to address various challenges tion problem in cellular IoT networks are reviewed along
in ultra-dense cellular IoT networks. In the following, we with the highlights on three emerging techniques.
highlight the main contributions of this survey paper. 5) The potential benefits, challenges and promising use
1) The major challenges faced by the existing cellular case scenarios for the applications of emerging ML
IoT networks in supporting the massive number of techniques in ultra-dense cellular networks are identified
MTC devices are identified and the potential enabling and the existing ML techniques are reviewed by broadly
technologies are highlighted along with the key features, categorizing them into supervised, unsupervised and RL
traffic characterization and the application scenarios of techniques.
the mMTC. 6) A framework for the application of low-complexity Q-
2) The inefficiency of the legacy LTE RA procedure in learning in addressing the RACH congestion problem
supporting MTC devices is pointed out and its adaptation is presented along with different exploration strategies,
for mMTC systems is presented along with the main and some performance enhancement techniques are sug-
features and channel access mechanisms of emerging gested in multi-agent and dynamic wireless environ-
cellular IoT standards (LTE-M and NB-IoT). ments.
3) A mathematical framework for the performance analysis 7) Various research issues are identified and some interest-
of transmission scheduling with the QoS support in an ing future directions are presented to stimulate future
mMTC system is presented, and several limitations and research activities in the related domains.
the design aspects of short data packet transmission are
identified.
IEEE COMMUNICATIONS SURVEYS & TUTORIALS (DRAFT) 6

TABLE II
D EFINITIONS OF ACRONYMS

Acronyms Definitions Acronyms Definitions

ACB Access Class Barring NPSS Narrowband Primary Synchronization Signal
ACK Acknowledgement NSSS Narrowband Secondary Synchronization Signal
AI Artificial Intelligence NPBCH Narrowband Physical Broadcast Channel
BS Base Station NRS Narrowband Reference Signal
CE Coverage Enhancement NPDCCH Narrowband Physical Downlink Control Channel
CRA Coded Random Access NPDSCH Narrowband Physical Downlink Shared Channel
CS Compressive Sensing NOMA Non-Orthogonal Multiple Access
CIoT Cellular IoT OFDM Orthogonal Frequency-Division Multiplexing
DCI Downlink Control Information OFDMA Orthogonal Frequency-Division Multiple Access
DNC Deterministic Network Calculus PSK Phase-Shift Keying
DQCA Distributed Queuing Collision Avoidance PDU Periodic Update
eDRX Extended Discontinuous Reception PDCCH Physical Downlink Control Channel
EPDCCH Enhanced PDCCH PRACH Physical RACH
eMBB Enhanced Mobile Broadband PUSCH Physical Uplink Shared Channel
ED Event Driven RB Resource Block
FDD Frequency Division Duplex RRC Radio Resource Control
HARQ Hybrid Automatic Repeat Request RAN Random Access Network
HetNet Heterogeneous Network RA Random Access
HTC Human-Type Communications RACH Random Access Channel
H2H Human to Human RAR Random Access Response
IoT Internet of Things RAW Random Access Window
LSA Licensed Shared Access RL Reinforcement Learning
LTE Long Term Evolution SAS Spectrum Access System
LTE-A LTE-Advanced SCMA Sparse Code Multiple Access
MAC Medium Access Control SC-FDMA Single Carrier Frequency Division Multiple Access
MCL Maximum Coupling Loss SDN Software Defined Networking
MDP Markov Decision Process SL Sequential Learning
MTC Machine-Type Communications TA Timing Alignment
mMTC Massive MTC TDD Time Division Duplex
M2M Machine-to-Machine TTI Transmission Time Interval
ML Machine Learning UE User Equipment
MPRACH MTC PRACH UDN Ultra-Dense Network
MPDCCH MTC PDCCH URLLC Ultra-Reliable and Low-latency Communications
MUD Multi-User Detection UFMC Universal Filtered Multi-Carrier
NB-IoT Narrowband IoT QoS Quality of Service

F. Paper Organization II. Q O S P ROVISIONING IN U LTRA -D ENSE I OT N ETWORKS

The remainder of this paper is organized as follows: Section In this section, we first discuss various aspects related to
II identifies the main features, application areas and the QoS provisioning in ultra-dense IoT networks in the general
potential enablers for mMTC in cellular networks along with context, and then provide specific details related to cellular IoT
a discussion on various issues associated with QoS provi- scenarios. The International Mobile Telecommunication (IMT)
sioning in ultra-dense IoT networks. Section III highlights vision for 2020 and beyond envisions to provide the connec-
the inefficiency of the conventional LTE RA procedure in tion density target of about 106 devices per square kilometers
mMTC systems along with the basics of RA procedure in [3]. However, the available radio resources and communication
legacy LTE systems, and then present key features and channel infrastructures are limited and there is a significant amount
access mechanisms in two emerging cellular IoT standards, of cost involved while acquiring new radio resources such as
namely, LTE-M and NB-IoT. Also, the characterization of spectrum and building new infrastructures. Because of this
MTC traffic is presented along with the 3GPP-based MTC issue, existing communication networks need to be made as
traffic models and related works. Subsequently, Section IV efficient as possible to support the massive number of devices,
provides a mathematical framework for the performance anal- thus leading to the concept of ultra-dense IoT networks. Due
ysis of transmission scheduling with QoS support in the to diverse types of emerging IoT services and heterogeneous
mMTC systems, and also present various design aspects and capabilities of MTC devices, QoS provisioning in ultra-dense
limitations of short data packet transmission in the mMTC IoT networks is a crucial challenge.
environment. Section V presents a review of the existing so- The overall network performance and QoS of emerging
lutions for RAN congestion problem in cellular IoT networks Internet protocol-based networks significantly depend on the
along with some emerging solutions while Section VI presents effective management of instantaneous traffic flowing in the
the advantages and challenges of ML techniques in wireless network. In wireless IoT networks, the instantaneous aggre-
IoT systems and also provides an overview of the existing gated traffic at the IoT gateway can be bursty and may greatly
ML techniques. Also, it provides a detailed explanation on exceed the average aggregated traffic since it aggregates pe-
the Q-learning mechanism from the perspective of addressing riodic transmissions from a large number of sensor nodes
RAN congestion minimization along with some Q-learning with different periods and frame sizes [12]. Due to this,
performance enhancement techniques. Finally, Section VII there may occur congestion during possible bursty intervals
provides some research challenges and future directions, and and much higher backhaul link (from aggregators to the
Section VIII concludes this paper. To improve the flow of this cloud) bandwidth is needed than that required for the non-
paper, we provide the structure of the paper in Fig. 1 and the bursty traffic. In this direction, one of the important research
definitions of acronyms in Table II. challenges is how to make the aggregated traffic as close to
IEEE COMMUNICATIONS SURVEYS & TUTORIALS (DRAFT) 7

TABLE III
A DVANTAGES OF ACHIEVING DESIRED Q O S LEVELS INSTEAD OF MAXIMIZING Q O S

Advantages Related Causes

1. Reduction of energy consumption 1. Extraneous energy will be wasted while maximizing QoS.
2. Better compliance with the fixed data rate services 2. No need to maximize data rates for fixed data rate services
such as video surveillance and online gaming
3. A good perceived performance at the users’ end 3. End-users are usually insensitive to small changes.
in QoS levels, allowing the room for energy saving
4. Better support for emerging application-oriented networks 4. Desired QoS level is required only within a specific coverage area.
5. The set of feasible regions of optimization solutions are enlarged. 5. Relaxation of global optimum for the QoS maximization makes
the mathematical problems less restrictive.
6. Adaptive resource allocation problem leads to the cost-effective solutions. 6. No need to waste additional radio resources by considering
the conventional optimization assumptions such as full-buffer traffic.

the average traffic as possible. maximization may lead to unnecessary energy consumption
In contrast to the conventional HTC traffic, there are sev- and achieving satisfactory levels of QoS may be sufficient to
eral unique features of MTC traffic [49] which need to be balance other performance metrics of systems such as energy
considered while devising transmission scheduling and traffic efficiency. In contrast to the existing works related to the
management strategies for wireless IoT networks. The amount maximization of QoS, the objective of achieving satisfactory
of small packets in IoT-type networks could become signifi- QoS levels can provide several benefits as highlighted in Table
cantly large due to the resource-constrained sensor devices and III [41].
the transmission of short packets from mMTC devices [11]. As Moreover, in time-critical MTC applications, several re-
compared to the dominant downlink traffic in the conventional quirements in terms of low end-to-end delay, deterministic
cellular systems, uplink to downlink ratio for the MTC traffic delay, bounds on systematic delay variations and linear delay-
is much higher. Furthermore, MTC devices usually have payload (packet size) dependence need to be considered [53].
limited power budget and the MTC traffic consists of packets The packet arrival period in the HTC systems such as multi-
with the short payload length. Also, MTC traffic may arrive in media ranges from 10 ms to 40 ms whereas this may range
the batch-mode due to high density of devices and correlated from about 10 ms to several minutes in MTC systems [54].
transmission [50]. Moreover, the standard Poisson process may In addition, some applications such as data reporting in the
not be suitable for modeling the MTC traffic since the MTC smart grid have deterministic (hard) timing constraints and
transmissions usually exhibit spatial and temporal synchro- serious consequences may occur in the case of violence of
nism. In addition, in contrast to the conventional voice traffic these constraints. In this regard, multiplexing massive accesses
which has a constant sampling rate for a given codec, MTC with these diverse QoS characteristics effectively is a crucial
traffic usually comprises of different packet sizes and inter- challenge in ultra-dense IoT networks.
arrival patterns [51]. Different MTC applications have distinct In the following subsections, we describe several aspects
characteristics and service requirements such as priority and of MTC systems, highlight existing challenges for QoS provi-
delay constraints, thus leading to the need of separate traffic sioning in ultra-dense cellular IoT networks, present potential
modelling and scheduling schemes to incorporate mMTC enablers for the incorporation of MTC devices in cellular IoT
devices in the current LTE-based cellular networks. systems, and then present the characterization and modeling
of the MTC traffic.
In addition, MTC devices have completely different QoS
requirements than that of the conventional HTC devices. MTC
nodes are usually constrained in terms of battery power and the
A. Machine-Type Communications
employed protocols need to be as energy-efficient as possible.
Although the previous research in the area of wireless sensor MTC has got a wide variety of application areas ranging
network protocols mainly focused on monitoring applications from industrial automation and control to environmental mon-
based on low-rate delay-tolerant data collection, the current itoring towards building an information ambient society. The
IoT-based research has moved to several new applications main applications of the MTC are listed below [6].
such as eHealthCare, industrial automation, military and smart 1) Industrial automation and control: This class includes
home. These applications have different QoS requirements in several scenarios such as production on demand, quality
terms of delay, throughput, priority, reliability and different control, automatic interactions among machines, opti-
traffic patterns such as event-driven, periodic and streaming mization of packaging, logistics and supply chain, and
[52]. To this end, it is crucial to consider these distinct QoS inventory tracking.
features while designing transmission and access techniques 2) Intelligent transportation: Under this category, MTC
for the MTC devices. finds applications in different scenarios such as logistic
In the above context, several existing works deal with services, M2M assisted driving, fleet management, e-
the maximization of QoS while attempting to minimize the ticketing and passenger services, smart parking and
energy consumption. Several techniques such as sleep mode smart car counting.
optimization, power control mechanisms, adaptation of the 3) Smart-grid: This category include various application
data rates and learning-assisted algorithms have been em- scenarios including automatic meter reading, power de-
ployed in various settings [41]. However, the objective of QoS mand management, smart electricity distribution and
IEEE COMMUNICATIONS SURVEYS & TUTORIALS (DRAFT) 8

patrolling, online monitoring of transmission lines and In addition, the 3GPP has identified the following per-
transmission tower protection [55]. formance objectives to support mMTC in the emerging air
4) Smart environment: Several application scenarios in- interface 5G New Radio (NR) [5, 58].
cluding smart homes/offices/shops, smart lighting, smart 1) Very high connection density of about 106 devices per
industrial plants, smart water supply, environmental km2 in an urban environment
monitoring and green environment can be enabled with 2) Ultra-low complexity and low-cost IoT devices/networks
MTC. 3) Battery life in extreme coverage beyond 10 years with
5) Security and public safety: Several applications such the battery life evaluated at 164 dB MCL, and a battery
as remote surveillance, personal tracking and public capacity of 5 Wh.
infrastructure protection can be considered under this 4) Maximum Coupling Loss (MCL) of about 164dB for a
category. data rate of 160 bps at the application layer
6) e-Health: In this application area, various scenarios exist 5) Latency of about 10 seconds or less on the uplink to
such as tracking or monitoring a patient or a segment of deliver a 20-byte application layer packet (measured at
an organ in a patient, identification and authentication 164dB MCL)
of patients, diagnosing patient conditions and providing
real-time information on patients health related data to
B. Challenges for QoS Provisioning in Ultra-Dense IoT Net-
the remote monitoring center.
works
Since MTC applications are significantly different from
their HTC counterparts, they have distinct QoS requirements There arise several challenges in incorporating MTC devices
with different service features such as time-controlled, time- in LTE/LTE-A based cellular networks. First, the massive
tolerant, small data transmission, low or no mobility, group- number of devices try to access the scarce network resources
based connection, priority-based transmissions and low power in a short period of time and there may arise the need of either
consumption [56]. In this regard, the 3GPP has identified utilizing the available resources efficiently or allocating addi-
the following 14 features for M2M communications1 : (i) low tional bandwidth to incorporate these devices [59]. Secondly,
mobility, (ii) time-controlled, (iii) time-tolerant, (iv) packet there are significant differences in the transceiver properties
switched only, (v) mobile originated only, (vi) small data and the applications of MTC devices from the existing LTE-
transmission, (vii) infrequent mobile terminated, (viii) M2M based user terminals [28]. In most of the applications, MTC
monitoring, (ix) priority alarm message, (x) secure connection, devices consume low power and have intermittent low rate
(xi) location specific trigger, (xii) network-provided destina- transmissions. Furthermore, due to the need of cost-effective
tion for uplink, (xiii) infrequent transmission, and (xiv) group- deployment of massive devices, MTC devices have degraded
based policing and addressing. transceiver performance and reduced coverage as compared
Furthermore, the 3GPP has specified various general re- to the LTE user terminals. Besides, their effects in the com-
quirements for the MTC systems [6, 9] in order to effectively munication performance of the existing LTE-A users need to
operate MTC devices and also to establish successful linkage be monitored and mitigated carefully. In this regard, one of
between an MTC subscriber and the network operator. Some the important research questions is how to provide concurrent
of the main technical requirements include: (i) providing a access to a large number of MTC devices without degrading
control mechanism to the network operators for the addi- the QoS of the existing cellular users.
tion/removal/restriction of individual MTC device features, (ii) Since a network interface is fully utilized during the peak
exploring a peak reduction mechanism for data and signaling time, the devices may not be able to send or receive data and it
traffic when a number of MTC devices concurrently attempt is crucial to optimize the peak traffic in the emerging content-
for their data transmissions, (iii) yielding a mechanism to centric wireless networks [60]. Furthermore, as highlighted
restrict downlink data traffic and also limiting access towards earlier in Section II-A, the emerging MTC applications are
a specific access point name in case of network overload, and quite different from the traditional HTC applications due to
(iv) investigating techniques to maintain efficient connectivity unique features such as group-based communications, time-
for a large number of MTC devices and to lower the corre- controlled, small data transmissions, and low or no mobility
sponding energy consumption. [56, 61]. These distinct features of MTC applications result
In comparison to the conventional HTC, the emerging MTC in diverse QoS requirements and it is important to take
has the features of infrequent transmissions and low data these QoS requirements into account while devising multiple
rates. Also, the size of signalling data packets can be much access techniques for future cellular IoT networks. The main
larger than the size of user data packets in M2M applications performance indicators of an mMTC system are the number
[37]. Furthermore, although M2M devices need to transmit of concurrent connections to be supported, energy efficiency
small amounts of data, communication infrastructure may get and network coverage [62].
congested if a huge number of M2M devices attempt to access Existing cellular networks face the following major prob-
the network near-simultaneously [57]. Hence, the performance lems in supporting MTC devices [9, 63, 64]. In Figure 2, we
of the existing cellular standards has to be evaluated for this present the pictorial representation of these issues in the form
emerging type of traffic. of device-level and network-level challenges.
1) Highly dynamic traffic and random access time:
1 For the description of these features, interested readers may refer to [9]. The data traffic arisen from the MTC devices is highly
IEEE COMMUNICATIONS SURVEYS & TUTORIALS (DRAFT) 9

Congestion
C ti ini
Need for
radio Distributed
improving
Need for access,core radio,
network
highly and signalling computing and
coverage
scalable networks caching
network resources

Network-level
Challenges

Challenges of
existing cellular
networks for
mMTC

Device-level
Challenges

Dynamic
traffic and Diverse QoS
random requirements
access time Ultra-low Low Small data-
device battery packets
complexity lifetime

Fig. 2. Challenges of existing cellular networks to support emerging massive machine-type communications.

dynamic in nature as compared to more predictable HTC I-B, the incorporation of massive MTC devices in the
traffic. Furthermore, there arises a need to handle the existing LTE/LTE-based cellular network may result in
mixed traffic models with the event-driven and periodic congestion in different segments of the network includ-
traffics. In addition, the existing contention-based radio ing RAN, the core network and the signalling network.
access schemes will need to coordinate random trans- 7) Highly scalable network: Because of the need to sup-
missions from the massive number of devices [13]. port a significantly large number of connected devices
2) Ultra-low device complexity: Due to the requirement ranging from a factor of 10× to 100× as compared to
of cheap MTC devices for mass deployment, the devices the cellular devices, it is crucial to maintain the system
are constrained in terms of computational and memory performance with the increase in the connection density.
resources, thus providing the limited performance. 8) Need for improving network coverage: There arises
3) Low battery lifetime: Because of the cost and space significant shrinkage in the link budget due to the
constraints, MTC devices are limited in their battery reduced capability of MTC devices. In order to increase
capacity. Furthermore, due to distributed nature of IoT the coverage to the areas where MTC devices are
devices and the involved cost-issues in replacing the bat- deployed (such as deep inside a building), LTE release
teries, the battery lifetime of MTC devices is expected 13 targeted the coverage extension of at least 15 dB for
to be more than 10 years with the battery capacity of the MTC devices. This coverage improvement enables
5 Wh, thus leading to the need of investigating power the support of the devices in the locations where the
saving methods for ultra-dense cellular IoT networks. conventional cellular networks face difficulty.
4) Small data packet transmissions: In addition to the 9) Distributed radio, computing and caching resources:
huge signalling burden associated with a large number With the recent trend of migrating communications
of small packet transmissions from MTC devices, there networks from the connection-oriented to the content-
arise other challenges such as the requirement of higher oriented nature, it is important to investigate syner-
resource granularity and efficient channel coding for gies among communications, computing and caching
short block lengths in contrast to the channel coding resources which are distributed across different devices
schemes designed for long packets in the conventional in ultra-dense IoT networks [19]. However, the con-
cellular systems [14]. ventional cellular networks based on the centralized
5) Diverse QoS requirements: MTC devices have diverse management are sluggish in terms of network resource
QoS requirements in terms of data rate and latency management and they need to evolve to deal with the
requirements and existing cellular technologies need to management of distributed resources.
adapted to handle these features.
6) Network congestion: As highlighted earlier in Section Towards modeling and analysis of the QoS of wireless
networks, one of the important mathematical tools is Deter-
IEEE COMMUNICATIONS SURVEYS & TUTORIALS (DRAFT) 10

ministic Network Calculus (DNC), which is useful to calculate

delay parameters such as delay bound, backlog bound and Collaborative
Advanced edge-cloud
other service quality parameters by utilizing the traffic/packet transmission processing Flexible
arrival and service curves [65]. This DNC tool enables the scheduling waverforms
determined boundary analysis for the system performance and techniques
offers a strict service guarantee by considering the worst- Dynamic
case scenarios. The main QoS metrics that can be evaluated Low signalling resource
overhead MAC allocation
include delay bound and backlog bound. The metric delay protocols schemes
bound represents the maximum between arrival and service
curves while the backlog bound denotes the maximum vertical
deviation between these two curves. Enablers for mMTC Advanced
Compressive in cellular IoT spectrum
In addition to providing high capacity to the fairly limited networks
sensing based sharing
number of traditional user equipments to support high data MUD techniques
rate services such as video streaming, the air interface of 5G
cellular network has to provide connectivity to the massive
number of concurrent transmissions coming from the MTC Constant Clustering
envelope and data
devices [66]. Also, the exchange of signalling information based aggregation
needs to be minimized both in the uplink and downlink due modulation
schemes like Advanced RA
to a large number of MTC devices to be supported with schemes such SDN and
CPM
the limited available radio resources. Furthermore, the RA as NoMA, CRA virtualization
SCMA and techniques
procedure at the device-side should be simplified as much as DQCA
possible by shifting the burden to the network side/eNodeB
due to the resource constrained and low-cost nature of MTC
devices. Fig. 3. Potential enabling techniques for mMTC in cellular IoT networks.
Moreover, the conventional centralized approaches for con-
gestion management in cellular networks are not scalable To this end, it is important to look into the mMTC network
as desired by the MTC systems and also the distributed design problem from a different perspective than the tradition
scheduling approaches can not easily acquire the knowledge approach followed for the HTC systems.
about the network load and requirements of other applications In Figure 3, we present the main enabling techniques being
[67]. Furthermore, the conventional congestion management considered to facilitate the incorporation of mMTC in the
process is mostly a reactive process instead of the proactive upcoming cellular IoT networks. In the following, we briefly
one needed for MTC devices. By deferring and shaping describe these enabling techniques.
transmissions at the source itself in a network and being aware 1) Flexible waveform design: The design of flexible wave-
of the underlying application properties, better congestion forms can enable the in-band mMTC channels within
management can be obtained for MTC devices [67]. the LTE carrier [14]. In this regard, the traditional
Authors in [29] highlighted the difference between HTC waveforms designed for HTC communications need to
over cellular and MTC over cellular in terms of various param- be adapted to support mMTC while considering various
eters such as uplink, downlink, subscriber load, device types aspects such as end-to-end latency, robustness against
and requirements in terms of delay, energy, signalling and time and synchronization errors, out-of-band radiations,
cellular architecture. Furthermore, the signalling overhead in spectral efficiency and transceiver complexity [68].
the uplink and downlink MTC links are analyzed considering 2) Dynamic resource allocation techniques: The 3GPP
the SMS-type raw data size (< 248 bytes) and email-type raw suggested to allocate RACH resources dynamically to
data size for smart metering and vehicular sensing applications address the RAN congestion problem [69]. For exam-
via experimental measurements. It has been shown that the ple, by deciding the number of preambles adaptively
signalling overhead for the downlink control messages is without knowing the number of devices and access
considerably higher than for the uplink case, and higher probability, the RACH throughput can be maximized
signalling overhead occurs in vehicular applications than in [70]. Furthermore, since this approach can dynamically
the smart metering applications. change the size of the RACH resource pool and other re-
sources, the total data collection time from the resource-
constrained MTC devices can be minimized in delay-
C. Potential Enablers for mMTC in Cellular Networks sensitive/emergency applications [71]. The two main
Due to some distinct transitions while going from the issues for employing this process are the requirement
conventional HTC platform to the emerging mMTC such as of estimating the number of contending devices in an
from the larger packet sizes to the smaller packet sizes, from RA slot and determining the preamble pool size.
the downlink-focused communication scenario to the uplink 3) Advanced spectrum sharing methods: Although both
dominant, and from high data rate to low data rate transmis- the licensed and unlicensed bands can be exploited
sions from MTC devices, mMTC systems have new design for mMTC applications, lack of QoS guarantees in the
requirements than those of the conventional HTC systems. unlicensed band becomes highly problematic [4]. In this
IEEE COMMUNICATIONS SURVEYS & TUTORIALS (DRAFT) 11

regard, emerging advanced spectrum sharing techniques MTC devices in cellular networks [29].
such as Licensed Shared Access (LSA) and Spectrum 10) Advanced transmission scheduling techniques: The
Access System (SAS) [72] could be potential solutions transmission scheduling techniques designed for cellu-
for mMTC applications since they can provide better lar IoT systems should be able to accommodate the
interference characterization. MTC devices with heterogeneous QoS requirements in
4) Clustering and data aggregation schemes: By group- addition to the legacy cellular users. In this regard,
ing MTC devices into smaller clusters based on some advanced scheduling techniques such as latency-aware
suitable criteria such as geographical locations or QoS scheduling [76], fast uplink grant [77] and learning-
requirements and then aggregating the individual device assisted scheduling [10] seem promising to schedule the
data at the MTC gateway/aggregator, the RAN conges- sporadic transmissions from a huge number of MTC
tion can be significantly minimized [62]. Furthermore, devices over limited RACH resources.
the investigation of energy-efficient clustering schemes 11) Collaborative cloud-edge processing: Cloud comput-
facilitates the deployment of low-power MTC devices ing platform has very high computational and storage
[62]. capacity, and has a global view of the network but
5) Software Defined Networking (SDN) and virtualiza- is not suitable for delay sensitive applications. On the
tion techniques: Based on the functionalities of MTC other hand, edge-computing is suitable for applications
devices and their QoS requirements, a physical cellu- demanding low delay and high QoS but has lower
lar network can be virtualized into different networks computational resources and storage capacity. In this
such as industrial, vehicular, smart grids and emergency regard, collaborative processing between these two plat-
networks, with all these networks sharing the same set forms will be a promising approach to address various
of radio, computing and networking resources [73]. The issues including latency minimization [78], dynamic
dynamic sharing of resources and the reconfiguration spectrum sharing [79], peak traffic management and data
of network elements among thus virtualized networks offloading in ultra-dense IoT networks [19].
can be carried out by utlizing an SDN paradigm which The ML-assisted techniques, detailed later in Section VI,
decouples the control plane from the data plane and can address various issues related to self-configuration, self-
incorporates the capability of programming in the IoT optimization and self-healing in emerging wireless networks
network. and seem promising in facilitating the implementation of the
6) Advanced RA schemes: Several emerging RA schemes most of the technology enablers listed in Fig. 3 towards
such as Non-Orthogonal Multiple Access (NOMA), enhancing the performance of mMTC systems. However, the
Sparse Code Multiple Access (SCMA), Coded Random ML techniques should be as simple as possible to be applied
Access (CRA) [14] and distributed queueing based ac- in the MTC devices and the investigation of low-complexity
cess protocol [74] can be considered as the promising adaptive ML techniques is one of the emerging future research
enablers for the mMTC in cellular networks. directions as highlighted later in Section VII.
7) Constant envelope coded-modulation schemes: Due
to space/cost constraints, MTC devices need to use low-
cost amplifiers which are prone to non-linearities and D. Traffic Characterization and Modeling for mMTC Systems
hardware imperfections. In this scenario, constant en- The characterization and modeling of mMTC traffic is
velope signals can enable the non-linear power-efficient crucial to support MTC devices in the existing cellular net-
and cost-effective operation at the MTC devices. There- works due to various reasons specified in the following. The
fore, constant envelope coded modulation schemes such incorporation of MTC devices in cellular networks may cause
as Continuous Phase Modulation (CPM) can be consid- harmful interference to the existing cellular users and may
ered as enablers for the mMTC [14]. significantly degrade the system performance of LTE/LTE-
8) Compressed Sensing (CS)-based Multi-User Detec- A based cellular systems. To this end, it is important to
tion (MUD): The amount of collisions in the IoT analyze the impact of MTC traffic on the existing cellular
access network can be further minimized by employing users by utilizing suitable interference modeling in realistic
advanced interference cancellation receivers. As an ex- wireless environments [80]. Furthermore, suitable interference
ample, the CS-MUD can enhance the resource efficiency mitigation, resource allocation and resource sharing schemes
and serve higher number of users by using the combi- need to be investigated to ensure the sufficient protection of the
nation of non-orthogonal RA and joint detection of user cellular users against harmful interference caused by the mas-
data and activity [14]. In this regard, the combination of sive number of MTC devices and by utilizing the given MTC
advanced MAC protocols with the CS-based MUD can traffic models, these schemes can be designed in an efficient
be utilized by exploiting the sparse joint activity in the manner. Moreover, since MTC traffic is uplink dominant and
mMTC environment [75]. the rigid QoS support framework of LTE designed for voice
9) Low signalling overhead MAC protocols: One of and data services may not be capable of addressing specific
the main technical challenges in an mMTC system is QoS requirements of MTC traffic in terms of latency, jitter and
to reduce the amount of signalling overhead generated packet loss, suitable transmission scheduling techniques need
by the MTC devices and the design of low-signalling to be investigated to support a large number of MTC devices
overhead protocols will facilitate the deployment of while fulfilling their specific QoS requirements. Besides, to
IEEE COMMUNICATIONS SURVEYS & TUTORIALS (DRAFT) 12

2.5
investigate suitable traffic management schemes such as peak
traffic reduction in wireless IoT networks, it is essential
to understand and characterize the traffic models applicable 2

for a particular IoT application [13]. In addition, the traffic

characteristics depend on the application scenarios and the
1.5
MTC devices usually have heterogeneous traffic patterns in
terms of their amplitudes, starting times and activation periods
[52]. 1

Existing traffic models in telecommunication systems can be

categorized into: (i) source traffic models, mainly applicable
0.5
for video, data and voice transmissions, and (ii) aggregated
traffic model applicable for Internet, high-speed links and
backbone networks [81]. Since an IoT network consists of a 0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
large number of sensors or MTC devices which are usually
controlled by a gateway/server, IoT traffic at the gateway
usually fits into the aggregated traffic model. In this regard, Fig. 4. Access intensity for two 3GPP based MTC traffic models.
the 3GPP has defined the following two types of aggregated
MTC traffic models [69]. realize, it is less precise than the source traffic modeling since
1) Model 1: Uniform distribution over a duration T in it is not able to capture the real traffic features at the source
which MTC devices access the network uniformly over a level [49, 81]. On the other hand, source traffic modeling treats
period of time, i.e., in a non-synchronized manner. This the traffic for every devices separately, and hence is more
model does not take account of the correlation between precise. However, modeling source traffic becomes complex
the transmissions of the devices. for a large number of source devices. Therefore, it is crucial
2) Model 2: Beta distribution over T in which a large to investigate suitable traffic models which can combine the
amount of MTC devices access the network in a highly benefits of both the source and aggregate traffic models. In
synchronized manner. This model generates correlated this regard, coupled Markov modulated Poisson processes [81]
traffic in a specific time interval. seems a promising approach, which has higher accuracy than
For the first model, the following function can be consid- the aggregated modeling and has lower complexity than the
ered.  conventional source traffic modeling.
Pi , 0 ≤ t ≤ τi , Considering a variety of applications, the MTC traffic can
fi (t) = (1)
0, τi ≤ t ≤ Ti , be categorized into the following three traffic patterns: (i)
where Pi denotes the amplitude of the traffic profile for Periodic Update (PU), (ii) Event-Driven (ED) and (iii) payload
the ith device and τi /Ti represents the corresponding duty exchange [82]. The PU traffic has a regular pattern, constant
cycle. Similarly, for the second model, the probability density data size and is non-real time type (example: smart meter
function of the Beta distribution, is given by [69] reading) while the ED traffic has a variable pattern, varying
data size and is a real time traffic (example: health emergency
1
p(t, α, β) = t(α−1) (1 − t)(β−1) , (2) alarming). On the other hand, payload exchange traffic follows
B(α, β) either of the above traffic types and it may be of constant size
where t denotes a single realization over the time axis, α > or variable size, real time or non-real time depending on the
0, β > 0 are the scale parameters, and B(α, β) denotes the application scenario. In addition, there exist three main types
Beta function, which is a normalization constant to ensure of traffic shaping policies [83]: (i) traffic shaping for bulk
R 1 probability equals to 1 and can be defined as
that the total applications where each flow is assigned a fixed bandwidth,
B(α, β) = 0 xα−1 (1 − x)β−1 dx [13]. (ii) traffic shaping for the aggregate traffic, and (iii) time-based
Considering M number of MTC devices and their activation traffic shaping which is applied only at the peak-time to reduce
periods between t = 0 and t = T , the RA intensity for the congestion and cost.
mth device is given by the probability distributions either in The uplink traffic generated from the sensors in most of
(1) or (2) depending on the employed model. The number of MTC applications is heterogeneous and can be classified into
arrivals in the ith access slot is given by [69] [84]: (i) non real-time with no task completion deadline, (ii)
Z ti+1 soft real-time with the decreased utility if the deadline not
Iaccess (i) = M p(t)dt, (3) met and (iii) firm real-time having zero utility if the deadline
ti is not met. As an example, industrial M2M traffic has very low
where ti denotes the time of the ith access opportunity and latency requirements in the order of a few milliseconds [85].
p(t) given by (1) or (2). The distribution of access attempts In general, the PU traffic is periodic with tight service deadline
should beRlimited over a certain observation period T in such while the ED traffic is random with all three traffic categories,
T
way that 0 p(t)dt = 1. Figure 4 illustrates the RA intensity i.e., non real-time, firm or soft real-time. From the scheduler
of two 3GPP-based MTC traffic models with T = 1. designer perspective, it is crucial to maximize a system utility
Although aggregated traffic modeling is suitable for scenar- metric in order to maximally satisfy the delay requirements of
ios involving a large number of devices and is less complex to all the classes [84].
IEEE COMMUNICATIONS SURVEYS & TUTORIALS (DRAFT) 13

Moreover, possible network applications in wireless IoT queue size and setting a bandwidth limit so that aggregate
networks can be classified into the following [56]. traffic size in the active queue does not exceed the limit.
However, scheduling techniques cause delay for processing the
1) Elastic applications: This category corresponds to more
network demand and it is crucial to investigate suitable tech-
traditional HTC applications such as electronic email,
niques to minimize the delay introduced by traffic scheduling
file transfer as well as the downloading of remote data
mechanisms. Furthermore, a suitable bandwidth limit should
from the MTC servers. These applications are mostly
be applied to balance the trade-off between latency and energy
delay tolerant in nature and the user utility usually has
saving [83].
diminishing marginal improvements with the incremen-
tal increase in the achievable data rate.
2) Hard real-time applications: These applications have III. R ANDOM ACCESS P ROCEDURE IN C ELLULAR I OT
a desired delay constraint with hard real-time require- N ETWORKS
ments. Beyond the desired time frame, there is no An MTC device must go through the access procedure to
additional utility gain while increasing the data rate establish a connection to the Base Station (BS)/eNodeB/access
and the user utility becomes the step function of the point mainly in the following situations [33]: (i) while es-
achievable data rate. tablishing an initial access to the network, (ii) while receiv-
3) Delay adaptive applications: Some delay sensitive ing/transmitting new data and MTC device is not synchronized
applications can occasionally tolerate a small delay with to the network, (iii) during the transmission of new data when
a certain delay-bound violation and the packet dropping no scheduling resources are allocated on the uplink control
probability. The user utility in these applications (such channel, (iv) to perform a seamless handover, (v) in order to
as remote monitoring of e-Health services) deteriorates re-connect to the network in the case of radio link failure.
rapidly when the achievable data rate becomes less than The RA methods in the LTE-based cellular systems can
the required intrinsic data rate. be categorized into contention-based (for delay-tolerant access
4) Rate-adaptive applications: These applications try to requests) and contention-free (for delay-sensitive requests)
adjust their transmission rates based on the available schemes [87]. Out of these, the contention-based scheme is of
radio resources with the moderate delays. A highly the main interest here due to the limitation in the number of
efficient scheduler is needed to enhance the performance available Resource Blocks (RBs) as compared to the massive
of these applications in time-varying channel conditions. number of access requests to be supported. In the contention-
based RA approach, a huge number of MTC devices have
Traffic shaping, also called packet shaping, delays certain
to select the same preambles because of the limitation in the
types of data packets in order to optimize the overall per-
available preambles in the existing LTE-based cellular sys-
formance of a network. To achieve the optimized network
tems, and this results in significantly high number of collisions
performance, Internet traffic thesedays is intentionally shaped
in the access network and subsequently leads to the RAN
into ON/OFF pattern [86]. Also, ON/OFF pattern is generated
overload or radio access congestion problem in ultra-dense
due to some inherent characteristics of applications such as
IoT networks. In this direction, one important question to be
HTTP web browsing and MapReduce operation at the data
answered is how to concurrently support the massive number
center/server. The main benefits in performing ON/OFF traffic
of MTC devices in ultra-dense cellular IoT networks without
shaping include: (i) reduction of computing overhead at the
affecting the performance of the existing cellular devices by
server-side, (ii) energy saving at the wireless terminals and (iii)
using the current communication technologies/standards.
minimizing the bandwidth waste while delivering streaming
In the following, we briefly describe the RA procedure in
services. However, this On/OFF traffic shaping faces several
the legacy LTE systems and its inefficiency in handling the
challenges such as the impact on packet drop probability,
massive number of devices in the mMTC environment, then
harmful effect on other real-time applications and weakening
present some adaptations made to support MTC devices in
the congestion control function of the transmission control
cellular networks, and subsequently highlight the main features
protocol [86]. To address these issues, it is crucial to de-
and access mechanisms in emerging cellular IoT standards,
sign suitable models to characterize the relation among the
namely, LTE-M and NB-IoT.
associated parameters of ON/OFF traffic such as the ratio
of ON/OFF duration, burst size, and burst transmission rate,
and also the models for packet loss probability and temporary A. RA Procedure in Legacy LTE Systems
congestion caused by the bursty transmissions. After an eNodeB broadcasts the system information to the
Another approach to manage the peak-traffic is to employ devices, the contention-based RA procedure follows a four-
demand-side management, which adopts suitable measures at stage message handshake procedure as depicted in Fig. 5
the customer-side/sensor-side to optimize the overall network [33, 88], which mainly involves the following four stages: (i)
performance [83]. On one hand, the demand profile can be RA preamble transmission from the device to the eNodeB
flattened to limit the amplitude fluctuations while simultane- (Message 1), (ii) RA Response (RAR) from the eNodeB to
ously accommodating the same amount of traffic volume. On the device (Message 2), (iii) connection request message from
the other hand, the utilization profile of network resources the device to the eNodeB (Message 3), and (iv) connection
can be flattened by rescheduling or delaying the services [60]. resolution message from the eNodeB to the device (Message
Some approaches for traffic scheduling include limiting the 4).
IEEE COMMUNICATIONS SURVEYS & TUTORIALS (DRAFT) 14

In the first stage, each device randomly selects an RA Device eNodeB

preamble from the set of available preambles broadcasted System Information
by the eNodeB during the initial network synchronization Broadcast
phase and sends the RA request (Message 1) by transmitting
thus selected preamble in an RACH. At this stage, the User
Equipments (UEs) just transmit the selected preambles and not

4-stage message handshake RA Procedure

Random Access Preamble (Message 1)
the device IDs. In the second stage, the eNodeB acknowledges
On the PRACH
the received distinct preambles with an RA response (Message
2) which includes the preamble index being acknowledged,
Random Access Response (Message 2)
instructions for the timing alignment and the command for
the RB allocation. Subsequently, in the third stage, the UE On the PDCCH
recognizes the RA response addressed to it by noting the
preamble index it has used for the RA request in the first Connection Request (Message 3)
step and utilizes the dedicated RB on the Physical Uplink On the PUSCH
Shared Channel (PUSCH). The devices which made current
Connection Resolution (Message 4)
transmissions of the RA request with the same preamble in
On the PDSCH
the first stage will be instructed to use the same RB in the step
3 and such transmissions will go through collisions. On the
other hand, for the packets (which contain the corresponding
device IDs) which are successfully decoded in step 3, the Connection set-up complete, service
eNodeB sends a contention resolution message (Message 4) request
to the corresponding devices.
In this RA procedure, after sending the preamble in the Fig. 5. Illustration of four-stage message handshake-based RA procedure
RA request (Message 1), the device sets an RAR window and in LTE-based cellular systems (PRACH: Physical Random Access Channel,
PDCCH: Physical Downlink Control Channel, PUSCH: Physical Uplink
waits for the eNodeB’s response with an uplink grant (Message Shared Channel, PDSCH: Physical Downlink Shared Channel).
2) in the RAR message. If the UE successfully receives its
Message 2 within the defined RAR window, the UE sends the
Radio Resource Control (RRC) connection request (Message 3) Failure of Message 3 transmission: This occurs due
3) to the eNodeB. At this stage, the device starts the Message to the failure in transmitting Message 3 to the eNodeB
4 timer and waits to receive its own RRC connection setup by employing the Hybrid Automatic Repeat Request
message (Message 4) from the eNodeB [89]. (HARQ) process at the device.
4) Failure of Message 4 reception: This occurs due
In the above-mentioned RA procedure, the physical-layer
to the failure in receiving Message 4 by the devices
mapping of RACHs is called Physical RACHs (PRACHs)
within the Message 4 expiration time while using HARQ
which are time-frequency blocks specified by the eNodeB.
transmission from the eNodeB either due to insufficient
The periodicity of RA slots is broadcasted by the eNodeB
PDCCH resources or an imperfect channel condition.
in terms of the PRACH configuration index, which varies
between every 1 ms (i.e., a maximum of 1 RA slot per 1 Out of 64 preambles used in LTE-A networks, 54 preambles
subframe) to 20 ms (i.e., the minimum of 1 RA slot every are used for the contention-based access, while the remaining
2 frames) [33]. The transmission scheduling in terms of time 10 preambles are reserved for the contention-free access which
and frequency depends on the configuration of the PRACH. is needed for high-priority services such as handover. In every
For example, for the PRACH configuration index of 6, there 5 ms, there arises an access opportunity and 200 access oppor-
will be RACHs in every 5 ms within a bandwidth of 180 kHz, tunities per second. This corresponds to the absolute maximum
with a duration ranging from 1 ms to 3 ms. capacity of 10, 800 preambles per seconds in the absence of
access collisions [33]. However, due to ALOHA type access
protocol and random backoffs, performance becomes much
B. Failure of RA Procedure and Its Inefficiency in mMTC
lower than this maximum limit in practical cellular systems.
systems
Furthermore, the situation becomes worse while supporting
The aforementioned RACH procedure in LTE/LTE-A based the MTC devices. Besides the problem of supporting higher
networks may fail due to the following reasons [89]. number of nodes in ultra-dense IoT networks, other perfor-
1) Failure of preamble transmission: This occurs either mance metrics such as access delay and energy consumption
due to the collision of RACH preambles (caused due to are important to be considered.
concurrent transmission of the same set of preambles To study the performance of the aforementioned 4-stage
by more than one devices) or insufficient preamble message handshake RA procedure in LTE systems, authors
transmission power. in [90] studied the stability limit of this legacy RA procedure,
2) Failure of Message 2 reception: This occurs mainly due which indicates the probability of failure of the RA proce-
to the lack of downlink radio resources (i.e., PDCCH) dure and the maximum achievable throughput. It has been
to send the RA response (Message 2) to all the received shown that the performance of the RA procedure deteriorates
preambles within the devices’ RAR windows. rapidly while sharing the Physical Downlink Control Channel
IEEE COMMUNICATIONS SURVEYS & TUTORIALS (DRAFT) 15

(PDCCH) resources between Messages 2 and 4 with different RACH structure adapted for low-cost MTC devices, the RA
priorities and the overall RA performance can be enhanced by requests from the devices are distributed across multiple NB
increasing the size of the PDCCH resource. channels, thus reducing the congestion caused due to wideband
Furthermore, in mMTC systems, the system performance nature of PDCCH in the conventional RA structure. Despite
may severely degrade in the presence of concurrent massive this enhanced RACH structure designed for low-cost MTC
access requests due to high probability of collision caused devices, the capacity of this RACH structure is not sufficient
by the signaling and traffic load spikes since the contention- to handle the massive number of RA requests coming from
based operation of the RACH in LTE-A networks is based on the ever-increasing number of devices.
ALOHA-type access [91]. One of the possibilities to reduce Towards addressing the problem of RACH overload in the
the load of physical RACH is to increase the number of access cellular IoT systems, several methods have been proposed
opportunities scheduled in a frame, however, this approach will in the literature [87]. From the perspective that whether the
reduce the amount of resources needed for data transmission. device or the eNodeB employs the solution, the existing
In this regard, it is crucial to balance the tradeoff between schemes can be broadly categorized into push-based and pull-
the amount of resources available for data transmission and based. In the first approach, the RA requests are controlled
the amount of access opportunities to be scheduled per frame from the device-side while in the pull-based approach, the
while designing an uplink scheduler for MTC applications by contention in the RA procedure is controlled from the eNodeB.
taking into account of limited available bandwidth. Besides, Besides, there are some strict separation schemes and soft
the main performance metrics to be improved include access separation schemes to concurrently support both the HTC and
success probability, preamble collision rate, access delay and MTC traffic in LTE-A networks [33, 91]. The strict separa-
device energy consumption [33]. tion schemes mainly comprise of the following: (i) resource
The LTE RA procedure employed in legacy LTE systems separation: orthogonal allocation of resources between HTC
is not efficient to support MTC devices mainly due to the and MTC traffic and dynamic shifting of resources among
following main reasons [66]. two classes, (ii) slotted access methods which define access
1) Because of the limited number of available preambles cycles including the RA slots dedicated to the MTC device
for the contention-based RA procedure, the massive access, and (iii) pull-based scheme in which the MTC devices
number of concurrent transmissions of the same pream- are allowed to access the PRACH only upon being paged by
bles would cause the overload of the RA procedure both the corresponding eNodeB. On the other hand, soft-separation
in the uplink and the downlink and this will result in high schemes include the following: (i) backoff tuning which as-
collision probability, access failure rate and the access signs longer back-off intervals to the MTC devices which
delay. do not succeed during the preamble transmission of the RA
2) To support a huge number of access requests, additional procedure and (ii) Access Class Barring (ACB) scheme. A
downlink resources need to be allocated since each RAR brief description of various existing and emerging solutions
message for one MTC device consists of 56 bits. for the RAN congestion problem is provided in Section V.
3) Even after an MTC device becomes successful in the RA Most of the existing MTC related works focus on the BS
procedure, the signalling overhead degrades the overall load balancing, radio resource management and the grouping
system efficiency since the size of the upload data of MTC devices, and only a few studies have been conducted
payload from the MTC device is significantly smaller in optimizing the access control of massive requests from the
than the traditional cellular terminals. MTC devices [65]. The incoming requests from the MTC
devices can be categorized into delay-sensitive and delay-
tolerant based on the delay tolerance level of the underlying
C. Adaptation of RA Procedure for MTC devices applications and the aggregator/BS can be equipped with two
It should be noted that the RACH in the RA procedure is queues with one having higher priority over the other in order
related to two different channels, namely, PRACH and PDCCH to deal with the two traffic classes. The criteria used for
as illustrated in [87]. A single PRACH consists of six physical defining delay tolerant and delay sensitive may differ from
RBs and has a bandwidth of 1.08 MHz. Over the whole one scenario to another [65].
system bandwidth, a maximum of 6 RACHs can be deployed The RA delay is one of the important aspects to be consid-
for time-division multiplexing with one RACH for frequency- ered while designing RA techniques for mMTC in the existing
division multiplexing. However, while sending RAR in the cellular networks. In this regard, authors in [94] derived lower
downlink channel, only a single PDCCH is responsible for bounds for the LTE-A RA delay by considering uniformly
handling multiple PRACHs. Although this is not a problem in distributed and Beta-distributed traffic arrivals and analyzed
the conventional HTC devices, this becomes a serious problem the effect of frequency of RA opportunities and the number
in MTC devices due to their hardware limitations in terms of of preambles. It has been shown that the RA delay can be
their capacity to listen to the wideband PDCCH. In general, reduced by several orders of magnitude by effectively tuning
a low-cost MTC device consists of a single RF interface these system parameters.
operating with 1.4 MHz bandwidth. To address this issue, the As briefly highlighted in Section I, cellular IoT standards
3GPP has proposed Enhanced-PDCCH (EPDCCH) with the mainly comprise of two categories, namely, LTE-M and NB-
narrow bandwidth of 1.4 MHz for low-cost MTC devices and IoT, which are described in the following subsections. Also,
each PRACH has a dedicated NB EPDCCH. In this modified in Table IV, we highlight the key differences between these
IEEE COMMUNICATIONS SURVEYS & TUTORIALS (DRAFT) 16

TABLE IV
C OMPARISON OF TWO MAIN CELLULAR I OT (LTE-M AND NB-I OT) TECHNOLOGIES

Feature/parameter LTE-M NB-IoT

Channel bandwidth 1.4 MHz 180 KHz
Transmission mode HD-FDD/FDD/TDD HD-FDD
Peak data rate 375 kbps (HD-FDD), 1 Mbps (TDD) 50 kbps (HD-FDD)
Latency 50-100 ms 1.5-10 seconds
Noise figure 9 dB (uplink), 5 dB (downlink) 5 dB (uplink), 3 dB (downlink)
Maximum coupling loss (MCL) 155.7 dB 164 dB
Modes of operation In-band Inband, guard-band and standalone
Power consumption Best at medium data rates Best at very low data rates
Mobility support Full mobility No connected mobility
Voice over LTE support Yes No

two technologies optimized to provide cellular connectivity to 3) Release-15: The main features of this release are re-
IoT devices [95, 96]. For the detailed differences among LTE- duced latency and power consumption, lower UE power
M, NB-IoT and legacy LTE in terms of supported features class, improved spectral efficiency, improved load con-
and functionalities for different uplink and downlink physical trol of idle UEs, eDRX enhancements and support for
channels, interested readers may refer to [31]. higher UE velocity.
The four-stage message handshake procedure followed in
D. LTE-M: Key Features and Channel Access Mechanisms the current LTE standard results in very high overhead for
While looking at the history of MTC, the first generation most of the IoT devices since the packets transmitted by the
of a full featured MTC device emerged in 3GPP Release resource-constrained IoT devices are quite short as compared
R12. In this release R12, the 3GPP has defined the category to the conventional cellular packets [33]. In this regard, several
0, i.e., CAT-0 for the low-cost MTC operation [30]. In the approaches are being investigated to design efficient channel
subsequent releases, the efforts to incorporate mMTC devices access mechanisms to support MTC in the existing cellular
continued and LTE release 13 (R13) in 2016 introduced two systems. One approach investigated in the literature is to
special categories, namely, CAT-M (also called LTE-M) for follow ALOHA-like immediate access without any reservation
MTC and CAT-N for the NB-IoT to support various features [97]. Although this scheme completely eliminates the channel
of MTC/IoT applications. LTE Rel-14 enhancements were reservation phase and provides very low latency, the system
completed in June 2017, and the improvements under the Rel- throughput is limited by the slotted ALOHA capacity of 1/e.
15 are ongoing and are expected to be released by June 2018. Besides, another approach is to utilize a preamble-initiated
With respect to Cat-1 category which was the lowest UE contention-based mechanism in which the nodes transmit
category in LTE Release 11 from the perspective of transmis- a randomly selected preamble to reserve a time/frequency
sion capability (peak rate of 10 Mbps in the downlink and 5 resource [7]. In contrast to the conventional RACH procedure
Mbps in the uplink), Cat-0 devices have a reduced complexity followed in LTE, this method eliminates Message 3 and
of about 50 % and have a reduced transmission rate of 1 Mbps Message 4 of the four-stage message handshake procedure
for both the downlink and the uplink [31]. Also, Cat-0 category and the data is transmitted on the RB specified in the RAR
enables the use of only one receiver antenna with a maximum message, thus significantly lowering the delay. However, if two
receiver bandwidth of 20 MHz and supports FDD half-duplex or more nodes choose the same set of preambles for the RA
operation with relaxed switching time, eliminating the need request, the collisions occur which are detected by the lack of
of dual receiver chains and duplex filters for low cost MTC Acknowledgement (ACK) message.
devices, respectively. In the subsequent LTE releases after the From the performance analysis carried out in [7], it is shown
introduction of LTE-M in release 13, several new features have that the preamble-initiated access achieves 86 % more capacity
been added. The key features of LTE-M in different releases in comparison to both the conventional LTE access mechanism
are included in the following [58]. and ALOHA-like immediate transmission scheme for small
1) Release-13: The main features included in this re- data packet transmissions in IoT application scenarios. How-
lease include Coverage Enhancement (CE) mode A/B, ever, in terms of delay, the ALOHA-like scheme reduces the
bandwidth limited operations (1.4 MHz), half-duplex delay by about 62 % for low traffic loads in comparison to
support, in-band operation mode, RRC connection, data the preamble initiated access and by about 77 % as compared
transmission via a control plane, mobility support and to the conventional LTE access mechanism.
eDRX. To capture the signature of the LTE signal, an MTC device
2) Release-14: The key features incorporated in this ver- will need to receive the synchronization signals which occupy
sion include multi-cast support with single-cell point-to- 6 RBs of the eNodeB’s bandwidth [31]. Although decoding
multipoint, positioning enhancements such as enhanced- PDCCH becomes impossible due to the bandwidth limita-
cell ID requirements and observed time difference of ar- tion (only 1.4 MHz) of MTC devices, the enhanced version
rival support, larger channel PDSCH/PUSCH bandwidth EPDCCH, which uses only one RB, is a good candidate,
(up to 5 & 20 MHz), Voice over LTE enhancements, and but is not sufficient for the required coverage enhancement
support for HARQ-ACK bundling and inter-frequency [98]. However, increasing its bandwidth to 6 RBS in the 1.4
measurements. MHz bandwidth on the MTC device along with the repetition
IEEE COMMUNICATIONS SURVEYS & TUTORIALS (DRAFT) 17

will provide the good coverage of about −14 dB and the better indoor coverage and support to a massive number of
EPDCCH also supports beamforming to enhance the coverage low-throughput devices, with low power consumption and re-
[28]. Therefore, 6 RBs, i.e., one narrowband is usually used laxed delay requirements [63]. To accomplish these objectives,
as the basic unit for the MTC bandwidth [31]. the NB-IoT follows the procedures of optimizing control plane
The main changes incorporated in the physical layer oper- and user plane of Cellular-IoT (CIoT) evolved packet system
ation of LTE-M as compared to the legacy LTE are briefly towards reducing the signalling overhead for small data packet
described below [31]. transmissions [99].
1) Frequency hopping: Due to narrow-bandwidth and a For both the uplink and downlink operations, NB-IoT can
single receiver chain at the MTC device, the benefits due operate with an effective narrowband operation of 180 kHz
to spatial diversity and frequency diversity are not avail- bandwidth corresponding to one RB in the LTE network. In
able. To compensate for the performance loss caused the downlink of an NB-IoT system, Orthogonal Frequency-
due to frequency diversity, the concept of frequency Division Multiple Access (OFDMA) is employed with the
hopping, which allows MTC transmissions to hop from subcarrier spacing of 15 kHz over 12 sub-carriers while in the
one NB channel to another, is employed. The challenges uplink, both single tone and multi-tones are supported (single-
associated with frequency hopping in MTC devices tone with the subcarrier spacing of either 3.75 kHz or 15 kHz)
include the need of retuning the RF chain, and the prior [100]. The NB-IoT usually can be operated in the following
knowledge of the hopping pattern at the eNodeB and the three operation modes [64, 101].
device. 1) In-band operation: This mode of operation utilizes the
2) Repetitions: To achieve sufficient link budget in the RBs within an LTE carrier by reserving one RB for the
downlink for the coverage enhancement, repeated copies NB-IoT system.
of the same signal are transmitted over time to boost 2) Guard band operation: This mode uses the unused
the link performance via time diversity. The main issue resources within the guard band of the LTE carriers
involved with this repeated transmission strategy is the while ensuring that this does not affect the normal
requirement of increased decoding time, i.e., latency, capacity of the LTE carrier.
demanding for longer wake-up time for the MTC device. 3) Stand-alone operation: This mode utilizes the re-
3) MTC Physical RACH (MPRACH) and Physical farmed GSM low band already existing in several coun-
Downlink Control Channel (MPDCCH): To com- tries (700 MHz, 800 MHz, and 900 MHz) [101].
pensate for the additional path-loss caused due to the
The NB-IoT technology provides greater flexibility for the
extended coverage for the MTC device, the PRACH of
deployment of IoT devices in different applications such as
legacy LTE needs to be modified. For this, frequency
smart city, smart home, smart metering and smart agriculture
diversity and repetitions need to applied to the MPRACH
by reusing the existing network architectures. The main re-
to achieve the required diversity. Furthermore, additional
quirements for the NB-IoT system include the following [101].
features such as defining downlink control formats and
enhancing control channel assignment procedure need to 1) Low power consumption: NB-IoT systems utilize the
be added to support frequency hopping and repetitions power saving mode and eDRX to maximize the battery
in MPDCCH. life.
4) MTC search spaces: In order to reduce the number 2) Low channel bandwidth: Due to low channel band-
of decoding trials by the devices in the LTE systems, width of 200 kHz (180 kHz plus guard bands),
each MTC device can be assigned only a defined search GSM channel re-farming is applicable for NB-IoT sys-
space area of the whole control region to be monitored. tems since a single NB-IoT channel can utilize one
In contrast to the legacy Enhanced PDCCH, there are GSM/GPRS channel.
mainly two classes of search spaces in MPDCCH, 3) Low cost for the end-device: Due to low channel
namely, device-specific search space and common search bandwidth of 200 kHz, the front-end and digitizer of
space. NB-IoT receivers are much simpler than that of the
5) MTC Downlink Control Information (DCI) formats: existing LTE-based systems operating on the bandwidth
To reduce blind decoding iterations, i.e., device com- of 1.4 MHz, thus leading to low-complexity (cheaper)
plexity as well as to facilitate the use of frequency devices.
hopping, repetition and enhanced coverage, three dif- 4) Low deployment cost: Besides the device cost, due to
ferent DCI formats have been defined for uplink grant, the capability of reusing existing GSM bands, the de-
downlink scheduling and paging in MTC devices. ployment cost for the network operators is significantly
reduced.
5) Extended coverage: NB-IoT can provide about ten
E. Narrowband IoT: Key Features and Channel Access Mech- times better coverage area compared to the legacy GPRS
anisms systems as it can be achieve the additional 20 dB link
To address various challenges of supporting MTC devices in budget gain.
cellular IoT networks specified in Section II-B, the 3GPP has 6) Support for massive number of connections: Due to
proposed the concept of NB-IoT in its Release 13 [64]. The improved coverage and low channel bandwidth, it can
main objectives behind the NB-IoT concept include providing support significantly higher number of MTC devices.
IEEE COMMUNICATIONS SURVEYS & TUTORIALS (DRAFT) 18

The main signals and channels involved in the downlink the performance of packet-switching based wireless networks.
of an NB-IoT system are Narrowband Primary Synchro- For analyzing the queuing behavior, the characterization of
nization Signal (NPSS), Narrowband Secondary Synchroniza- source traffic as well as the services is an important aspect to
tion Signal (NSSS), Narrowband Physical Broadcast Channel be considered [103].
(NPBCH), Narrowband Reference Signal (NRS), Narrowband Due to distinct QoS requirements of MTC devices, it is cru-
Physical Downlink Control Channel (NPDCCH) and Narrow- cial to provide QoS support for MTC devices in future wireless
band Physical Downlink Shared Channel (NPDSCH) [102]. networks. For example, non-real time MTC applications such
Out of these, NPSS and NSSS are used by an NB-IoT device as data transmissions aim to enhance the reliability of trans-
to carry out cell search procedure including cell identity de- mission and do not have strict delay constraint. Whereas, real-
tection, and frequency and time synchronization. The NPBCH time MTC applications such as video surveillance/demand,
includes the master information block while the NRS is used the important QoS metrics are strict latency and data rate
to provide phase reference required for the demodulation of requirements rather than high spectral efficiency [104]. To
downlink signals. Similarly, NPDCCH includes the scheduling meet the QoS requirements of different network applications
information for both the uplink and downlink data channels highlighted in Section II-D, it is crucial to design efficient
while the NPDSCH carries various information such as system radio resource allocation algorithms for the MTC devices in
information, paging message, RAR message and also data the uplink while considering the constraints on the available
from the higher layers. radio spectrum.
Besides, the uplink transmission scheduling of devices in the One of the potential candidate platforms to support MTC
NB-IoT mainly comprises of Narrowband PRACH (NPRACH) devices is the LTE-A standard and the 3GPP has been working
and Narrowband PUSCH (NPUSCH) [100]. Out of these, on several enhancements of LTE-A standard towards this
NPRACH corresponds to the time-frequency resource used to direction. The 3GPP uses Single Carrier Frequency Division
transmit RA preambles and the NPUSCH is used for carrying Multiple Access (SC-FDMA) [105] as the multiple access
the uplink data. The differences of the above-mentioned uplink scheme in the uplink of LTE cellular networks due to its main
and downlink channels from the legacy LTE systems are high- advantage of low Peak-to-Average Power Ratio (PAPR) as
lighted in [102]. The key technique employed by an NB-IoT compared to that of the OFDMA. Due to this feature, the
system to obtain enhanced coverage with low complexity is reduced requirements on the processing power and battery
repetition, which utilizes the repeated transmission of both data are suitable for the resource-constrained MTC devices [104].
transmission and the involved control signalling transmission However, the allocation of RBs in the SC-FDMA becomes
[100]. complex as compared to that in the OFDMA scheme due to
The RA procedure in the NB-IoT system is responsible the sequential transmission of the RBs in the SC-FDMA in
for establishing a radio link during the initial access, for contrast to the transmission of orthogonal RBs in the OFDMA-
scheduling the transmission requests and to achieve uplink based systems.
synchronization among the NB-IoT devices [102]. Three The minimum resource unit used for scheduling downlink
different types of NPRACH resource can be configured by and uplink transmissions in the LTE-A based cellular systems
assigning separate repetition values for a basic RA preamble to is referred to as an RB. Each RB comprises of 12 sub-carriers
serve the devices belonging to different coverage classes with with each sub-carrier having the bandwidth of 180 kHz in
different ranges of path loss. The device estimates its coverage the frequency domain and one sub-frame of 1 ms duration in
level by measuring the downlink received signal power, and the time domain [22]. The RB can be considered as a time-
then the device transmits an RA preamble in the NPRACH frequency resource in which an UE performs RA and each
resources configured for the estimated coverage level. The RA slot comprises of the bandwidth equivalent to the 6 RBs,
configuration of NPRACH resources is made flexible in a time- i.e., 1.08 MHz and its duration in the time domain is 1 ms.
frequency resource grid to enable the deployment of NB-IoT In LTE-based cellular networks, the eNodeB broadcasts the
systems in different scenarios. periodicity of the RA slots by means of a variable referred
to as the Physical RACH (PRACH) configuration Index [33],
IV. T RANSMISSION S CHEDULING FOR MTC S YSTEMS and subsequently, the MTC devices and the legacy cellular
WITH Q O S S UPPORT users can perform RA by using the PRACH channel. Even
Most of the models used to analyze the capacity of wireless though the data size from the MTC devices is significantly
systems are based on physical layer models and they can not small, the massive number of devices attempt to concurrently
capture the link-layer QoS requirements such as bounds on communicate over the same radio channel, thus leading to the
the delay [103]. Therefore, physical-layer only models are not network overload problem [22]. In contrast to the conventional
suitable for QoS support mechanisms such as resource reser- HTC services such as multimedia for which the packet arrival
vation and admission control. Furthermore, in contrast to the periods range from 10 ms to 40 ms, the packet arrival periods
wired links, it is challenging to guarantee QoS requirements in MTC applications may range from 10 ms to several minutes
in wireless systems due to low reliability, multi-path fading, [54].
co-channel interference and time-varying capacities. In order
to incorporate complex QoS requirements into account, it is A. Framework for Performance Analysis with QoS Support
important to understand the queuing behavior of the connec- In this subsection, we present a mathematical framework
tions and to capture the QoS requirements while characterizing to carry out the performance analysis of mMTC systems
IEEE COMMUNICATIONS SURVEYS & TUTORIALS (DRAFT) 19

with QoS support in terms of the effective capacity, effective requirement of θm for the mth MTC device, the following
SNR and the estimated number of MTC devices. For this condition should be satisfied [104]
analysis, we consider the uplink in a single-cell of 3GPP LTE-
Rem (θm ) ≥ λm , (6)
A networks serving multiple MTC devices with the SC-FDMA
scheme. This scenario can also be studied in conjunction with where λm is the traffic arrival rate for the mth MTC device.
the legacy cellular/HTC users as in [106], however, herein, By solving (6), one can obtain θm . Subsequently, by using
we deal only the case of MTC devices since we are interested the Shannon’s capacity formula, the maximum achievable
in providing QoS support for MTC devices while maximizing transmission rate for the mth MTC device can be expressed
some network performance metric subject to the constraints as
on the available radio spectrum. In practice, these devices 
Pm |hm |2

can be grouped based on the employed transmission protocols Rm = Blog2 (1 + γm ) = Blog2 1 + , (7)
σn2
and QoS requirements, and can be deployed on the cluster
basis by using different wireless technologies such as WiFi, where B is the bandwidth of each RB, Pm is transmission
Bluetooth and Zigbee [104]. Furthermore, MTC devices can power of the mth MTC device, |hm |2 is the channel gain, σn2
communicate to the eNodeB via an MTC gateway and the is the Additive White Gaussian Noise (AWGN) power, and
2
total available RBs can be divided between the access link γm = Pmσ|h2m | is the SNR for the mth MTC device.
n
(MTC devices to the MTC gateway) and the backhaul link Although the value of θm can be derived by finding the
(from the MTC gateway to the eNodeB) in the time domain probability density function of λm corresponding to (7) and
as considered in [106]. by subsequently solving (6), the evaluation process is quite
Let us assume that there are M number of total MTC complex. Another approach is to use an intuition from [103]
devices in the coverage area of the eNodeB, indexed by the and to obtain the QoS exponent θ(λ) in the following way
set M = {1, . . . , m, . . . , M } and there are L number of φ(λ)λ
available RBs, indexed by the set L = {1, . . . , l, . . . , L}. We θ(λ) = , (8)
λτs (λ) + E[Q(t)]
assume Poisson distribution for the traffic arrival rate of the
MTC devices and block fading wireless channel between MTC where Q(t) is the length of a queue at time t and τs denotes the
devices and the eNodeB/gateway as in [104]. Also, we assume the average remaining service time of a packet being served.
that channel coherence time is greater than the Transmission The detailed methodology to estimate the value of θ(λ) in (8)
Time Interval (TTI) and the channel gain remains constant for the considered MTC scenario has been illustrated in [104].
during a TTI. Despite the significant benefits of SC-FDMA in terms of
To incorporate QoS requirements of MTC devices into the power and battery requirements, there arise some restrictions
problem formulation, one way is to define a QoS exponent for for uplink resource allocation (RB and power allocations)
each MTC device and to introduce this exponent in the defi- while employing SC-FDMA in the uplink [105]. The main
nition of system capacity. Let θm denote the QoS exponent of aspects to be considered include: (i) a single RB can only be
the mth MTC device indicating a steady-state delay violation allocated to at most one user, (ii) multiple RBs allocated to a
probability of the mth M2M device. Considering a queue of single user should be adjacent, and (iii) the transmit power on
infinite buffer size required due to a constant arrival rate λ, all the RBs allocated to a user should be equal. Let us assume
the delay violation probability is given by [104] that the set of RBs Lm is allocated to the mth MTC device in
the current TTI, then the achievable rate (upper bound) from
δ = Pr(dm > dmax ) ≈ φm (λ)e−θm dmax , (4) (7) in terms of effective SNR can be written as
where Pr(. ) denotes the probability operation, dm represents Rm = B.Lm log2 (1 + γeff,m ), (9)
the delay experienced by a source packet of the mth MTC
device, dmax is a delay bound, and φm (λ) = Pr(dm > 0) where Lm = |Lm | denotes the cardinality of the set Lm and
indicates the probability of non-empty buffer. In this formula- γeff,m denotes the effective SNR for the mth MTC device.
tion, the pair of (φm (λ), θm (λ)) can be used to characterize Since each data symbol is spread over the whole bandwidth in
the link from the mth device to the gateway/eNodeB. SC-FDMA transmission, the effective SNR can be computed
In contrast to the conventional physical layer-based capacity, as an average of SNRs over the allocated set of RBs to a
we define the effective capacity to take the link-layer QoS particular MTC device as follows
requirements into account. The effective capacity [103] is 1 X
γeff,m = γm,l , (10)
defined as the maximum constant arrival rate that a given Lm
l∈Lm
service process can support to guarantee a QoS requirement
specified by θ and can be defined for the mth MTC device as where γm,l is the SNR of the mth device for the lth RB.
Another aspect to be considered is how to effectively design
1
Rem (θm ) = − lnE[e−θm Rm ], (5) the medium access scheme to support the massive number
θm of devices. One approach is to determine the optimal size of
where Rem denotes the effective capacity for the mth MTC the Random Access Window (RAW) based on the estimated
device, θm represents the statistical QoS exponent of the mth number of MTC devices in the following way [107]. If there
MTC device, E(. ) denotes the expectation, and Rm is the are idle slots available at the RAW, the eNodeB/access point
data rate of the mth MTC device. In order to guarantee a QoS can estimate the number of devices for the uplink access
IEEE COMMUNICATIONS SURVEYS & TUTORIALS (DRAFT) 20

by using suitable estimation techniques such as maximum that the metadata is negligible as compared to the size of
likelihood estimation method. Let I be the measured number the information payload. However, this assumption does not
of idle slots in the uplink RAW, LUL be the number of slots apply to the transmission of short packets since the metadata
of the uplink RAW and NUL be the number of devices for size becomes no longer negligible, resulting in the need of
the uplink access. When NUL devices contend in LUL , the highly efficient encoding schemes. Secondly, for the case of
probability of selecting a slot by a device for the uplink access long packets, there exist channel codes which enable the
1
becomes LUL and the corresponding complementary probabil- reconstruction of information payload with high probability.
1 The thermal noise and channel distortions average out for the
ity is (1 − LUL ). Thus, the idle probability that no devices for
the uplink access transmit the power save poll message, by case of long packets due to the law of large numbers, however,
which the device requests for the downlink data or the ACK this averaging does not occur for the case of short packets
frame from the eNodeB, is pidle = (1 − 1/LUL )NUL and the and the classical law of large numbers is not applicable for
probability pidle is estimated as: p̂idle = LIUL . Subsequently, mMTC applications, resulting in the need of new information
the estimated number of devices for the uplink access by theoretic principles. In this regard, authors in [11] discussed
utilizing the aforementioned idle probability can be calculated various information theoretic approaches to characterize the
as transmission of short packets in wireless communication sys-
log(p̂idle ) tems and applied these principles on the transmission of short
NUL = 1 . (11)
log(1 − LUL ) packets in various channels such as a two-way channel, a
On the other hand, the existing packet schedulers are mainly downlink broadcast channel and the uplink RACH. In addition,
designed for a specific wireless system such as LTE and do not authors in [111] investigated the tradeoffs among reliability,
fully capture the heterogeneous characteristics of ultra-dense throughput, and latency for the transmission of information
IoT networks. In this regard, authors in [84] proposed delay- over multiple-antenna Rayleigh block-fading channels.
efficient joint packet scheduling and subcarrier assignment In the above context, several recent works have investigated
by considering the classification of the uplink MTC traffic different physical layer approaches to support small packet
aggregated at the MTC aggregator into multiple classes based transmissions in mMTC/IoT environment by considering their
on traffic features such as packet size, arrival rate and delay re- specific characteristics, which are briefly reviewed in the fol-
quirements. By employing an MTC specific traffic model, the lowing paragraphs. Also, in Table V, we list the recent research
incoming data from the sensors at the aggregator is categorized works towards supporting small data packet transmission in
either as ED or PU types and the delay requirements of these wireless networks with their main themes and applicable
PU and ED traffic types are mapped onto sigmoidal and step systems.
utility functions, respectively. In addition, in order to ensure In short data packet transmissions, one effective way of
that the packets transmitted by an MTC device are within enhancing the packet transmission efficiency is to optimize the
the delay budget, authors in [108] introduced a new MAC pilot overhead [112]. However, most of the existing pilot over-
element, called Packet Age, with which the device informs head optimization works considering the objective of ergodic
the scheduler about the waiting time of the oldest packet in channel capacity maximization are based on the assumption of
the device buffer along with the buffer size specified in the sufficiently large packet length resulting in small packet error
buffer status report. probability, which is not suitable for short-packet transmission.
In this regard, authors in [112] formulated the optimization of
approximate achievable rate as a function of block length, pilot
B. Short Data Packet Transmission and Associated Issues length and error probability, and illustrated the importance of
In this subsection, we briefly discuss various issues related considering packet size and error probability while optimizing
to short data packet transmission in IoT systems along with pilot overhead via numerical results.
its information theoretic perspective. One of the emerging Another potential enabling approach to support short packet
areas in the MTC systems is ultra-reliable and low-latency transmissions in IoT/mMTC environments is to design suitable
communications, also known as mission-critical MTC. Some transmit waveforms. In the IoT environment, there are some
of the applications of mission-critical MTC are industrial applications with very small packet sizes such as the data
control, intelligent transportation systems and smart grids for transmitted from sensors like temperature sensors while some
power distribution automation [109]. As an example, industrial other applications such as car to car and car to infrastructure
control applications may need to transmit about 100 bits within communications demand very fast response time. In order to
100 µseconds with 10−9 PER [110]. support these diverse set of applications, the 5G and beyond
Existing wireless systems are designed to support the con- air interface should be able to support transmissions with very
ventional HTC traffic having long packet sizes and each packet small air interface latency enabled by very short transmission
consists of information payload and the control information frames [113]. In this regard, it is important to investigate
(metadata) which usually contains various information about suitable waveforms for supporting diverse applications in an
logical addresses, packet initiation and termination, synchro- IoT environment. Among potential multi-carrier waveform
nization and security. As compared to the transmission of long contenders such as filtered Cyclic Prefix-OFDM, Filter bank
packets, the transmission of short packets in the wireless IoT multi-carrier and Universal Filtered Multi-Carrier (UFMC),
systems differs mainly in the following two ways [11]. First, authors in [113] concluded UFMC as the best choice for
existing transmission techniques are based on the assumption IoT systems with short burst transmissions due to its several
IEEE COMMUNICATIONS SURVEYS & TUTORIALS (DRAFT) 21

TABLE V
R ECENT RESEARCH WORKS TOWARDS SUPPORTING SMALL DATA PACKET TRANSMISSIONS IN WIRELESS NETWORKS

Main Theme Applicable systems References

Optimization of pilot overhead IoT Sensor networks [112]
Design of air interface and waveforms Multicarrier 5G systems [113]
Coding and modulation schemes IoT Sensor networks [114]
Non-orthogonal multiple access MTC scenarios [115]
Minimization of the core network signalling 5G cellular network [116]
Joint encoding of grouped messages Wireless broadcast channel [117]
Autonomous transmission mode Delay tolerant IoT/MTC scenarios [118]
Receiver algorithms to enhance the reception quality 5G wireless networks [119]
Energy and information outage performance analysis Wireless powered network [120]
Exploitation of frequency diversity to enhance reliability Tactile Internet [121]

benefits in terms of supporting fast Time Division Duplex affecting the data transmission duration, i.e., to balance the
(TDD) switching, low latency modes, low energy consumption trade-off between the pilot training period and the data trans-
and small packet transmission. mission period. In this regard, authors in [119] investigated
In addition, investigating suitable coding and modulation an efficient receiver structure which can exploit information
schemes is crucial to achieve high energy efficiency for short- received during the data transmission period to enhance the
packet transmissions having low-duty cycles. Due to lower reception quality for the short packet transmissions. Moreover,
duty cycle, time synchronization and phase coherency for in the context of energy harvesting networks, authors in [120]
short-packet transmissions become non-trivial. Furthermore, provided a comprehensive analysis of the backscatter wireless
because of short packet length, a large coding can not be powered communication with sporadic short data packets by
achieved as in the conventional voice or data networks. More- using a stochastic geometry framework.
over, the overhead required to maintain time synchronization
and phase coherency becomes significantly large while using V. S OLUTIONS FOR RAN C ONGESTION P ROBLEM IN
the conventional coherent modulation schemes [114]. In this C ELLULAR I OT NETWORKS
regard, the time synchronization overhead can be reduced In this section, we first review several existing techniques
by employing either non-coherent modulation/demodulation towards addressing RAN congestion problem in cellular IoT
schemes such as Phase-Shift Keying (PSK) with differential networks, and then discuss some emerging solutions.
encoding or orthogonal modulations. To this end, authors in
[114] analyzed the tradeoff between energy efficiency and A. Existing Techniques
bandwidth in non-coherent short packet transmission systems.
Towards addressing the RAN congestion problem in LTE-
Towards addressing the problem of scalability and efficient based cellular networks, 3GPP has specified the following six
connectivity to the massive number of MTC devices with short different solutions of LTE RA congestion [69]:: (i) ACB, (ii)
packets, the NOMA scheme is considered as one candidate MTC-Specific backoff, (iii) dynamic resource allocation, (iv)
multiple access solution [115]. Due to its benefit of improving Slotted random access, (v) separate RA resources and (vi) pull-
fairness and spectral efficiency for low-latency transmission based RA. In the following, we briefly describe the principles
with respect to the orthogonal multiple access technique, it of these techniques along with other related solutions in the
is considered promising for IoT applications. In this regard, literature [35, 69, 122]. Also, in Table VI, we provide the list
authors in [115] analyzed a trade-off among the transmission of these schemes along with their main principles and the
rate, transmission delay (in terms of block-length) and de- corresponding references.
coding error probability by considering a two user downlink 1) Back-off based scheme: In this scheme, the devices
NoMA system with finite block-length constraints. retransmit after a backoff time if they encounter a
Furthermore, towards minimizing the signalling overhead collision. This scheme can enhance the network per-
for small data packet transmissions, authors in [116] proposed formance under a low congestion level, however, be-
a framework based on 5G RAN controlled user-centric mobil- comes problematic in high-level congestions [123] This
ity, in which an anchor node is allocated and updated for each is the conventional approach followed in contention-
end-device and it maintains the connection of the device to the based wireless networks and 3GPP has suggested several
core network within its coverage area. In this approach, an user improvements to solve the RAN overload problem. To
centric area is dynamically allocated so that an user/device can support MTC devices in the existing cellular networks,
move freely and communicate with the network without any 3GPP has suggested the use of MTC-specific backoff
state transitions signaling required in the existing connection scheme in which MTC devices are subject to a larger
management schemes with RRC protocols [116]. backoff interval than the HTC devices [69].
Moreover, while implementing multiple-antenna based in- 2) Access Class Barring (ACB) scheme: This scheme
terference suppression in IoT systems with small data packet classifies the contending devices into multiple access
structures, the insufficient training period may result in severe classes with different access probabilities and each class
degradation in the estimation of the desired channel and is assigned to an ACB parameter and an access barring
interference covariance matrix. The main challenge here is timer [69]. The working principle of the ACB scheme
to obtain the reliable channel estimation without significantly can be summarized in the following way. First, the BS
IEEE COMMUNICATIONS SURVEYS & TUTORIALS (DRAFT) 22

TABLE VI
S UMMARY OF EXISTING SOLUTIONS FOR RAN CONGESTION PROBLEM IN CELLULAR I OT NETWORKS

RA Scheme Main principle References

Back-off based scheme In the occurrence of collisions, devices retransmit after an MTC-specific backoff period. [69, 123]
Access Class Barring (ACB) Multiple access classes of devices are assigned different access probabilities. [91]
Extended Access Barrier (EAB) A certain access class of devices is barred from the channel access. [124]
Cooperative ACB scheme ACB parameter is designed by many BSs in a collaborative way. [92]
Dynamic ACB ACB parameter is updated dynamically based on previous collisions. [93]
Prioritized RA with dynamic ACB Utilizes class-dependent back-offs and dynamic ACB. [125, 126]
Dynamic resource allocation Congestion level is predicted at the BS and additional RACH resources are allocated dynamically. [35]
Slotted random access Each device is assigned a dedicated RA slot and is allowed to perform RA only in that slot. [69]
Separation of RA Resources Available preambles or RA slots are divided between MTC and HTC devices. [35, 69]
Pull-based/Paging-based Devices perform RA attempts only after receiving paging messages from the BS. [69]
Group-based RA RACH resources are allocated on the basis of groups formed based on some defined criterion. [127, 128]
Code-Expanded RA RA codewords are generated and each device utilizes a set of preambles in each RA slot. [129]

broadcasts the ACB parameter, i.e., 0 ≤ p ≤ 1 to the load among the BSs in a heterogeneous multi-
MTC devices and each MTC device trying to connect tier cellular network with the objective of reducing
to the BS generates a random number 0 ≤ r ≤ 1 the congestion level and also improving the access
uniformly. Then, the MTC device is allowed to start the delay.
RA procedure if r < p and otherwise, the access to that c) Dynamic ACB scheme [93]: In this approach, the
particular device is barred and the device has to wait for ACB parameters are updated dynamically based on
a random backoff time determined based on the barring the information about the number of collisions in
duration of that class. Therefore, by controlling the ACB the previous time slots.
parameter p, the BS can control the stabilization of RA d) Prioritized RA with dynamic ACB [125, 126]:
to optimize some network performance metrics such as This scheme pre-allocates the RACH resources
throughput [91]. for different classes of MTC devices with class-
However, in the presence of severe congestion caused dependent backoff procedures and reduces the
by the presence of massive number of IoT devices, number of concurrent requests for the RACH by
the value of p may be set to be extremely low, thus employing the dynamic ACB method.
leading to the intolerable delay. Also, the ACB scheme
is not suitable for event-driven applications in which the 3) Dynamic Resource Allocation: In this scheme, the BS
contention may arise within a short time duration [33]. predicts the congestion level of the access network over-
Furthermore, the operating parameters such as transmis- load caused due to MTC devices and allocates additional
sion probability should be adjusted based on the network RACH resources dynamically in the time domain or
status and estimating the number of devices/network frequency domain or both for the MTC devices [35,
status becomes challenging due to highly bursty traffic 69]. However, the allocation of more radio resources for
in event-driven MTC communications [123]. RACH will reduce the radio resources available for the
To address the above drawbacks of the ACB scheme, traffic channels and this trade-off needs to considered
there have been some attempts in the literature. Some while implementing this solution.
of the important ones include the following. 4) Slotted Random Access: In this method, a dedicated
RA opportunity is provided to each MTC device and is
a) Extended Access Barrier (EAB) scheme [124]: In allowed to perform RA only in the access slot allocated
this scheme, devices belonging to a certain access to it [69]. However, in ultra-dense IoT scenarios, this
class are barred from the channel access to provide method will result in very high access delay since the
some form of service differentiation [124]. The duration for each RA cycle will be significantly large.
operation of this scheme depends mainly on the 5) Separation of RA Resources: In this approach, different
following two factors: (i) the sets of barred access RACHs are allocated to MTC devices and HTC devices
classes and (ii) the time of turning EAB on or to avoid the impact of RA congestion on HTC devices.
off. The larger the set of barred access classes, The separation of RA resources can be done either by
the higher will be the access success probability splitting the available preambles into MTC and HTC
which comes at the cost of increased mean access subsets or by allocating different RA slots for MTC and
delay. Furthermore, the timing of turning EAB on HTC devices [35, 69].
or off relies on the input network load which is 6) Pull-based/Paging-based scheme: All the schemes de-
proportional to the number of devices concurrently scribed above fall under the category of push-based
accessing the network. approach in which RA attempts are done randomly by
b) Cooperative ACB scheme [92]: In this scheme, the devices. However, in the pull-based method, the
ACB parameters are determined across the network devices perform RA attempts only after receiving paging
jointly by many BSs interconnected via the X2 messages from the BS. To reduce the number of paging
interface [92] rather than individually calculated at load in this approach, a number of MTC devices can
each BS. This scheme aims to balance the traffic be paged together by following a group paging method
IEEE COMMUNICATIONS SURVEYS & TUTORIALS (DRAFT) 23

[69]. each other without involving a central entity and after the
7) Group-based RA Scheme [127, 128]: The MTC devices learning convergence, each MTC device gets a unique RACH
can be grouped based on some criterion such as having slot. Furthermore, authors in [23] applied a Q-learning based
similar QoS/delay requirements and being deployed in a unsupervised algorithm in order to select an appropriate BS
specific geographical region, and RACH resources can for MTC devices on the basis of QoS parameters in dynamic
be allocated on the group-basis to reduce the access network traffic conditions. Moreover, a hierarchical stochastic
network congestion. In a group-based RA scheme pro- learning algorithm has been applied in [134] to enable each
posed in [127], the devices within one paging group are device to make the access decision with the assistance of
partitioned into different access groups based on some common control information broadcasted from the BS. In
criterion and only one device within each access group, addition, in [21], a Q-learning algorithm has been applied to
called group delegate/header, is made responsible for dynamically adjust the value of a barring factor to be allocated
communicating with the BS. The group delegate can be to the MTC device in the ACB scheme.
decided by the BS based on some suitable metrics such 2) Distributed Queueing: The existing approaches to en-
as transmission power and channel conditions. hance the RACH performance are mainly based on the
Another grouping approach is to divide the cell coverage ALOHA-type mechanisms which suffer from some level of
area into a different spatial groups and to enable the inefficiency, instability and uncertainty in the outcome of the
use of same preambles at the same RA slot by the access opportunities to be assigned to the devices [33]. In
MTC devices located in different groups if the minimum this regard, one promising approach is Distributed Queuing
distance of these MTC devices is larger than the multi- Collision Avoidance (DQCA) [135] which is a distributed and
path delay spread [87, 130]. While sending the RAR always-stable high performance protocol. This MAC protocol
message, the BS sends distinct RARs to all the detected behaves as an RA mechanism for low traffic load and switches
devices having different Timing Alignment (TA) values automatically and smoothly to a reservation scheme when
even if they use the same preamble during the RA the traffic volume increases [135, 136]. More specifically, the
preamble transmission phase. DCQA protocol utilizes two distributed queues which operate
8) Code-Expanded RA Scheme [129]: In this approach, in parallel [136]. The first queue, called collision resolution
the contention space is expanded to the code domain queue, deals with the resolution of access-request signal col-
by creating the RA codewords. While initiating an RA lisions, while the other queue, called data transmission queue,
attempt, each device sends a set of preambles over the helps to manage the data transmission. The main features of
given RA slots instead of transmitting only a single this protocol are the following [135].
preamble at any random RA slot, thus creating a set 1) It can eliminate back-off periods and avoid collisions in
of preambles in each RA slot. data packet transmissions.
9) Tree-based RA scheme [131, 132]: This category of 2) Its performance is independent of the number of trans-
RA schemes utilizes the tree-based algorithms such as mitting nodes.
q-ary tree splitting technique [133], which rely on the 3) It is stable independently of the traffic conditions.
utilization of feedback obtained after each contention 4) As compared to other centralized or distributed MAC, it
attempt [132]. This RA scheme is mostly used to ad- utilizes very few bits for signaling operation purposes.
dress the contention problem caused due to synchro- Furthermore, authors in [74] proposed a distributed queuing-
nized arrivals of the traffic from a large number of based access protocol for LTE with the objective of improving
MTC devices. Furthermore, the combination of collision the RA performance for MTC systems without altering the
avoidance techniques such as access barring can be used existing frame structure of LTE systems. The original version
in combination with tree-based collision resolution in of distributed queueing protocol envisions orthogonal mini-
order to form a hybrid RA scheme [131]. slots as access opportunities and its implementation requires
a change in the LTE frame structure since the preambles in
LTE are not orthogonal in time domain. To address this issue,
B. Emerging Solutions
the authors in [74] considered the distribution of allocated
In the following, we provide some of the emerging research preambles for MTC devices among Ng virtual groups, with
directions to address RAN congestion problem in wireless IoT each virtual group having Np number of preambles and each
networks. preamble being equivalent to one mini-slot considered in the
1) Learning-based Techniques: Recently, learning-based original distributed queueing protocol.
techniques have received important attention in addressing 3) SDN and Virtualization for RAN Management: To sup-
the RAN congestion problem in cellular IoT networks. In port differentiated MTC services with diverse QoS require-
this direction, an RL scheme has been applied in [22] for ments, a physical wireless network can be abstracted and sliced
the selection of an appropriate BS for the MTC devices into multiple virtual networks by employing suitable network
with the objective of avoiding access network congestion and function virtualization techniques [73]. On the other hand,
minimizing the packet delay. In addition, a Q-learning based Software Defined Networking (SDN) enables the separation
access scheme has been studied in [10] to support MTC of a data plane and a control plane, and provides the capa-
traffic in the existing cellular networks. In this Q-learning bility of programming a network via a centralized controller.
based approach, MTC devices learn to avoid collisions among Due to the global view of the underlying network, an SDN
IEEE COMMUNICATIONS SURVEYS & TUTORIALS (DRAFT) 24

controller enables the efficient management of radio resources general inversely related to the data rate. In order to predict the
in dynamic network traffic and channel conditions. In cellular reliability, existing systems form explicit input/output models
IoT networks, a hypervisor can divide the physical network of a wireless channel and then analyze the performance of
into different IoT networks based on device classes and func- physical layer for each set of the parameters.
tionalities, and the SDN controller can dynamically allocate However, due to the increasing trend of using multiple an-
the available radio resources among these virtual networks to tennas, wideband signals and a number of advanced signal pro-
meet the QoS requirements of different IoT networks. Among cessing algorithms, the above-mentioned reliability prediction
these virtual networks, each MTC device can select one of process becomes extremely complex [17] and the prediction of
the virtual networks to access to the physical network while PER with good accuracy becomes difficult in practice [145].
meeting its connection requirements [137]. In addition to radio Furthermore, due to a significantly large number of environ-
resources, it is also possible to enhance the utilization of other mental parameters such as channel state information, signal
network resources such as computing, caching and networking power, noise variance, non-Gaussian noise effect, transceiver
resources [73]. hardware impairments such as power amplifier non-linearity
and quantization error, it becomes challenging to provide the
VI. L EARNING -A SSISTED S OLUTIONS FOR RAN near-optimal/optimal tuning of the transmission parameters to
C ONGESTION P ROBLEM IN C ELLULAR I OT N ETWORKS achieve the efficient link adaptation [18]. The severity of this
problem greatly increases in ultra-dense networks due to the
A. Advantages of Learning Techniques in Wireless Communi- involvement of various agents and system parameters such
cations as Signal to Interference plus Noise Ratio (SINR) mismatch
The main questions this section attempts to answer are why in ultra-dense small cell networks [146], and therefore, the
learning techniques are important in wireless communication link adaptation in emerging ultra-dense networks becomes ex-
systems, which parameters to learn and for what purposes. tremely challenging. Also, the existing link adaptation systems
First, we present the main advantages of learning techniques are localized to individual links and small coverage areas, and
in wireless communications systems in general, and then do not take into account of the consequences on other systems
discuss why learning techniques are needed on the top of from the system-level perspective.
the conventional link adaptation techniques. Subsequently, we In order to make the link/system adaptation more flexible
discuss various parameters which can be learnt by using and efficient, existing works have applied ML techniques in
learning techniques in different application scenarios. different settings [17, 18, 141–143]. The contribution in [17]
As highlighted earlier in Section I-C, the number of config- investigated an online learning framework for the link adapta-
urable system parameters has increased significantly from one tion by using a modified k nearest neighbor (kNN) algorithm
cellular generation to the next one. For example, the number to learn the mappings between the channel conditions and
of configurable parameters has increased to about 1500 in a PER values for all possible Modulation and Coding Schemes
4G node from about 500 in a 2G node and from about 1000 (MCS) supported by the system. In this online learning frame-
in a 3G node, and it is predicted to be around 2000 in a 5G work, when a new packet is delivered, the predicted PER
node [15, 16]. In this regard, the process of optimizing these associated with each MCS is calculated by using the kNN
reconfigurable parameters in 5G and beyond systems becomes algorithm and the best MCS is selected. After a packet is
extremely complex and performing self-configuration, self- transmitted, the packet is stored as a prior data of the selected
optimization and self-healing operations will be challenging. MCS for future prediction purpose. Although the accuracy of
Also, emerging ultra-dense networks will need to observe PER prediction becomes more accurate with the increase in
environmental variations, learn uncertainties, plan response the number of packet transmissions, this kNN-based approach
actions and configure the network parameters effectively to requires to store all the previous samples and has higher time
handle these operations. To this end, emerging ML techniques complexity, not suitable for a real time operation [17]. In
could bring potential benefits in efficient handling of these op- this regard, the authors in [18] proposed an online kernelized
erations. The main role of learning techniques include learning support vector regression method which can work with the
the system variations/parameter uncertainties, classifying the minimal memory size and has low computational complexity
involved cases/issues, predicting the future results/challenges while providing a comparable performance to those of the
and investigating potential solutions/actions [16]. existing algorithms.
Wireless systems utilize link adaptation techniques to adapt Learning techniques are expected to provide significant
the physical layer parameters such as modulation and coding benefits by adaptively learning numerous parameters in various
scheme based on the reliability/quality of the communication application scenarios as listed below. Also, we provide a brief
link. In practice, different applications such as wireless video summary of these use cases along with the corresponding
broadcasting and VoIP demand for different reliability con- references in Table VII.
straints. Before performing this link adaptation, the reliability 1) Learning to exploit an unique RA slot for each MTC
of a wireless link in the form of some metrics such as PER device within the considered transmission frame in a
is predicted for each set of physical layer parameters, needs way that concurrent transmissions in the same RACH
to be predicted [17]. During the link adaptation process, there opportunity can be avoided [10].
arises a tradeoff between data rate and reliability since the 2) Learning to adapt an access control parameter, i.e.,
PER calculated for a set of physical layer parameters is in access barring factor for the RACH congestion [21].
IEEE COMMUNICATIONS SURVEYS & TUTORIALS (DRAFT) 25

TABLE VII
U SE CASES FOR THE APPLICATIONS OF LEARNING TECHNIQUES IN ULTRA - DENSE CELLULAR SYSTEMS

Use Case Sub-cases/related description Reference

RACH congestion minimization Learning to find the unique access slot for each MTC device [10]
Learning to adapt an ACB parameter [21]
Learning to associate MTC devices with the best eNodeBs [22, 23]
Autonomous adaptive resource allocation Learning to find the existence of the critical delay-sensitive messages [24]
Dynamic spectrum sharing Learning to predict the occupancy status of radio channels [138]
Learning-assisted edge-side processing Extracting the useful information from the raw sensor data [25]
at the edge devices to reduce the communication burden [25]
Selection of a suitable RAT Learning to select a suitable RAT among different RAT technologies [139]
under network conditions and user preferences constraints
Network traffic control Learning to control network traffic to enhance computational efficiency and scalability [140]
Adaptation of transmission parameters Learning link quality/reliability to adapt parameters such as MCS and transmission slot [17, 18, 141–143]
Data analytics To extract user activity/mobility patterns, temporal, spatial and social correlations [42]
Provisioning of personalized services To learn network contexts for a global view of communications, computing [144]
and caching resources

3) Learning to associate MTC devices with suitable unstructured/semi-structured data coming from massive
BSs/eNodeBs with the objective minimizing overall ac- number of sensors.
cess network congestion [22, 23].
4) Learning the existence of delay sensitive/critical mes-
sages by IoT devices in heterogeneous ultra-dense IoT
networks so that enough resources can be dynamically B. Learning Techniques for IoT/MTC Systems
allocated and critical information can be successfully
transmitted to the BS/eNodeB/aggregator as soon as they The main challenges of applying learning techniques in an
are generated [24]. Based on the learned information IoT environment include the following [26].
about critical messages, IoT devices can collectively
1) MTC devices have low computational capability, how-
adjust their uplink transmission parameters such as
ever, the widely-used ML techniques such as RL and
orthogonal codes, transmission slot period, periodicity
decision trees can be computationally complex to im-
of transmission and the received power for performing
plement.
autonomous resource allocation and the coordination for
2) Because of the distributed nature of IoT devices and high
the usage of available codes.
energy required to maintain constant communication
5) Learning the radio spectrum by dynamic spectrum shar-
with the BS/centralized aggregator, distributed learning
ing among the systems/nodes in a collaborative manner
needs to be investigated for an IoT environment.
to predict the occupancy status of radio channels [138].
3) Due to limited radio resources and energy constraints,
6) Learning the relationship of the contextual information
only the limited amount of information is available at
(related to the surrounding radio environment) collected
the IoT devices, and therefore, it is necessary to adapt
from IoT sensors to extract knowledge and to pre-
the learning mechanisms based on the limited amount
dict the future context at the edge devices [25]. In-
of information.
stead of transferring all the raw contextual data to the
4) In some critical applications such as eHealthCare and
network/cloud-centre, only the inferred knowledge can
industrial control, IoT devices need to learn quickly
be transferred, thus reducing the communication burden.
in order to satisfy the ultra-reliable and low-latency
This approach deals with pushing learning intelligence
requirements. For this purpose, learning time should be
from the network to the distributed edge devices having
as small as possible to quickly adjust the performance
heterogeneous computing abilities.
parameters.
7) Learning for the selection of Radio Access Technology
5) To enable the harmonious coexistence of MTC and HTC
(RAT) while performing vertical handovers among het-
systems, learning techniques should consider both the
erogeneous networks having different RAT technologies
existing traffic as well as the new traffic from the MTC
under the constraints of network conditions and user
devices.
preferences, which can be designed on the device-side,
network side or in a hybrid manner [139]. Existing works have applied learning techniques in the
8) Learning for network traffic control (such as routing) context of MTC/IoT in the following ways: (i) adaptation
in ultra-dense heterogeneous networks to alleviate the of an access control parameter, i.e., access barring factor to
issues of computational efficiency and scalability of the minimize the RACH overload [21], (ii) learning a dedicated
existing approaches [140]. slot within the MTC transmission frame by using an intelligent
9) Learning link quality/reliability to adapt the transmission slot assignment strategy to avoid the collisions of access re-
parameters (such as MCS, transmission slot, received quests [10], (iii) BS/eNodeB selection by using Q-learning/RL
power etc.) of a wireless link. [17, 18, 141–143] techniques to minimize the access network overload [22, 23],
10) Learning to extract user activity/mobility patterns, and (iv) sequential learning with finite memory in order to
temporal, spatial and social correlations from raw learn transmission parameters under stringent memory and
computational constraints [24, 147].
IEEE COMMUNICATIONS SURVEYS & TUTORIALS (DRAFT) 26

Machine
Learning
Techniques

Supervised Unsupervised
Learning Learning

Reinforcement
Regression
Learning
Density
estimation
Classification
Dimensionality
reduction
Policy-iteration
Clustering
based
Value-iteration
based
· Support vector
· Neural
machine
Network (NN) · Boltzmann
· Binary
· Trees Machine
decision tree · Monte-Carlo
· Deep NNs · Auto- · Kernel
· Naive Baysian based · Q-learning associative NN density
classifier · Temporal · K-means
· Variations of · Isometric · Gaussian
· NN difference · PCA
Q-learning Feature mixtures
· Deep NNs methods (e.g., · Spectral
(collaborative Mapping
SARSA) clustering
adaptive, (ISOMAP)
· Dirchlet
modular, · Local linear
process
fuzzy-Q) embedding

Fig. 6. Classification of existing machine learning techniques.

C. Overview of Existing Machine Learning Techniques and caching/memory resources [26].

ML techniques learn necessary information either from the On the other hand, the second category of ML techniques,
available data-sets or by interactions with the surrounding i.e., unsupervised learning does not require the need of labels
environments and make suitable decisions on future actions to of data-sets and is more complex than supervised learning
be followed by the agents either based on some learned models techniques in terms of the computational cost [140]. Although
or in a model-free manner. The classification of existing this learning class has not been widely used as compared
ML techniques in terms of different bases such as learning to the supervised learning in the current context, it can be
principles, objectives and the employed algorithm is provided considered as a promising future ML paradigm since the
in Fig. 6 [140]. On the basis of the employed learning mecha- main objective of ML is to make the learning agent capable
nisms, the existing ML techniques can be broadly categorized of learning without any supervision or human intervention.
into the following [26, 140, 148]: (i) supervised learning, (ii) Since the learning agent does not have any training data-
unsupervised learning, and (iii) reinforcement learning, which set and the knowledge of the output, the learning process
are briefly described below. Figures 7 and 8 provide the is quite complex as compared to the supervised counterpart.
illustrations of the principles of these three categories of ML This learning approach divides the unlabelled heterogeneous
techniques [149]. data into smaller homogeneous sub-sets which can be eas-
The first category of ML techniques, i.e., supervised learn- ily understood and managed [43]. Therefore, unsupervised
ing requires the need of training data to be labelled and the learning techniques are mainly based on following three
output of the algorithm needs to be already fed to the machine. objectives [140]: (i) clustering, (ii) dimensionality reduction,
Being aware of the output, the learning agent builds a model and (iii) density estimation. For clustering-based unsupervised
to move from the input to the output guided by the input learning, different ML algorithms such as K-means, spectral
training set. Based on the employed learning algorithms, the clustering, Principal Component Analysis (PCA) and Dirchlet
supervised learning techniques can be classified into [43, 140]: processes can be utilized. Similarly, unsupervised learning
Artificial Neural Networks (ANNs), Deep NNs, Bayesian Net- for dimensionality reduction can be employed by using ML
works (BNs), Support Vector Machine (SVM), Deep Learning algorithms like auto-associative NN, local linear embedding
(DL), Case-based Reasoning (CBR), Decision Trees (DTs), and Isometric Feature Mapping (ISOMAP). Furthermore, the
K-Nearest Neighbor (KNN), Instance-based Reasoning (IBR) density estimation-based unsupervised learning can be realized
and Naive Bayesian Classifier. The main difficulty of applying by using ML algorithms such as Kernel density, Gaussian
supervised ML techniques in IoT scenarios is that they require mixtures, Boltzmann machine and Deep Boltzmann machine.
the processing of extensive data-set to learn from the dynamic In order to take the advantages of supervised and unsu-
environment but IoT devices are limited in terms of computing pervised learning, there is another intermediate type of ML
IEEE COMMUNICATIONS SURVEYS & TUTORIALS (DRAFT) 27

techniques, called semi-supervised learning technique [150]

which uses both the labelled and unlabelled data for learning. Training
Supervisor
Desired
The main objective of semi-supervised learning scheme to data set output
· No training data set
· No desired output
improve the learning accuracy by utilizing a small amount
Output Raw data Output
of the labelled data along with the large amount of the Raw data
Learning and Interpretation
Learning and
Processing processing
unlabelled data. The above-mentioned types of ML techniques
have been applied mainly in the centralized framework [148].
For example, cloud-based processing can enable the operation
of big data analytics to handle the massive amount of data Classification/
regression
Algorithms like
clustering and
gathered from the heterogeneous IoT sensors/devices [19]. algorithms density
estimation
The third category of learning techniques, i.e., RL enables a
number of agents to interact with the environment and involves
(a) (b)
an environment of states, actions to be taken by the agents,
state transition functions, an immediate reward and an initial
observation function [26]. In this approach, an agent learns Fig. 7. Illustrations of the principles of supervised and unsupervised learning.
from the previous experience in the absence of the training
Environment
data set. This learning mechanism deals with finding a proper
balance between exploration for the random actions and ex-
ploitation of current knowledge, i.e., exploration-exploitation
trade-off. The role of exploration phase in RL learning is
to attempt some random actions towards searching better
rewarding actions, while the exploitation phase tries to utilize
the previously learned utility to maximize the reward for the

Action
agent [151]. Although an RL technique requires the knowledge Interpreter
of state transition function and it has slow convergence, its
unique feature of action-reward feedback to the agents makes
this suitable for the applications in IoT systems.
Besides the above-discussed three categories of ML tech-
niques, some researchers have also considered another cat-
egory of learning techniques, called as Sequential Learning Algorithms
such as Q-
(SL) [26], which helps the autonomous agents to learn the true
learning
underlying state of the environment having binary states. In Learning device
this approach, the agents learn the system state in the sequence
by following a given order while observing the environment
Fig. 8. Illustration of the principles of a reinforcement learning technique.
and the actions or observations of previous agents, and then
eventually converge to a true underlying state with repeated
hypothesis testing [26]. The main advantage of employing the achievable RACH throughput in the presence of massive
SL techniques in the IoT systems is its flexibility in terms access requests/loads gets significantly reduced [10]. For ex-
of memory requirements since it enables the convergence of ample, the maximum throughput of the widely-used slotted
finite memory SL in the resource-constrained IoT devices ALOHA technique is e−1 ( 37%). Furthermore, because of
[147]. However, the main drawback of the SL approach is low RACH throughput and the employed backoff strategies,
that it relies on direct communication links among MTC the aggregated traffic including both the newly generated and
devices since the information required for SL comes from the retransmitted exceeds the RACH channel capacity at a cer-
other agents, thus leading to the requirement of additional tain point, thus making the system unstable. Although Slotted
network resources. ALOHA can work well with the conventional cellular/HTC
Among several RL techniques, Q-learning requires low traffic despite the instability issue, the support of M2M traffic
computational resources for its implementation and does not becomes problematic due to infrequent and massive number of
require the knowledge of the model of the environment, access requests, thus causing the problem of RACH overload.
thus being a suitable learning technique for the resource- Learning techniques can be employed at the MTC devices in
constrained IoT devices [151]. Furthermore, it is possible to order to enable them to learn to avoid concurrent transmissions
implement this technique in a distributed way. Therefore, in during the RACH contention period without any assistance
the following subsection, we utilize the Q-learning technique from the central entity. After a learning technique achieves
to address the problem of RACH congestion in cellular IoT its convergence, each MTC device can get a unique dedicated
networks. slot, thus avoiding collisions among their transmissions. In
the following, first, we present a framework for the RL with
D. Q-learning for RACH Congestion Problem a single MTC device and then develop the formulation of Q-
The main problem with the existing contention-based RA learning in the considered context.
schemes is that the occurrence of collisions is unavoidable and The environment perceived by an MTC device can be
IEEE COMMUNICATIONS SURVEYS & TUTORIALS (DRAFT) 28

usually described by a Markov Decision Process (MDP) and Q(x, u) and can be calculated as the expected value of the sum
a finite MDP can be denoted by a tuple < X, U, f, ρ >, where of immediate reward and discounted utility of the resulting
X represents the finite set of environment states, U is the finite state after executing the action ‘u’. In the Q-learning process,
set of device actions, f denotes the state transition probability the current estimate of Q∗ value, i.e., Qt (xt , ut ) is updated
function and captures the environmental dynamics, and ρ is the by using the estimated samples given by the right-hand side
reward function [44]. In this MDP modeling, a state parameter of (14), which are computed by relating with the actual
xt ∈ X indicates the characteristics of the environment at the experience from the execution of the action, in the form of
tth time instance. At each time-step, the device can change the pairs of subsequent states (xt , xt+1 ) and the rewards rt+1 .
its state by taking actions ut ∈ U and due to this action, In this way, this Q-learning process transforms (14) to the
the environmental state alters from the current state xt to following iterative procedure [44].
the other some state xt+1 based on the employed transition
Qt+1 (xt , ut ) = Qt (xt , ut ) + αt [rt+1
probability function f (xt , ut , xt+1 ). During this transition, the
device receives an instantaneous reward rt+1 ∈ R with the +γ max
0
Qt (xt+1 , u0 ) − Q(xt , ut )]. (15)
u
defined function ρ, i.e., rt+1 = ρ(xt , ut , xt+1 ).
where αt denotes the learning rate applies at the tth time-
Given a state, the device chooses its action based on its
step and the expression inside the square brackets indicates
policy π and the policy can be either stochastic or determin-
the difference between the estimates of Q∗ (xt , ut ) at two
istic. Each time the device applies a policy, it accumulates
successive time steps.
the Prewards from the environment, resulting in the return
L l To provide an example framework for the application of
of l=0 γ rl+1 , where γ ∈ [0 1] is the discount factor Q-learning in an MTC scenario with N number of devices,
which provides more weights on the immediate rewards and
we consider a frame-based slotted ALOHA scheme as in [10,
L denotes the length of one episode, which denotes the time
152], in which a frame is divided into K number of access
period after which the state is reset for an episodic MDP [45].
slots. Each MTC node has individual Q values corresponding
For the case of non-episodic MDP, L = ∞. At each time-step,
to every slots in the frame and these values are updated based
the learning device aims to maximize the expected discounted
on the outcomes of transmission, i.e., success or failure. At
return in the long-term, i.e., long-term reward, given by [44,
the start, all the MTC nodes can start with zero or random Q
45]
X L
! values, learn gradually via their transmissions, and then finally
Rt = E γ l rt+l+1 , (12) reach to the optimal transmission strategy after finding unique
l=0 RA slots for their transmissions.
Let Q(i, k) indicate the preference of the ith node to trans-
where E denotes the expectation operator and this is taken
mit a packet in the kth RA slot. After every data transmission,
over probability state transitions, i.e., dynamics of the consid-
the new Q value, i.e., Qt+1 (i, k) is updated based on the
ered environment.
previous Q value and the current reward based on the following
In this RL process, the learning device attempts to maximize
relation
its long-term performance, while only receiving feedback
about its immediate action, i.e., one-step performance. To Qt+1 (i, k) = Qt (i, k) + α(R − Qt (i, k)), (16)
achieve this, the device needs to compute an optimal action-
value function, known as a Q-function. Given a certain policy where R is the current reward and α is the learning rate.
π, the expected return of a state-action pair, Qπ (x, u) is given The reward value of R = +1 is assigned if the transmission
by becomes successful, and otherwise, R = −1. At each instance
! of transmission, the node selects a slot with the highest Q
L
π
X
l
value and in case of two or more maximum values, a random
Q (x, u) = E γ rt+l+1 |xt = x, ut = u, π . (13) selection approach can be applied. Regarding the selection of
l=0 α, the higher the value, the faster will be the convergence of Q
Subsequently, the optimal Q-function can be written as: value towards the reward value R. However, a small value of
Q∗ (x, u) = maxπ Qπ (x, u) and it satisfies the well-known α is usually preferred to enhance the robustness with respect
Bellman optimality equation in the following way. to infrequent collisions caused by channel variations [152].
X Furthermore, although learning rate α is usually considered
Q∗ (x, u) = f (x, u, x0 )[ρ(x, u, x0 ) fixed in the existing works, it is typically time varying in
x0 ∈X
nature, decreasing with time and each state-action pair may
+γ max
0
Q (x , u0 )], x ∈ X, u ∈ U.
∗ 0
(14) be associated with a different learning rate [44].
u

One simplest way of choosing a future action by a de-

vice is to employ a greedy policy which selects the action E. Exploration Strategies for Q-Learning
with the highest Q-value at every state as follows: π(x) = Q-learning aims at finding an optimal policy to select an
arg maxu Q(x, u). action in the current state and one of the design aspects for the
Among different ML techniques listed in Fig. 6, Q-learning Q-learning algorithm is to balance the exploration-exploitation
is simple to implement and uses a look-up table with the tradeoff. The exploitation is performed by executing one of
Q-values representing the utilities for state-action pairs. The the actions which maximizes Q(x, u) whereas the exploration
utility of taking an action ‘u’ in a state ‘x’ is denoted by is carried out by randomly selecting an action to build a
IEEE COMMUNICATIONS SURVEYS & TUTORIALS (DRAFT) 29

better estimation of the optimal Q-function.There are several number of agents [155]. This becomes difficult even in the
strategies to create this balance and the widely-used three case of a single state case. For instance, when M number of
strategies are described below [153, 154]. learning devices play the repeated game with only two actions,
1) -greedy strategy: This is the most commonly used the size of the table becomes 2M . Therefore, it requires more
exploration strategy in which the Q-learning algorithm state space, information space and action space. Furthermore,
utilizes a parameter 0 ≤  ≤ 1 to decide on the in the multi-agent case, state transitions, the instantaneous
action to follow. The algorithm chooses the action with rewards and future expected return are based on the joint
the highest Q-value in the current state with (1 − ) action of the multiple agents. In addition, instead of the
probability and a random action with the probability individual policies, a joint policy formed by individual policies
. The value of the parameter  can be varied over will impact the Q-function. Therefore, the design of state-
time as the learning progresses. The main drawback of action pairs or the optimal policy to maximize the overall
this approach is that it treats all the possible actions reward becomes the joint/collaborative problem in a multi-
equivalently during its exploration by choosing an action agent environment, thus leading to the need of collaborative
uniformly from the set of possible actions. learning.
2) Soft-max strategy: To address the drawback of -greedy Some existing works have applied collaborative Q-learning
stage during the exploration phase, this soft-max strategy in different settings [25, 155, 157, 158]. One way of realiz-
uses either a Gibbs or Boltzmann distribution, in which ing collaborative Q-learning is to estimate the belief of the
the learning device at a state x selects an action u with opponent and the environment knowledge instead of the Q-
the following probability value function in a way that the learning agent does not
Q(x,u) need to observe the opponents’ reward and their Q learning
e τ
π(x, u) = P Q(x,u)
, (17) parameters [155]. In this regard, the authors in [156] applied
u e τ the collaborative Q-learning framework to optimize the waiting
where τ > 0 denotes the temperature parameter of the time in intelligent traffic control applications. Furthermore,
Boltzmann distribution and depicts how randomly the the authors in [25] employed a collaborative ML technique
values are chosen. For example, τ = 0 case represents at the edge computing devices with the objective of extracting
no exploration at all while T → ∞ case reflects the the statistical relationships among the contextual information
scenario in which the learning device chooses the action collected by the edge devices and constructing predictive
values almost with equal probability. models to maximize the communication efficiency. With this
3) Optimism in the face of uncertainty: This approach edge-centric learning approach, only the inferred knowledge
encourages exploration by assigning higher initial values can be transferred to the network instead of transmitting all
to the Q-function. However, the convergence time will the raw contextual information. Furthermore, the authors in
be increased since the estimated Q-values can be quite [157] used collaborative Q-learning in finding an optimal path
bad estimates of the actual Q-value and these bad esti- between any starting point and a target in a grid environment
mates may last longer during the learning process. Also, for a mobile robot. In contrast to the conventional approach
in the presence of dynamic uncertainty, this technique of using a single Q-table, the work in [157] exploited the use
is not useful since the exploration mostly occurs at the of two Q-tables, i.e., one local Q table and another Master Q
beginning of the learning process. table.
Moreover, in a multi-agent environment, it is necessary for
F. Performance Enhancement of Q-Learning a learning agent to keep the track of its environment, as well
as other agents’ actions. In such a multi-agent environment,
In the following, we present several variants of Q-learning
rather than considering individual actions of the agents as in
techniques which aim towards improving the performance of
the ordinary Q-learning, the joint actions need to be considered
the ordinary Q-learning.
while devising a learning strategy [159]. Instead of the Q-
1) Collaborative Q-learning: In this subsection, we de-
value used in the ordinary Q-learning, a Nash Q-value function
scribe the need of collaborative Q-learning in ultra-dense
should be defined. For the mth learning device in a multi-
IoT networks and review the related works. The traditional
agent environment, the Nash Q-value function can be defined
Q-learning algorithm which is based on single state-action
as [159, 160]
may not be suitable for the multi-agent environment with
multiple policies. In this regard, collaborative learning can
Qt+1 t t ˜t 0
be utilized by exploiting the overall/global objective/reward m (x, u1 , ...uM ) = (1−α)Qm (x, u1 , ...uM )+α[rm +γ Qm (x )],
of the collective environment instead of the reward for a (18)
t
single learning device. Furthermore, the global reward can be where (u1 , ...uM ) is a joint action, rm is an one-period reward
utilized in addition to the individual reward to improve the for the mth learning device and Q̃tm (x0 ) denotes the payoff of
performance of the existing learning schemes. the mth device in the state x0 for the chosen Nash equilibrium.
In a multi-agent environment like the case of ultra-dense The convergence of the Q-learning algorithm using the Q-
IoT networks, if all agents keep mappings of their joint states value function defined in (18) becomes slower as the number
and actions, this will require each learning device to maintain of agents increases due to the resulted increase in the joint
very large Q-value tables, whose sizes are exponential in the action set [159].
IEEE COMMUNICATIONS SURVEYS & TUTORIALS (DRAFT) 30

2) Situation-Aware Adaptive Q-learning: The collisions of environment due to discrete set of actions and spaces, and it
the RA requests caused due to concurrent transmissions of becomes complex to learn the problem with multiple objec-
multiple RA requests in one RACH sub-frame results in higher tives [167]. To solve these issues, fuzzy-logic based rules can
access delay since the devices have to retransmit their RA be employed to tune the learning parameters of the Q-learning
requests. Although the average access delay can be minimized technique towards making it more adaptive.
by using a higher number of RACH sub-frames, the sub- In the context of cooperative fuzzy Q-learning, authors in
frames available for data transmission will be reduced since [166] utilized this learning approach to optimize the coverage
the sub-frames allocated for RACH procedure can not be used and capacity of cellular networks by adapting the tilting of
for the data transmission purpose [161]. In this regard, it is vertical antennas. The employed cooperative Fuzzy Q-learning
crucial to balance the tradeoff between the radio resources mechanisms enable cooperation among the learning agents
allocated for RACH procedure and data transmission process. during the exploration phase and is fully distributed in the
Furthermore, in many cases, the network usage pattern (the exploitation phase. The cooperation in the exploration phase
number of devices/users trying to connect to the network) is is employed by utilizing the global reward of all the considered
time varying in nature. In this context, a network should be cells instead of the local rewards belonging to individual cells.
capable of detecting the variation in the arrival rate of the This cooperation with the help of global reward helps to speed
access request and should adapt the number of sub-frames up the exploration phase of the Q-learning process while also
to be allocated for RACH accordingly [161]. In addition, it allowing the learning agents to exploit the learned knowledge
is important to balance a tradeoff between exploration and independently while selecting their actions. Furthermore, a
exploitation while adapting the Q-learning parameters to the self-learning cooperative strategy is developed in [163] by
dynamic uncertain environment [162]. combining adaptive Q-learning with the fuzzy method for its
Although Q-learning has been shown to converge and has application in robot soccer systems.
been used in many fields including mechatronics control Moreover, the contribution in [168] recently proposed a
and robotics, it has some issues such as how to improve fuzzy Q learning-based user centric backhaul-aware user as-
the convergence rate and to avoid the convergence in the sociation scheme in which the BSs broadcast their constraints
local optimum [163]. To address these issues, three different and capabilities in terms of meeting the requirements of
parameters of the Q-learning technique, namely, learning rate heterogeneous UEs in terms of the optimized bias factors.
α, discount rate γ and temperature parameter τ in Boltzmann The employed fuzzy-logic based Q-learning scheme helps to
distribution, should be dynamically adapted based on the dynamically adjust these bias factors based on network condi-
dynamicity of the underlying learning environment. One of tions and users’ requirements in an automated and distributed
the widely used methods to adapt the learning parameters in manner.
various applications is fuzzy-logic based learning, which is Similarly, authors in [169] proposed a fuzzy Q-Learning
briefly described in the following subsection along with the based energy controller for a small cell powered by local
related literature. renewable energy, local storage, and the smart grid to elongate
3) Fuzzy-logic based Adaptive Q-learning: In the Q- the lifespan of the storage devices and to minimize the
learning process, the Q-values are usually stored in a look-up electricity expenditures of the mobile operators. The employed
table but this storage process becomes infeasible in practice fuzzy Q-learning based controlled can be utilized without the
in the presence of a large number of state-action spaces and prior knowledge of the mobile traffic demand, energy pricing
with the continuous state space [164]. Although Q-values and weather.
can be stored by using feed-forward neural networks or self- In order to avoid the inefficient and expensive manual tuning
organizing maps, the learning process becomes slower. In of cellular network parameters in 5G small cell networks, it is
contrast, incorporation of Q-learning into fuzzy environments crucial to perform automatic configuration and optimization of
seems promising since fuzzy interference systems being uni- the network parameters including the handover parameters. In
versal approximations can be considered as good candidates this regard, authors in [170] employed a fuzzy logic controller
to store Q-values and the prior knowledge can be provided based dynamic fuzzy Q-Learning algorithm for mobility ro-
to the fuzzy rules in order to significantly reduce the training bustness optimization in a heterogeneous network. With the
part. proposed dynamic fuzzy Q-learning algorithm, the system
The implementation of Q-learning becomes impractical learns necessary parameter values towards optimizing the call
and even impossible in continuous state spaces [164]. In dropping ratio and handover ratio, and it has been shown that
such cases, fuzzy-logic based approach helps to discretize the Q-Learning algorithm can lower the handover ratio while
the continuous state or action spaces into finite states by keeping the call-dropping ratio at the lower level. Besides,
employing suitable fuzzy rules and also the speed of fuzzy- by considering various parameters such as link quality, the
logic based Q-learning can be increased by incorporating the available bandwidth, link quality, and relative vehicle move-
prior knowledge via fuzzy rules [165]. In other words, Fuzzy ment, authors in [171] proposed a fuzzy constraint Q-learning
Q-learning discretizes continuous variables by using fuzzy algorithm for vehicular ad-hoc networks in order to evaluate
labels and a fuzzy rule-based inference system is employed the quality of a wireless link towards finding the optimal route.
to find an action for these discretized states [166]. 4) Model-based Q-Learning: The main drawback of
Other drawbacks with the ordinary RL are that it is difficult model-free learning is the convergence time. By predicting a
with the continuous states and behaviors in the real world model about the transition state probabilities, the performance
IEEE COMMUNICATIONS SURVEYS & TUTORIALS (DRAFT) 31

of the learning techniques could be improved. In time-varying spectrum sharing solutions for 5G and beyond systems in-
dynamic scenarios, it becomes advantageous to predict the clude interweave cognitive communications, underlay cogni-
environmental dynamics in the centralized entities such as tive communications, carrier aggregation, Licensed Assisted
cloud-center and to utilize the corresponding model to enhance Access (LAA), Licensed Shared Access (LSA) and Spectrum
the performance of distributed Q-learning at the resource- Access System (SAS) [72]. Another option is to explore higher
constrained IoT devices. In such a collaborative cloud-edge frequency bands such as millimeter wave (mmWave) bands.
processing framework [19], the predicted model at the cloud- However, due to propagation related and hardware imper-
center can be communicated to the edge-side to improve the fections issues in mmWave bands, it becomes challenging
performance of distributed learning at the edge-side of the to operate mMTC devices in mmWave bands as they are
network. constrained in terms of resources and are mostly deployed in
Existing works have used model-based Q-learning in dif- indoor or underground environments where propagation issues
ferent applications such as robotic applications [172, 173] and could become problematic. Therefore, for mMTC applications,
wireless channel allocation [174]. the frequency bands below 6 GHz is of more importance. In
this spectrum region, there arises the issue of whether licensed
or unlicensed band becomes suitable for mMTC applications
VII. R ESEARCH C HALLENGES AND F UTURE D IRECTIONS as pointed out in the following.
A. Design Issues for Low-Power MTC Device Transmission In the spectrum region below 6 GHz, the operation in
Schemes the unlicensed band by using carrier aggregation and LAA
could be a cost-effective solution for mMTC applications
Current protocols designed for cellular IoT such as NB- since these frequencies are freely available to any device.
IoT and LTE-M are based on the assumption of the low- However, uncontrolled interference conditions resulted from
latency requirement. This requirement results in significant the free access from the massive number of devices and the
cost in terms of device price and system capacity [34]. Besides, lack of QoS guarantees may severely limit the ability of
the main issues involved in providing cellular connectivity utilizing unlicensed bands for mMTC applications [4]. On
to the low-power MTC devices include the device battery the other hand, emerging techniques such as LSA and SAS
life, system capacity, coverage and cost. Due to low cost provide better interference characterization due to the central-
and low capability of MTC devices, a significant compromise ized control and seem more suitable for mMTC applications.
in the link performance needs to be made resulting in the Besides, some of the mMTC applications based on periodic
shrinkage of the coverage area. One way of compensating data reporting such as smart metering can effectively work
this coverage loss is to utilize an extended transmission time with the shared spectrum bands in a demand-based manner
interval, which, however, will lead to the increase in the battery instead of exclusively assigning the portion of a licensed
consumption. Therefore, it is crucial to balance the tradeoff spectrum. Due to low bit-rate requirements of mMTC appli-
between energy consumption and coverage expansion while cations, the bandwidth requirement may not be significant,
designing transmission technologies for the MTC devices. however, the exclusive licensed spectrum can be allocated to
Furthermore, transmission schemes should be able to sup- more demanding applications such as eMBB and URLLC.
port a significantly large number of devices within a given To this end, it is an important future research direction to
bandwidth while ensuring higher battery efficiency. In this re- investigate the feasibility of utilizing shared spectrum for
gard, the concept of effective bandwidth [34] could be utilized mMTC applications.
to achieve a good balance between the spectral efficiency for
a given coverage and the corresponding transmission time.
On one hand, the bandwidth allocated for a device should C. Traffic Characterization Issues for mMTC Systems
not be far less than the effective bandwidth to avoid the Since traffic characteristics in IoT sensory networks usually
excessive transmission time. On the other hand, for a given depend on the application scenarios, traffic characterization
sensitivity level of the device, it is preferred to have allocated is considered to be an important issue for the design and
bandwidth not more than the effective bandwidth in order to optimization of the network infrastructure. The mMTC net-
accommodate more number of devices in the saved bandwidth works may generate various types of traffic patterns such as
without significantly increasing the transmission time. More- PU, ED and streaming, and these traffic patterns may have
over, advanced transmission scheduling techniques and low different amplitudes, activation periods and starting times [52].
signalling overhead MAC protocols need to be investigated to Besides, the data packets generated by MTC devices can be
effectively support the massive number of MTC devices in the of varying sizes and bandwidth requirements. For example,
upcoming 5G and beyond cellular networks. the data packets generated by the temperature and humidity
sensors are usually of small size in the order of bytes, whereas
video monitoring devices can generate data sizes in the order
B. Spectrum Issues for mMTC Systems
of megabytes.
Since the available usable spectrum below 6 GHz is limited, As highlighted in Section II-D, the 3GPP has defined two
it is crucial to investigate suitable spectrum sharing solu- different models for the aggregated traffic models for the MTC
tions for emerging 5G and beyond systems including eMBB, traffic. The first model based on uniform distribution is for the
URLLC and mMTC. The most commonly discussed dynamic non-synchronized type of traffic whereas the second model
IEEE COMMUNICATIONS SURVEYS & TUTORIALS (DRAFT) 32

based on Beta distribution is for the highly synchronized distributed across the different entities of the network includ-
traffic. This aggregated traffic modeling is simpler but is less ing the devices, aggregator/eNodeBs and core-network/cloud-
precise than the source traffic modeling which models the center. The coordination of these distributed resources to
traffic for each MTC device separately and is more complex. In enhance the performance of ultra-dense IoT networks involv-
this regard, investigating suitable low-complexity and precise ing low-end MTC devices is an important research issue.
traffic modeling is an important future research direction. Furthermore, in the emerging cloud-assisted IoT networks, it
Furthermore, the Transmission Control Protocol (TCP) em- is advantageous to handle computationally intensive task at
ployed at the transport layer of current LTE-A networks is the cloud-center due to the availability of huge computing
not efficient for MTC traffic due to several issues associated and storage capabilities while it becomes beneficial to handle
with connection set up, congestion control, data buffering and delay-sensitive applications at the edge-side of the network.
real-time applications [52]. Therefore, it is crucial to develop In this regard, it is an important future research direction
an enhanced version of TCP over LTE/LTE-A in order to to investigate suitable techniques for edge-cloud collaborative
accommodate the MTC traffic. processing in various applications including dynamic spectrum
sharing, event detection, context-aware resource allocation,
live data analytics and security enhancement [19].
D. Issues with Machine Learning in the mMTC environment Moreover, various features of MTC device transmissions
While employing ML techniques in an mMTC environment, such as time-controlled, time tolerant, priority alarm message,
an important aspect to be considered is how to make ML tech- infrequent transmission, group-based policing and addressing
niques more practically realizable for the resource-constrained can be useful to utilize the distributed cache embedded in MTC
MTC devices. To achieve this objective, the following issues devices. The caching capabilities distributed across heteroge-
need to be considered. First, the convergence rate/learning time neous IoT devices can enable the scheduling of sporadic trans-
of the employed learning algorithm should be as small as pos- missions from the MTC devices in order to significantly reduce
sible and there may arise the issue of a local minimum. Since the peak traffic in an IoT access network as demonstrated in
the required learning time may reduce the time for data trans- [13]. This will subsequently reduce the demand for the radio
mission purpose, their trade-off should be designed properly. resources at the peak time, thus significantly saving the radio
Second, in mMTC environments, the distributed implementa- resource cost for the telecommunication operators. In addition,
tion of the algorithm needs to be considered across multiple distributed caching can be exploited to enable aggregate trans-
learning devices. Third, different parameters associated with mission for various other applications such as saving energy
the learning algorithms such as learning rate α, discount rate for low-power IoT devices and reducing signalling/protocol
γ and exploration-exploitation tradeoff parameter  should be overhead for MTC device transmissions. Also, the physical-
adapted dynamically to enhance the performance of Q-learning level cache embedded in low-end distributed MTC devices
algorithm in dynamic environments. Furthermore, there may can be exploited to facilitate the implementation of a cross-
arise the fairness issue while applying learning algorithms layer based transmission scheduling in pushing data from the
in a multi-agent environment since different devices may physical layer to the MAC layer at the suitable intervals.
reach to convergence at different time intervals, thus creating
different learning times for different devices. Moreover, the F. Device Heterogeneity and Grouping-based Transmission
heterogeneity of MTC devices need to be considered in terms Schemes
of different aspects such as learning capability, cache size, data The heterogeneity of MTC devices in terms of different as-
rate and delay tolerance limit. pects such as computing capability, cache size, battery power,
In the above context, future works should focus on address- data rate and latency requirements becomes problematic for
ing the aforementioned issues to employ the ML-assisted solu- the efficient QoS provisioning in ultra-dense IoT networks.
tions towards enabling the incorporation of MTC devices in the Furthermore, RAN congestion, high signalling overhead and
upcoming 5G and beyond cellular networks. Various emerging power consumption are critical issues to be addressed in ultra-
techniques described in Section VI-F such as collaborative dense IoT networks. In this regard, grouping-based features of
learning, situation-aware adaptive learning, fuzzy-logic based MTC transmissions such as group-based policing and group-
Q-learning and model-based learning could be exploited to based addressing [9] can be utilized to enable the group-
enhance the performance of the ML techniques in ultra-dense based operation in the mMTC environment towards alleviating
IoT networks involving MTC devices. various problems such as signalling congestion and power con-
sumption. Furthermore, by implementing a group-based access
authentication technique, severe signalling congestion caused
E. Distributed Resource Management in Ultra-Dense IoT Net- by the conventional independent access authentication scheme
works in the existing cellular systems can be avoided [175] and the
In contrast to the connection-oriented design approach of security of emerging MTC applications can be significantly
the existing wireless networks while considering only the enhanced.
communication resources, future content-oriented networks By grouping MTC devices on the basis of either service
are expected to utilize other resources such as computing requirements or the physical locations of MTC devices, group-
and caching resources. In ultra-dense IoT networks, these based data gathering, aggregation and reporting can be uti-
communications, computing and caching resources are usually lized in ultra-dense IoT networks including IEEE 802.11ah
IEEE COMMUNICATIONS SURVEYS & TUTORIALS (DRAFT) 33

based systems [176]. In such group-based schemes, a group However, the application of deep learning in IoT systems
header/cluster head collects the access requests, uplink data faces the crucial challenge of meeting the computational
packets, and device status information from the resource- requirements. The main associated issues include the high-
constrained MTC devices belonging to that group, and then speed training of large-scale IoT networks with the massive
forwards the aggregated traffic to an eNodeB/aggregator [6]. data-sets and embedding deep learning capability in low-power
Also, the downlink data packets and signalling messages IoT devices [179]. This computational challenge caused due
can be relayed to the MTC devices by the group header, to the expected growth in the size of the data-sets and the
thus significantly reducing the radio resources required for algorithmic complexity of ML algorithms is demanding the
direct communications between the devices and the eNodeB. need of improving the computational efficiency of existing
Moreover, as the jittering constraints become challenging computing platforms. Also, it is not feasible to offload all the
while transmitting small data packets from a large number of data to the cloud-center for processing due to constraints on
devices, this issue can be addressed by employing grouping- the bandwidth, privacy and battery life, and the computational
based resource scheduling which allocate radio resources to efficiency at the device-side needs to be accelerated for deep
the specific groups having similar QoS characteristics [61]. learning applications. In this regard, some of the potential
However, the existing device grouping mechanisms are formed future solutions to improve computational efficiency of ML
mostly in a traditional way by selecting the devices in a algorithms include deep-learning accelerators, approximate
random fashion or in an uniform manner, and it is an important computing and post-CMOS device technologies [179].
research direction to investigate efficient grouping mechanisms Many IoT products have already used the ML techniques
by exploiting the QoS characteristics and locations of hetero- to acquire and analyze the environmental data. For example,
geneous MTC devices. Google’s Nest Learning Thermostat uses ML algorithms to
understand the patterns of its users’ temperature schedules
and preferences by utilizing the temperature data recorded in
G. Deep Learning for Emerging IoT Applications and Asso- a structure way. However, unstructured multimedia data such
ciated Issues as visual images and audio signals are difficult to learn by
In real-world IoT environments, one of the major issues using the conventional ML techniques. In this regard, some
is how to reliably extract the meaningful information out of emerging IoT devices are already using sophisticated deep
the massive unstructured/semi-structured IoT data obtained learning techniques to capture and understand the complex
from a complex and noisy environment. Conventional ML environments [27]. As an example, the face-recognition se-
learning techniques may fail in such complex and dynamic curity system from the Microsoft’s Windows IoT team uses
environments and deep learning can be considered as a deep-learning technology to perform tasks such as unlocking
promising solution [40]. Due to multi-layer structure of deep a door by recognizing its users’ faces.
learning, it is considered as an effective approach for the Due to demanding real-time requirements of IoT applica-
edge-computing environment in order to accurately extract tions in terms of latency and high cost of radio resources
necessary information from the raw IoT sensor data [177]. required in delivering information to the cloud-center, it is
Since it is possible to offload the parts of learning layers in the advantageous to implement deep learning techniques at the
edge-side of the network and transfer the reduced intermediate device-side. However, due to the limited computing power
data to the remote cloud-center, the deep learning model seems and low memory size of IoT devices, it is challenging to
suitable for the emerging edge computing paradigms in IoT implement deep learning at the device-side. Therefore, most
systems. Another interesting advantage of deep learning in of the time, existing deep-learning applications require third-
edge-computing environment is that it can provide privacy party libraries and it may be difficult to migrate them to the IoT
preservation in transferring the intermediate data. In contrast to devices [27]. In this regard, it is an important future research
the traditional big data systems such as Spark or MapReduce in direction to investigate suitable paradigms such as convolution
which intermediate data contains the user privacy information, neural networks based inference engine [27] to facilitate the
the intermediate data generated in deep learning networks implementation of deep learning at the IoT devices.
usually have different semantics than that in the original source
data [40]. VIII. C ONCLUSIONS
In practical IoT systems, the data is usually dynamic and Future cellular IoT networks are expected to support the
unlabelled, and the conventional statistically trained models massive number of resource-constrained MTC devices while
are not efficient to handle the large unlabelled and dynamic satisfying their diverse QoS requirements and will need to
data-set. Furthermore, it is highly impractical to manually deal with several challenges for enhancing the access la-
label all the IoT raw data. Due to this, the conventional tency, scalability, connection reliability, energy efficiency and
supervised training-based learning techniques are not suitable network throughput. To this end, this paper has discussed
for large-scale IoT/mMTC environments [178]. Moreover, the various challenges of mMTC systems such as QoS provision-
conventional cloud-based architecture requires the transfer of ing, mMTC traffic characterization, transmission scheduling
a huge amount of IoT data from the edge-devices to the cloud- with QoS support, small data packet transmission and RAN
center. To address these issues, the application of deep learning congestion, and has provided a detailed review on the existing
with collaborative cloud-edge/fog processing seems to be a studies attempting to address these issues. By considering
promising future research direction [19, 178]. machine learning as an important enabler to address some of
IEEE COMMUNICATIONS SURVEYS & TUTORIALS (DRAFT) 34

these issues in ultra-dense cellular IoT networks, the paper [21] J. Moon and Y. Lim, “Access control of MTC devices using re-
has identified the potential advantages, research challenges inforcement learning approach,” in Proc. Int. Conf. on Information
Networking, Jan. 2017, pp. 641–643.
and the application scenarios of ML-assisted solutions. Among [22] M. Hasan, E. Hossain, and D. Niyato, “Random access for machine-
potential ML techniques, the application of Q-learning in to-machine communication in LTE-advanced networks: issues and
minimizing the RAN congestion has been presented along with approaches,” IEEE Commun. Mag., vol. 51, no. 6, pp. 86–93, June
2013.
some performance enhancement techniques. Finally, some [23] A. H. Mohammed, A. S. Khwaja, A. Anpalagan, and I. Woungang,
important research issues and interesting directions for future “Base station selection in M2M communication using Q-learning
research have been provided. algorithm in LTE-A networks,” in Proc. IEEE 29th Int. Conf. Advanced
Info. Networking and Applications, March 2015, pp. 17–22.
[24] T. Park and W. Saad, “Resource allocation and coordination for critical
R EFERENCES messages using finite memory learning,” in Prof. IEEE Globecom
Workshops (GC Wkshps), Dec. 2016, pp. 1–6.
[1] A. Al-Fuqaha, et al., “Internet of things: A survey on enabling [25] K. Portelli and C. Anagnostopoulos, “Leveraging edge computing
technologies, protocols, and applications,” IEEE Commun. Surveys through collaborative machine learning,” in Proc. 5th Int. Conf. on
Tuts., vol. 17, no. 4, pp. 2347–2376, Fourthquarter 2015. Future Internet of Things and Cloud Workshops (FiCloudW), Aug.
[2] H. Wang and A. O. Fapojuwo, “A survey of enabling technologies of 2017, pp. 164–169.
low power and long range machine-to-machine communications,” IEEE [26] T. Park, N. Abuzainab, and W. Saad, “Learning how to communicate
Commun. Surveys Tuts., vol. 19, no. 4, pp. 2621–2639, Fourthquarter in the internet of things: Finite resources and heterogeneity,” IEEE
2017. Access, vol. 4, pp. 7063–7073, 2016.
[3] ITU-R, “IMT vision - framework and overall objectives of the future [27] J. Tang, D. Sun, S. Liu, and J. L. Gaudiot, “Enabling deep learning on
development of IMT for 2020 and beyond,” ITU-R Rec. M.2083-0, IoT devices,” Computer, vol. 50, no. 10, pp. 92–96, 2017.
Sept. 2015. [28] M. Wang, et al., “Cellular machine-type communications: physical
[4] B. A. Jayawickrama, Y. He, E. Dutkiewicz, and M. D. Mueck, challenges and solutions,” IEEE Wireless Commun., vol. 23, no. 2,
“Scalable spectrum access system for massive machine type commu- pp. 126–135, April 2016.
nication,” IEEE Network, vol. PP, no. 99, pp. 1–7, 2018. [29] Z. Dawy, et al., “Toward massive machine type cellular communica-
[5] 3GPP, “Study on scenarios and requirements for next generation access tions,” IEEE Wireless Commun., vol. 24, no. 1, pp. 120–128, 2017.
technologies (Release 14),” Technical Report 38.913, 2017. [30] A. Rico-Alvarino, et al., “An overview of 3GPP enhancements on
[6] F. Ghavimi and H. H. Chen, “M2M communications in 3GPP machine to machine communications,” IEEE Commun. Magazine,
LTE/LTE-A networks: Architectures, service requirements, challenges, vol. 54, no. 6, pp. 14–21, June 2016.
and applications,” IEEE Commun. Surveys Tuts., vol. 17, no. 2, pp. [31] M. Elsaadany, A. Ali, and W. Hamouda, “Cellular LTE-A technologies
525–549, Secondquarter 2015. for the future internet-of-things: Physical layer features and chal-
[7] M. Koseoglu, “Performance analysis of small data transmission lenges,” IEEE Commun. Surveys Tuts., vol. PP, no. 99, pp. 1–1, 2017.
schemes for cellular M2M communications,” in Proc. 16th Annual
[32] A. Hoglund, et al., “Overview of 3GPP release 14 enhanced NB-IoT,”
Mediterranean Ad Hoc Networking Workshop, June 2017, pp. 1–6.
IEEE Network, vol. 31, no. 6, pp. 16–22, Nov. 2017.
[8] Z. Dawy, et al., “Toward massive machine type cellular communica-
[33] A. Laya, L. Alonso, and J. Alonso-Zarate, “Is the random access
tions,” IEEE Wireless Communications, vol. 24, no. 1, pp. 120–128,
channel of LTE and LTE-A suitable for M2M communications? a
Feb. 2017.
survey of alternatives,” IEEE Commun. Surveys Tuts., vol. 16, no. 1,
[9] 3GPP, “Service requirements for machine-type communications
pp. 4–16, Firstquarter 2014.
(MTC), TS 22.368, V11.5.0, June 2012.
[34] W. Yang, et al., “Narrowband wireless access for low-power massive
[10] L. M. Bello, P. Mitchell, and D. Grace, “Application of Q-learning
internet of things: A bandwidth perspective,” IEEE Wireless Commun.,
for RACH access to support M2M traffic over a cellular network,”
vol. 24, no. 3, pp. 138–145, 2017.
in European Wireless 2014; 20th European Wireless Conference, May
2014, pp. 1–6. [35] M. S. Ali, E. Hossain, and D. I. Kim, “LTE/LTE-A random access for
[11] G. Durisi, T. Koch, and P. Popovski, “Toward massive, ultrareliable, massive machine-type communications in smart cities,” IEEE Commun.
and low-latency wireless communication with short packets,” Proc. of Mag., vol. 55, no. 1, pp. 76–83, Jan. 2017.
the IEEE, vol. 104, no. 9, pp. 1711–1726, Sept. 2016. [36] E. Soltanmohammadi, K. Ghavami, and M. Naraghi-Pour, “A survey of
[12] S. Oh and J. W. Jang, “A scheme to smooth aggregated traffic from traffic issues in machine-to-machine communications over LTE,” IEEE
sensors with periodic reports,” MDPI Sensors, vol. 17, no. 503, pp. Internet of Things J., vol. 3, no. 6, pp. 865–884, Dec 2016.
1–18, March 2017. [37] A. G. Gotsis, A. S. Lioumpas, and A. Alexiou, “M2M scheduling
[13] S. K. Sharma and X. Wang, “Distributed caching enabled peak traffic over LTE: Challenges and new perspectives,” IEEE Veh. Technol. Mag.,
reduction in ultra-dense IoT networks,” IEEE Commun. Letters, vol. vol. 7, no. 3, pp. 34–39, Sept. 2012.
22, no. 6, pp. 1252-1255, June 2018. [38] M. A. Mehaseb, Y. Gadallah, A. Elhamy, and H. Elhennawy, “Classi-
[14] C. Bockelmann, et al., “Massive machine-type communications in 5G: fication of LTE uplink scheduling techniques: An M2M perspective,”
physical and MAC-layer solutions,” IEEE Commun. Mag., vol. 54, IEEE Commun. Surveys Tuts., vol. 18, no. 2, pp. 1310–1335, Sec-
no. 9, pp. 59–65, Sept. 2016. ondquarter 2016.
[15] A. Imran, A. Zoha, and A. Abu-Dayya, “Challenges in 5G: how to [39] M. Marjani, et al., “Big IoT data analytics: Architecture, opportunities,
empower SoN with big data for enabling 5G,” IEEE Network, vol. 28, and open research challenges,” IEEE Access, vol. 5, pp. 5247–5261,
no. 6, pp. 27–33, Nov 2014. 2017.
[16] R. Li, , “Intelligentet al., 5G: When cellular networks meet artificial [40] S. Verma, et al., “A survey on network methodologies for real-
intelligence,” IEEE Wireless Commun., vol. 24, no. 5, pp. 175–183, time analytics of massive IoT data and open research issues,” IEEE
Oct. 2017. Commun.. Surveys Tuts., vol. 19, no. 3, pp. 1457–1477, thirdquarter
[17] R. C. Daniels and R. W. Heath, “An online learning framework 2017.
for link adaptation in wireless networks,” in Proc. Info. Theory and [41] H. Elhammouti, et al., “Self-organized connected objects: Rethinking
Applications Workshop, Feb. 2009, pp. 138–140. QoS provisioning for IoT services,” IEEE Commun. Mag., vol. 55,
[18] S. Yun and C. Caramanis, “Reinforcement learning for link adaptation no. 9, pp. 41–47, 2017.
in MIMO-OFDM wireless systems,” in Proc. IEEE GLOBECOM , Dec. [42] M. Mohammadi and A. Al-Fuqaha, “Enabling cognitive smart cities
2010, pp. 1–5. using big data and machine learning: Approaches and challenges,”
[19] S. K. Sharma and X. Wang, “Live data analytics with collaborative IEEE Commun. Mag., vol. 56, no. 2, pp. 94–101, Feb. 2018.
edge and cloud processing in wireless IoT networks,” IEEE Access, [43] O. B. Sezer, E. Dogdu, and A. M. Ozbayoglu, “Context-aware com-
vol. 5, pp. 4621–4635, March 2017. puting, learning, and big data in internet of things: A survey,” IEEE
[20] X. Wang, X. Li, and V. C. M. Leung, “Artificial intelligence-based Internet of Things J., vol. 5, no. 1, pp. 1–27, Feb. 2018.
techniques for emerging heterogeneous network: State of the arts, [44] L. Busoniu, et al., “A comprehensive survey of multiagent reinforce-
opportunities, and challenges,” IEEE Access, vol. 3, pp. 1379–1391, ment learning,” IEEE Trans. Systems, Man, and Cybernetics, Part C
2015. (Applications and Reviews), vol. 38, no. 2, pp. 156–172, March 2008.
IEEE COMMUNICATIONS SURVEYS & TUTORIALS (DRAFT) 35

[45] K. Arulkumaran, M. P. Deisenroth, M. Brundage, and A. A. Bharath, [70] J. Choi, “On the adaptive determination of the number of preambles
“Deep reinforcement learning: A brief survey,” IEEE Signal Process. in RACH for MTC,” IEEE Commun. Letters, vol. 20, no. 7, pp. 1385–
Mag., vol. 34, no. 6, pp. 26–38, Nov 2017. 1388, July 2016.
[46] R. Baldemair, et al., “Ultra-dense networks in millimeter-wave frequen- [71] S. J. SungHyung Lee and J. Kim, “Dynamic resource allocation of
cies,” IEEE Commun. Mag., vol. 53, no. 1, pp. 202–208, Jan. 2015. random access for MTC devices,” ETRI Journal, vol. 39, no. 4, pp.
[47] S. Chen, et al., “Machine-to-machine communications in ultra-dense 546–557, Aug. 2017.
networks: A survey,” IEEE Commun. Surveys Tuts., vol. 19, no. 3, pp. [72] S. K. Sharma, et al., “Dynamic spectrum sharing in 5G wireless
1478–1503, thirdquarter 2017. networks with full-duplex technology: Recent advances and research
[48] M. Kamel, W. Hamouda, and A. Youssef, “Ultra-dense networks: A challenges,” IEEE Commun. Surveys Tuts., vol. 20, no. 1, pp. 674–707,
survey,” IEEE Commun. Surveys Tuts., vol. 18, no. 4, pp. 2522–2545, Firstquarter 2018.
Fourthquarter 2016. [73] M. Li, et al., “Random access and virtual resource allocation in
[49] K. Smiljkovic, V. Atanasovski, and L. Gavrilovska, “Machine-to- software-defined cellular networks with machine-to-machine commu-
machine traffic characterization: Models and case study on integration nications,” IEEE Trans. Veh. Technol., vol. 66, no. 7, pp. 6399–6414,
in LTE,” in Proc. 4th Int. Conf. Wireless Commun., Veh. Technol., Info. July 2017.
Th. and Aerospace Electronic Systems (VITAE), May 2014, pp. 1–5. [74] A. Samir, et al., “Partial contention free random access protocol for
[50] X. Li, J. B. Rao, and H. Zhang, “Engineering machine-to-machine M2M communications in LTE networks,” J. of Wireless Networking
traffic in 5G,” IEEE Internet of Things J., vol. 3, no. 4, pp. 609–618, and Commun., vol. 6, no. 3, pp. 66–72, 2016.
Aug 2016. [75] G. Wunder, C. Stefanovic, P. Popovski, and L. Thiele, “Compressive
[51] N. Afrin, J. Brown, and J. Y. Khan, “An adaptive buffer based semi- coded random access for massive MTC traffic in 5G systems,” in Proc.
persistent scheduling scheme for machine-to-machine communications 49th Asilomar Conf. on Signals, Systems and Computers, Nov. 2015,
over LTE,” in Proc. Eighth Int. Conf. Next Generation Mobile Apps, pp. 13–17.
Services and Technol., Sept 2014, pp. 260–265. [76] I. M. Delgado-Luque, et al., “Evaluation of latency-aware scheduling
[52] J. Kim, J. Lee, J. Kim, and J. Yun, “M2M service platforms: Survey, techniques for M2M traffic over LTE,” in Proc. 20th European Signal
issues, and enabling technologies,” IEEE Commun. Surveys Tuts., Processing Conference (EUSIPCO), Bucharest, 2012, pp. 989-993.
vol. 16, no. 1, pp. 61–76, Firstquarter 2014. [77] S. Ali, N. Rajatheva and W. Saad, “Fast Uplink Grant for Ma-
[53] J. Fabini and T. Zseby, “M2M communication delay challenges: chine Type Communications: Challenges and Opportunities”, CoRR,
Application and measurement perspectives,” in Proc. IEEE Int. Instru- abs/1801.04953, 2018, online: http://arxiv.org/abs/1801.04953.
mentation and Measurement Technol. Conf., May 2015, pp. 1859–1864. [78] S. Bhandari, S. K. Sharma and X. Wang, “Latency Minimization in
[54] S. Y. Lien and K. C. Chen, “Massive access management for QoS guar- Wireless IoT Using Prioritized Channel Access and Data Aggregation,”
antees in 3GPP machine-to-machine communications,” IEEE Commun. in Proc. IEEE Global Commun. Conf., Singapore, Dec. 2017, pp. 1-6.
Letters, vol. 15, no. 3, pp. 311–313, March 2011. [79] S. K. Sharma and X. Wang, “Cooperative sensing delay minimization
[55] Y. Saleem, N. Crespi, M. H. Rehmani and R. Copeland, “In- in cloud-assisted DSA networks” in Proc. IEEE 28th Annual Int. Symp.
ternet of Things-aided Smart Grid: Technologies, Architectures, on Personal, Indoor, and Mobile Radio Commun. (PIMRC), Montreal,
Applications, Prototypes, and Future Research Directions”, CoRR, QC, Oct. 2017, pp. 1-6.
abs/1704.08977,2017, online: http://arxiv.org/abs/1704.08977. [80] T. P. C. de Andrade, C. A. Astudillo and N. L. S. da Fonseca,
“Impact of M2M traffic on human-type communication users on the
[56] K. Zheng, et al., “Radio resource allocation in LTE-advanced cellular
LTE uplink channel,” in Proc. 7th IEEE Latin-American Conference
networks with M2M communications,” IEEE Commun. Mag., vol. 50,
on Communications (LATINCOM), Arequipa, 2015, pp. 1-6.
no. 7, pp. 184–192, July 2012.
[81] M. Laner, P. Svoboda, N. Nikaein, and M. Rupp, “Traffic models for
[57] M. Y. Cheng, et al., “Overload control for machine-type-
machine type communications,” in Proc. The Tenth Int. Symp. Wireless
communications in LTE-advanced system,” IEEE Commun. Mag.,
Commun. Systems, Aug. 2013, pp. 1–5.
vol. 50, no. 6, pp. 38–45, June 2012.
[82] N. Nikaein, et al., “Simple traffic modeling framework for machine
[58] R. Ratasuk, N. Mangalvedhe, D. Bhatoolaul, and A. Ghosh, “LTE-
type communication,” in Proc. The Tenth Int. Symp. Wireless Commun.
M evolution towards 5G massive MTC,” in Proc. IEEE Globecom
Systems, Aug 2013, pp. 1–5.
Workshops (GC Wkshps), Dec. 2017, pp. 1–6.
[83] A. M. AlAdwani, A. Gawanmeh, and S. Nicolas, “Traffic shaping and
[59] P. K. Wali and D. Das, “A novel access scheme for IoT communica- delay optimization in demand side management,” in Proc. 15th Int.
tions in LTE-advanced network,” in Proc. IEEE Int. Conf. Advanced Conf. on Computer Modelling and Simulation, April 2013, pp. 722–
Networks and Telecommun. Systems (ANTS), Dec 2014, pp. 1–6. 727.
[60] A. M. AlAdwani, A. Gawanmeh, and S. Nicolas, “A demand side [84] A. Kumar, A. Abdelhadi, and C. Clancy, “A delay efficient multiclass
management traffic shaping and scheduling algorithm,” in Proc. Sixth packet scheduler for heterogeneous M2M uplink,” in Proc. IEEE
Asia Modelling Symp., May 2012, pp. 205–210. MILCOM Conf., Nov 2016, pp. 313–318.
[61] S. Y. Lien, K. C. Chen, and Y. Lin, “Toward ubiquitous massive ac- [85] ——, “An online delay efficient packet scheduler for M2M traffic in
cesses in 3GPP machine-to-machine communications,” IEEE Commun. industrial automation,” in Proc. 2016 Annual IEEE Systems Conf., April
Mag., vol. 49, no. 4, pp. 66–74, April 2011. 2016, pp. 1–6.
[62] Z. Li, et al., “Enabling heterogeneous MMTC by energy-efficient and [86] Y. Zhao, B. Zhang, C. Li, and C. Chen, “On/off traffic shaping in the
connectivity-aware clustering and routing,” in Proc. IEEE Globecom internet: Motivation, challenges, and solutions,” IEEE Network, vol. 31,
Workshops (GC Wkshps), Dec. 2017, pp. 1–6. no. 2, pp. 48–57, March 2017.
[63] 3GPP, “Cellular system support for ultralow complexity and low [87] J. S. Kim, S. Lee, and M. Y. Chung, “Efficient random-access scheme
throughput Internet of Things (CIoT),”, TR 45.820, V13.1.0, Nov. for massive connectivity in 3GPP low-cost machine-type communica-
2015. tions,” IEEE Trans. Veh. Technol., vol. 66, no. 7, pp. 6280–6290, July
[64] P. Andres-Maldonado, et al., “Narrowband IoT data transmission 2017.
procedures for massive machine-type communications,” IEEE Network, [88] S. Duan, V. Shah-Mansouri, Z. Wang, and V. W. S. Wong, “D-ACB:
vol. 31, no. 6, pp. 8–15, Nov. 2017. Adaptive congestion control algorithm for bursty M2M traffic in LTE
[65] J. Huang, et al.,, “Optimizing M2M communications and quality of networks,” IEEE Trans. Veh. Technol., vol. 65, no. 12, pp. 9847–9861,
services in the IoT for sustainable smart cities,” IEEE Trans. on Dec. 2016.
Sustainable Computing, vol. PP, no. 99, pp. 1–1, 2017. [89] G. Y. Lin, S. R. Chang, and H. Y. Wei, “Estimation and adaptation for
[66] M. Centenaro, et al.,, “Comparison of collision-free and contention- bursty LTE random access,” IEEE Trans. Veh. Technol., vol. 65, no. 4,
based radio access protocols for the internet of things,” IEEE Trans. pp. 2560–2577, April 2016.
Commun., vol. 65, no. 9, pp. 3832–3846, Sept. 2017. [90] P. Osti, et al., “Analysis of PDCCH performance for M2M traffic in
[67] U. Devi, et al., “SERA: A scheduling framework for LTE,” IEEE Trans. Veh. Technol., vol. 63, no. 9, pp. 4357–4371, Nov.
M2M transmission in cellular networks,” online, web: 2014.
http://researcher.ibm.com/researcher/files/in-umamadev/M2M- [91] A. Biral, et al., “The challenges of M2M massive access in wireless
long.pdf. cellular networks,” Digital Commun. and Networks, vol. 1, no. 1, pp.
[68] Y. Medjahdi, et al., “On the road to 5G: Comparative study of physical 1–19, 2015.
layer in MTC context,” IEEE Access, vol. 5, pp. 26556–26581, 2017. [92] S. Y. Lien, et al., “Cooperative access class barring for machine-to-
[69] 3GPP, “Study on RAN improvements for machine-type communica- machine communications,” IEEE Trans. Wireless Commun., vol. 11,
tions,” Tech. Rep., TR 37.868, 2012. no. 1, pp. 27–32, Jan. 2012.
IEEE COMMUNICATIONS SURVEYS & TUTORIALS (DRAFT) 36

[93] S. Duan, V. Shah-Mansouri, and V. W. S. Wong, “Dynamic access [118] K. Balachandran, J. H. Kang, K. Karakayali, and K. M. Rege, “Delay-
class barring for M2M communications in LTE networks,” in Proc. tolerant autonomous transmissions for short packet communications,”
IEEE Global Commun. Conf., Dec. 2013, pp. 4747–4752. in Proc. IEEE Globecom Workshops (GC Wkshps), Dec. 2016, pp. 1–6.
[94] M. Koseoglu, “Lower bounds on the LTE-A average random access [119] B. Lee, et al., “Packet structure and receiver design for low latency
delay under massive M2M arrivals,” IEEE Trans. Commun., vol. 64, wireless communications with ultra-short packets,” IEEE Trans. Com-
no. 5, pp. 2104–2115, May 2016. mun., vol. 66, no. 2, pp. 796–807, Feb. 2018.
[95] DIGI, “What are the difference between LTE-M and NB-IoT Cellular [120] Q. Yang, et al., “Wireless powered asynchronous backscatter networks
Protocols,” available online: https://www.digi.com/, last accessed 15th with sporadic short packets: Performance analysis and optimization,”
March 2018. IEEE Internet of Things J., vol. 5, no. 2, pp. 984–997, April 2018.
[96] I. Lusky, “CAT-M1 vs NB-IoT-examining the real differences,” avail- [121] C. She, C. Yang, and T. Q. S. Quek, “Uplink transmission design
able online: https://www.iot-now.com/, last accessed 16th March 2018. with massive machine type devices in tactile internet,” in Proc. IEEE
[97] S. Andreev, et al., “Efficient small data access for machine-type Globecom Workshops (GC Wkshps), Dec. 2016, pp. 1–6.
communications in LTE,” in Proc. IEEE Int. Conf. Commun. (ICC), [122] A. H. E. Fawal, et al., “RACH overload congestion mechanism for
June 2013, pp. 3569–3574. M2M communication in LTE-A: Issues and approaches,” in Proc. Int.
[98] J. Hu, et al., “Enhanced LTE physical downlink control channel Symp. on Networks, Computers and Commun. (ISNCC), May 2017, pp.
design for machine-type communications,” in Proc. 7th Int. Conf. New 1–6.
Technologies, Mobility and Security (NTMS), July 2015, pp. 1–5. [123] H. Wu, et al., “FASA: Accelerated S-ALOHA using access history
[99] 3GPP, “General packet radio service (GPRS) enhancements for evolved for event-driven M2M communications,” IEEE/ACM Trans. Network.,
universal terrestrial radio access network (E-UTRAN) access,” TS vol. 21, no. 6, pp. 1904–1917, Dec. 2013.
23.401 V14.3.0, March 2017. [124] R. G. Cheng, et al., “Modeling and analysis of an extended access bar-
[100] C. Yu, et al., “Uplink scheduling and link adaptation for narrowband ring algorithm for machine-type communications in LTE-A networks,”
internet of things systems,” IEEE Access, vol. 5, pp. 1724–1734, 2017. IEEE Trans. on Wireless Commun., vol. 14, no. 6, pp. 2956–2968, June
[101] R. Boisguene, S. C. Tseng, C. W. Huang, and P. Lin, “A survey on 2015.
NB-IoT downlink scheduling: Issues and potential solutions,” in Proc. [125] J. P. Cheng, C. h. Lee, and T. M. Lin, “Prioritized random access with
13th Int. Wireless Commun. and Mobile Computing Conf. (IWCMC), dynamic access barring for RAN overload in 3GPP LTE-A networks,”
June 2017, pp. 547–551. in Proc. IEEE GLOBECOM Workshops (GC Wkshps), Dec. 2011, pp.
[102] Y. P. E. Wang, et al., “A primer on 3GPP narrowband internet of 368–372.
things,” IEEE Commun. Mag., vol. 55, no. 3, pp. 117–123, March [126] T. M. Lin, et al., “PRADA: Prioritized random access with dynamic
2017. access barring for MTC in 3GPP LTE-A networks,” IEEE Trans. Veh.
[103] D. Wu and R. Negi, “Effective capacity: a wireless link model for Technol., vol. 63, no. 5, pp. 2467–2472, June 2014.
support of quality of service,” IEEE Trans. Wireless Commun., vol. 2, [127] G. Farhadi and A. Ito, “Group-based signaling and access control for
no. 4, pp. 630–643, July 2003. cellular machine-to-machine communication,” in Proc. IEEE 78th Veh.
[104] F. Ghavimi, Y. W. Lu, and H. H. Chen, “Uplink scheduling and Technol. Conf., Sept. 2013, pp. 1–6.
power allocation for M2M communications in SC-FDMA-based LTE- [128] T. H. Chuang, M. H. Tsai, and C. Y. Chuang, “Group-based up-
A networks with QoS guarantees,” IEEE Trans. Veh. Technol., vol. 66, link scheduling for machine-type communications in LTE-advanced
no. 7, pp. 6160–6170, July 2017. networks,” in Proc. IEEE Int. Conf. Advanced Info. Networking and
[105] H. G. Myung, J. Lim, and D. J. Goodman, “Single carrier FDMA for Applications Workshops, March 2015, pp. 652–657.
uplink wireless transmission,” IEEE Veh. Technol. Mag., vol. 1, no. 3, [129] N. K. Pratas, H. Thomsen, C. Stefanovi, and P. Popovski, “Code-
pp. 30–38, Sept. 2006. expanded random access for machine-type communications,” in Proc.
[106] A. Aijaz, et al., “Energy-efficient uplink resource allocation in lte net- 2012 IEEE Globecom Workshops, Dec 2012, pp. 1681–1686.
works with M2M/H2H co-existence under statistical QoS guarantees,” [130] H. S. Jang, et al., “Spatial group based random access for M2M
IEEE Trans. Commun., vol. 62, no. 7, pp. 2353–2365, July 2014. communications,” IEEE Communications Letters, vol. 18, no. 6, pp.
[107] C. W. Park, D. Hwang, and T. J. Lee, “Enhancement of IEEE 802.11ah 961–964, June 2014.
MAC for M2M communications,” IEEE Commun. Letters, vol. 18, [131] H. M. Grsu, M. Vilgelm, W. Kellerer, and M. Reisslein, “Hybrid
no. 7, pp. 1151–1154, July 2014. collision avoidance-tree resolution for M2M random access,” IEEE
[108] N. Afrin, J. Brown, and J. Y. Khan, “Performance analysis of an Trans. Aerospace and Electronic Systems, vol. 53, no. 4, pp. 1974–
enhanced delay sensitive lte uplink scheduler for M2M traffic,” in Proc. 1987, Aug. 2017.
2013 Australasian Telecommun. Networks and Applications Conf., Nov. [132] G. C. Madueno, C. Stefanovic, and P. Popovski, “Efficient LTE access
2013, pp. 154–159. with collision resolution for massive M2M communications,” in Proc.
[109] E. Dahlman, et al., “5G wireless access: requirements and realization,” 2014 IEEE Globecom Workshops (GC Wkshps), Dec. 2014, pp. 1433–
IEEE Commun. Mag., vol. 52, no. 12, pp. 42–47, Dec. 2014. 1438.
[110] N. A. Johansson, Y. P. E. Wang, E. Eriksson, and M. Hessler, “Radio [133] A. J. E. M. Janssen and M. J. de Jong, “Analysis of contention tree
access for ultra-reliable and low-latency 5G communications,” in Proc. algorithms,” IEEE Trans. Info. Th., vol. 46, no. 6, pp. 2163–2172, Sept.
IEEE Int. Conf. Commun. Workshop (ICCW), June 2015, pp. 1184– 2000.
1189. [134] Y. Ruan, W. Wang, Z. Zhang, and V. K. N. Lau, “Delay-aware mas-
[111] G. Durisi, et al., “Short-packet communications over multiple-antenna sive random access for machine-type communications via hierarchical
rayleigh-fading channels,” IEEE Trans. Commun., vol. 64, no. 2, pp. stochastic learning,” in Proc. IEEE Int. Conf. Commun. (ICC), May
618–629, Feb. 2016. 2017, pp. 1–6.
[112] M. Mousaei and B. Smida, “Optimizing pilot overhead for ultra-reliable [135] L. Alonso, R. Ferrus, and R. Agusti, “WLAN throughput improvement
short-packet transmission,” in Proc. IEEE Int. Conf. Commun. (ICC), via distributed queuing MAC,” IEEE Commun. Letters, vol. 9, no. 4,
May 2017, pp. 1–5. pp. 310–312, April 2005.
[113] F. Schaich, T. Wild, and Y. Chen, “Waveform contenders for 5G- [136] E. Kartsakli, et al., “Cross-layer enhancement for WLAN systems with
suitability for short packet and low latency transmissions,” in Proc. heterogeneous traffic based on DQCA,” IEEE Commun. Mag., vol. 46,
IEEE 79th Veh. Technol. Conf. (VTC Spring), May 2014, pp. 1–5. no. 6, pp. 60–66, June 2008.
[114] D. S. Yoo, et al., “Coding and modulation for short packet transmis- [137] M. Li, et al., “Energy-efficient M2M communications with mobile edge
sion,” IEEE Trans. Veh. Technol., vol. 59, no. 4, pp. 2104–2109, May computing in virtualized cellular networks,” in Proc. IEEE Int. Conf.
2010. Commun. (ICC), May 2017, pp. 1–6.
[115] X. Sun, et al., “Short-packet downlink transmission with non- [138] L. Li, X. He, and H. Li, “Learning the spectrum using collaborative
orthogonal multiple access,” IEEE Trans. Wireless Commun., pp. 1–1, filtering in directional millimeter wave networks,” in Proc. IEEE
2018. GLOBECOM, Dec. 2017, pp. 1–7.
[116] D. Aziz, H. Bakker, A. Ambrosy, and Q. Liao, “Signalling minimiza- [139] V. Rakovic and L. Gavrilovska, “Novel RAT selection mechanism
tion framework for short data packet transmission in 5G,” in Proc. based on hopfield neural networks,” in Proc. Int. Congress on Ultra
IEEE 84th Veh. Technol. Conf. (VTC-Fall), Sept. 2016, pp. 1–6. Modern Telecommun. and Control Systems, Oct. 2010, pp. 210–217.
[117] K. F. Trillingsgaard and P. Popovski, “Downlink transmission of short [140] N. Kato, et al., “The deep learning vision for heterogeneous network
packets: Framing and control information revisited,” IEEE Trans. traffic control: Proposal, challenges, and future perspective,” IEEE
Commun., vol. 65, no. 5, pp. 2048–2061, May 2017. Wireless Commun., vol. 24, no. 3, pp. 146–153, June 2017.
IEEE COMMUNICATIONS SURVEYS & TUTORIALS (DRAFT) 37

[141] R. C. Daniels, C. Caramanis, and R. W. Heath, “A supervised learning [164] P. Y. Glorennec and L. Jouffe, “Fuzzy Q-learning,” in Proc. Int. Fuzzy
approach to adaptation in practical MIMO-OFDM wireless systems,” Systems Conf., vol. 2, Jul 1997, pp. 659–662.
in Proc. IEEE GLOBECOM, Nov. 2008, pp. 1–5. [165] P. Jamshidi, et al., “Self-learning cloud controllers: Fuzzy Q-learning
[142] T. Desai and H. Shah, “Energy efficient link adaptation using machine for knowledge evolution,” in Proc. Int. Conf. Cloud and Autonomic
learning techniques for wireless OFDM,” in Proc. Int. Conf. Inventive Computing, Sept 2015, pp. 208–211.
Computation Technologies (ICICT), vol. 3, Aug. 2016, pp. 1–4. [166] M. N. ul Islam and A. Mitschele-Thiel, “Cooperative fuzzy Q-learning
[143] S. K. Pulliyakode and S. Kalyani, “Reinforcement learning techniques for self-organized coverage and capacity optimization,” in Proc. IEEE
for outer loop link adaptation in 4G/5G systems,” ArXiv e-prints, 2017. 23rd Int. Symp. (PIMRC), Sept. 2012, pp. 1406–1411.
[144] M. Chen, et al., “Data-driven computing and caching in 5G networks: [167] Y. Maeda, “Modified Q-learning method with fuzzy state division and
Architecture and delay analysis,” IEEE Wireless Communications, adaptive rewards,” in Proc. IEEE Int. Conf. Fuzzy Systems, vol. 2, 2002,
vol. 25, no. 1, pp. 70–75, Feb. 2018. pp. 1556–1561.
[145] Y. W. Blankenship, et al., “Link error prediction methods for multicar- [168] F. Pervez, et al., “Fuzzy Q-learning-based user-centric backhaul-aware
rier systems,” in IEEE 60h Veh. Technol. Conf., vol. 6, Sept. 2004, pp. user cell association scheme,” in Proc. 13th Int. Wireless Commun. and
4175–4179. Mobile Computing Conf., June 2017, pp. 1840–1845.
[146] Y. Park, et al., “A new link adaptation method to mitigate sinr mis- [169] M. Mendil, et al., “Fuzzy Q-learning based energy management of
match in ultra-dense small cell LTE networks,” IEEE Trans. Wireless small cells powered by the smart grid,” in Proc. IEEE 27th Annual Int.
Commun., vol. PP, no. 99, pp. 1–1, 2017. Symp. PIMRC, Sept. 2016, pp. 1–6.
[147] T. Park and W. Saad, “Learning with finite memory for machine type [170] J. Wu, J. Liu, Z. Huang, and S. Zheng, “Dynamic fuzzy Q-learning for
communication,” in Proc. Annual Conf. on Information Science and handover parameters optimization in 5G multi-tier networks,” in Proc.
Systems (CISS), March 2016, pp. 608–613. Int. Conf. Wireless Commun. Signal Process. (WCSP), Oct 2015, pp.
[148] M. A. Alsheikh, S. Lin, D. Niyato, and H. P. Tan, “Machine learn- 1–5.
ing in wireless sensor networks: Algorithms, strategies, and applica- [171] C. Wu, S. Ohzahata, and T. Kato, “Flexible, portable, and practicable
tions,” IEEE Commun. Surveys Tuts., vol. 16, no. 4, pp. 1996–2018, solution for routing in vanets: A fuzzy constraint Q-learning approach,”
Fourthquarter 2014. IEEE Trans. Veh. Technol., vol. 62, no. 9, pp. 4251–4263, Nov. 2013.
[149] R. van Loon, “Machine Learning Explained: Understanding [172] Z. Deng, et al., “Combining model-based Q-learning with structural
Supervised, Unsupervised, and Reinforcement Learning,” knowledge transfer for robot skill learning,” IEEE Trans. Cognitive
https://www.datasciencecentral.com/profiles/blogs/machine-learning- and Developmental Systems, pp. 1–1, 2017.
explained-understanding-supervised-unsupervised, online; accessed [173] A. Sharma, et al., “Model based path planning using Q-learning,” in
5th March 2018. Proc. IEEE Int. Conf. Industrial Technology (ICIT), March 2017, pp.
[150] J. Yoo and K. H. Johansson, “Semi-supervised learning for mobile 837–842.
robot localization using wireless signal strengths,” in Proc. Int. Conf. [174] E. S. El-Alfy, Y.-D. Yao, and H. Heffes, “A model-based Q-learning
on Indoor Positioning and Indoor Navigation (IPIN), Sept 2017, pp. scheme for wireless channel allocation with prioritized handoff,” in
1–8. Proc. IEEE GLOBECOM, vol. 6, 2001, pp. 3668–3672.
[151] K. Shah and M. Kumar, “Distributed independent reinforcement learn- [175] J. Cao, M. Ma and H. Li, “A group-based authentication and key
ing (DIRL) approach to resource management in wireless sensor agreement for MTC in LTE networks,” in Proc. IEEE Global Commu-
networks,” in IEEE Int. Conf. Mobile Adhoc and Sensor Systems, Oct. nications Conference (GLOBECOM), Anaheim, CA, 2012, pp. 1017-
2007, pp. 1–9. 1022.
[152] Y. Chu, et al., “Application of reinforcement learning to medium access [176] S. Bhandari, S. K. Sharma and X. Wang, “Device Grouping for Fast
control for wireless sensor networks,” Engineering Applications of and Efficient Channel Access in IEEE 802.11ah based IoT Networks,”
Artificial Intelligence, pp. 23–32, Nov. 2015. in Proc. IEEE Int. Conf. Commun. (ICC) Workshops, June 2018.
[153] A. D. Tijsma, M. M. Drugan, and M. A. Wiering, “Comparing [177] H. Li, K. Ota, and M. Dong, “Learning IoT in edge: Deep learning for
exploration strategies for Q-learning in random stochastic mazes,” in the internet of things with edge computing,” IEEE Network, vol. 32,
IEEE Symp. Series on Computational Intelligence (SSCI), Dec. 2016, no. 1, pp. 96–101, Jan. 2018.
pp. 1–8. [178] M. Song, et al., “In-Situ AI: Towards Autonomous and Incremental
[154] D. L. Pooole and Alan K. Mackworth, “Artificial Intelligence: Foun- Deep Learning for IoT Systems,” in 2018 Proc. IEEE Int. Symp. High
dations of Computational Agents,” in Cambridge University Press, Performance Computer Architecture (HPCA), Vienna, Feb. 2018, pp.
Second Edition, 2017. 92-103.
[155] C. g. Li, et al., “Multi-intersections traffic signal intelligent control [179] S. Venkataramani, K. Roy, and A. Raghunathan, “Efficient embedded
using collaborative Q-learning algorithm,” in Proc. Seventh Int. Conf. learning for IoT devices,” in Proc. 21st Asia and South Pacific Design
on Natural Computation, vol. 1, July 2011, pp. 185–188. Automation Conference (ASP-DAC), Jan. 2016, pp. 308–311.
[156] A. R. Rosyadi, T. A. B. Wirayuda, and S. Al-Faraby, “Intelligent traffic
light control using collaborative Q-learning algorithms,” in Proc. 4th
Int. Conf. Info. and Commun. Technol., May 2016, pp. 1–6.
[157] C. Lamini, Y. Fathi, and S. Benhlima, “Collaborative Q-learning
path planning for autonomous robots based on holonic multi-agent
system,” in Proc. 10th Int. Conf. on Intelligent Systems: Theories and
Applications, Oct. 2015, pp. 1–6.
[158] S. Kar, J. M. F. Moura, and H. V. Poor, “QD-learning: A collaborative
distributed strategy for multi-agent reinforcement learning through
consensus+innovations,” IEEE Trans. Signal Process., vol. 61, no. 7,
pp. 1848–1862, April 2013.
[159] J. Ni, M. Liu, L. Ren, and S. X. Yang, “A multiagent Q-learning-based
optimal allocation approach for urban water resource management
system,” IEEE Trans. Automation Science and Engg., vol. 11, no. 1,
pp. 204–214, Jan. 2014.
[160] J. Hu and M. P. Wellman, “NASH Q-learning for general-sum stochas-
tic games,” J. Mach. Learning Res., vol. 4, no. 6, pp. 1039-1069, 2003.
[161] S. Choi, et al., “Automatic configuration of random access channel
parameters in LTE systems,” in Proc. IFIP Wireless Days (WD), Oct.
2011, pp. 1–6.
[162] M. Rahimiyan and H. R. Mashhadi, “An adaptive Q-learning algorithm
developed for agent-based computational modeling of electricity mar-
ket,” IEEE Trans. Systems, Man, and Cybernetics, Part C (Applications
and Reviews), vol. 40, no. 5, pp. 547–556, Sept. 2010.
[163] K.-S. Hwang, S.-W. Tan, and C.-C. Chen, “Cooperative strategy based
on adaptive Q-learning for robot soccer systems,” IEEE Trans. Fuzzy
Systems, vol. 12, no. 4, pp. 569–576, Aug. 2004.

View publication stats