IOT DATA AND BIG DATA

INTRODUCTION
The development of big data and Internet of Things (IoT) is accelerating and affecting all areas of technology and
business by increasing the benefits of organizations and individuals. The growth of IoT-generated data has played
a major role in big data processing. Big data can be divided into three categories: (a) volume, (b) variety, and (c)
velocity.
These categories were first introduced by Gartner to explain the elements of major data challenges.
A) Variety
The data generated is not a single category as it includes not only traditional data but also structured data from
various resources such as Web Pages, Web Log Files, social media sites, e-mail, documents, data for both sensory
devices from active devices
B)Volume
The word Big in Big data itself means volume. Currently, existing data is in petabytes and should increase to
zettabytes in the near future. Existing social networking sites generate data in terabytes on a daily basis and this
amount of data is extremely difficult to manage using existing traditional systems.
C) Velocity
Velocity in Big data is a concept which deals with the speed of the data coming from various sources. This
characteristic is not being limited to the speed of incoming data but also the speed at which the data flows.

BIG DATA VS SMALL DATA
•Big data is more real-time in nature than traditional applications
•Big data architecture
•Traditional architectures are not well-suited for big data applications (e.g. Exa-data,Tera- data)
•Massively parallel processing, scale out architectures are well-suited for big data applications

OVERIVEW
A. IOT
 IoT provides a platform for sensors and devices to communicate freely within a smart environment and enables the
distribution of information across all platforms in an easy way.
 The latest innovation of different wireless technologies sets IoT as the next technology to benefit by taking full
advantage of the opportunities offered by Internet technology.
 IoT has seen its recent discovery in smart cities with an interest in developing smart programs, such as smart office,
smart marketing, smart agriculture, smart water, smart transport, smart health care, and smart energy.
B. BIG DATA
 The volume of data generated by sensors, devices, social media, health care applications, temperature sensors, and
various other software applications and digital devices that continuously generate large amounts of structured,
unstructured, or semi-structured data is strongly increasing.
 This massive data generation results in ―big data. Traditional database systems are inefficient when storing,
processing, and analysing rapidly growing amount of data or big data.
 The term ―big data‖ has been used in the previous literature but is relatively new in business and IT.An example of
big data-related studies is the next frontier for innovation, competition, and productivity; McKinsey Global Institute
defined big data as the size of data sets that are a better database system tool than the usual tools for capturing,
storing, processing, and analysing such data.
 This previous study also characterizes big data into three aspects: (a) data sources, (b) data analytics, and (c) the
presentation of the results of the analytics.
 This definition uses the 3V‘s (volume, variety, velocity) model proposed by Gartner. The model highlights an e-
commerce trend in data management that faces challenges to manage volume or size of data, variety or different
sources of data, and velocity or speed of data creation. Some studies declare volume as a main characteristic of big
data without providing a pure definition.
 However, other researchers introduced additional characteristics for big data, such as veracity, value, variability, and
complexity.The 3V‘s model, or its derivations, is the most common descriptions of the term ―big data.

BIG DATA ANALYTICS
 Big Data Analytics is the process of examining large data sets that contain a variety of data types to
reveal unseen patterns, hidden correlations, market trends, customer preferences, and other useful
business information.
 The capability to analyze large amounts of data can help an organization deal with considerable
information that can affect the business. Therefore, the main objective of big data analytics is to
assist business associations to have improved understanding of data, and thus, make efficient and
well-informed decisions. Big data analytics enables data miners and scientists to analyze a large
volume of data that may not be harnessed using traditional tools
EXITING ANALYTICS SYSTEMS
Different analytic types are used according to the requirements of IoT applications. These analytic types
are:
1.Real-time analytics is typically performed on data collected from sensors. In this situation, data
change constantly, and rapid data analytics techniques are required to obtain an analytical result within
a short period
2.Off-line analytics is used when a quick response is not required [32]. For example, many Internet
enterprises use Hadoop-based off-line analytics architecture to reduce the cost of data format
conversion.
3.Memory-level analytics is applied when the size of data is smaller than the memory of a cluster [32].
To date, the memory of clusters has reached terabyte (TB) level. Memory-level analytics is suitable for
conducting real-time analysis. MongoDB is an example of this architecture
4. BI analytics is adopted when the size of data is larger than the memory level, but in this case, data
may be imported to the BI analysis environment . BI analytic currently supports TB-level data.
5.Massive analytics is applied when the size of data is greater than the entire capacity of the BI
analysis product and traditional databases. Massive analytics uses the Hadoop distributed file system
for data storage and map/reduce for data analysis

Big Data Analytics is rapidly emerging as a key IoT initiative to improve decision making. One of the most
prominent features of IoT is its analysis of information about ―connected things.
Big data analytics in IoT requires processing a large amount of data on the fly and storing the data in various
storage technologies.
Given that much of the unstructured data are gathered directly from web-enabled ―things, big data
implementations will necessitate performing lightning-fast analytics with large queries to allow organizations
to gain rapid insights, make quick decisions, and interact with people and other devices.
The interconnection of sensing and actuating devices provide the capability to share information across
platforms through a unified architecture and develop a common operating picture for enabling innovative
applications.
RELATIONSHIP BETWEEN IOT AND BIG DATA ANALYTICS

IOT ARCHITECTURE FOR BIG DATA ANALTICS
 The architectural concept of IoT has several definitions based on IoT domain abstraction and identification.
 It offers a reference model that defines relationships among various IoT verticals, such as, smart traffic, smart home,
smart transportation, and smart health.
 The architecture for big data analytics offers a design for data abstraction.
• In this figure, the sensor layer contains all the sensor devices
and the objects, which are connected through a wireless
network.
• This wireless network communication can be RFID,WiFi, ultra-
wideband, ZigBee, and Bluetooth.
• The IoT gateway allows communication of the Internet and
various webs. The upper layer concerns big data analytics,
where a large amount of data received from sensors are
stored in the cloud and accessed through big data analytics
applications.
• These applications contain API management and a dashboard
to help in the interaction with the processing engine.

USE CASES
A) Smart metering
Smart metering is one of the IoT application use cases that generates a large amount of data from different sources. A smart meter is a
device that electronically records consumption of electric energy data between the meter and the control system. Collecting and
analyzing smart meter data in IoT environment assist the decision maker in predicting electricity consumption. Furthermore, the analytics
of a smart meter can also be used to forecast demands to prevent crises and satisfy strategic objectives through specific pricing plans.
Thus, utility companies must be capable of high-volume data management and advanced analytics designed to transform data into
actionable insights.
B) Smart agriculture
Smart agriculture is a beneficial use case in big IoT data analytics. Sensors are the actors in
the smart agriculture use case. They are installed in fields to obtain data on moisture level of
soil, trunk diameter of plants, microclimate condition, and humidity level, as well as to
forecast weather. Sensors transmit obtained data using network and communication devices.
The analytics layer processes the data obtained from the sensor network to issue commands.
Automatic climate control according to harvesting requirements, timely and controlled
irrigation, and humidity control for fungus prevention are examples of actions performed
based on big data analytics recommendations.
C) Smart transportation
A smart transportation system is an IoT-based use case that aims to support the smart city
concept. A smart transportation system intends to deploy powerful and advanced
communication technologies for the management of smart cities. Traditional transportation
systems, which are based on image processing, are affected by weather conditions, such as
heavy rains and thick fog. Consequently, the captured image may not be clearly visible.The
design of an e-plate system using RFID technology provides a good solution for intelligent
monitoring, tracking, and identification of vehicles.

OPPORTUNITIES
IoT is currently considered one of the most profound transitions in technology. Current IoT provides several data analytics opportunities
for big data analytics. Some opportunities are discussed below.
A) E-commerce
Big IoT data analytics offers well-designed tools to process real-time big data, which produce timely results for decision making. Big IoT
data exhibit heterogeneity, increasing volume, and real-time data processing features. The convergence of big data with IoT brings new
challenges and opportunities to build a smart environment. Big IoT data analytics has widespread applications in nearly every industry.
B) Smart cities
Big data collected from smart cities offer new opportunities in which efficiency gains can be achieved through an appropriate analytics
platform/infrastructure to analyze big IoT data.Various devices connect to the Internet in a smart environment and share information.
Moreover, the cost of storing data has been reduced dramatically after the invention of cloud computing technology. Analysis capabilities
have made huge leaps. Thus, the role of big data in a smart city can potentially transform every sector of the economy of a nation.
C) Healthcare
Recent years have witnessed tremendous growth in smart health monitoring devices. These devices generate enormous amounts of data.
Thus, applying data analytics to data collected from fetal monitors, electrocardiograms, temperature monitors, or blood glucose level
monitors can help healthcare specialists efficiently assess the physical conditions of patients. Moreover, data analytics enables healthcare
professionals to diagnose serious diseases in their early stages to help save lives.

OPEN CHALLENGES AND FUTURE DIRECTIONS
IoT and big data analytics have been extensively accepted by many organizations. However, these technologies are still in their early
stages. Several existing research challenges have not yet been addressed. Here are some challeneges in the filed of IoT and big data.
A. Privacy
Securing these huge sets of data is one of the daunting challenges of Big Data. Often companies are so busy in understanding, storing
and analyzing their data sets that they push data security for later stages. But, this is not a smart move as unprotected data repositories
can become breeding grounds for malicious hackers.
B. Data growth issues
One of the most pressing challenges of Big Data is storing all these huge sets of data properly. The amount of data being stored in data
centers and databases of companies is increasing rapidly. As these data sets grow exponentially with time, it gets extremely difficult to
handle. Most of the data is unstructured and comes from documents, videos, audios, text files and other sources. This means that you
cannot find them in databases.
In order to handle these large data sets, companies are opting for modern techniques, such as compression, tiering, and deduplication.
Compression is used for reducing the number of bits in the data, thus reducing its overall size. Deduplication is the process of removing
duplicate and unwanted data from a data set. Companies are also opting for Big Data tools, such as Hadoop, NoSQL and other
technologies.
C. Visualization
Visualization is an important entity in big data analytics, particularly when dealing with IoT systems where data are generated
enormously. But visualization can be a challenging task in the case of heterogeneous and diverse data. Designing visualization solution
that is compatible with advanced big data indexing frameworks is a difficult task. Similarly, response time is a desirable factor in big IoT
data analytics
D. Integrating data from a variety of sources
Data in an organization comes from a variety of sources, such as social media pages, ERP applications, customer logs, financial reports,
e-mails, presentations and reports created by employees. Combining all this data to prepare reports is a challenging task.
This is an area often neglected by firms. But, data integration is crucial for analysis, reporting and business intelligence, so it has to be
perfect.

CONCLUSION
The growth rate of data production has increased significantly over the years with the rise of smart and
sensor devices. The interaction between IoT and big data is currently in a phase where it is necessary to
process, convert and analyse big data at high frequency. We conducted this survey in the form of large
amounts of IoT data. First, we reviewed the latest mathematical solutions. The relationship between big data
analytics and IoT was also discussed. In addition, we have proposed the creation of large numbers of IoT data.
In addition, a wide variety of data mining methods, methods, and technology for large data mining have been
introduced. Other reliable use cases are also provided. In addition, we explored the domain by discussing
the various possibilities presented by data analytics in the IoT paradigm. Several open-ended research issues
were discussed as future indicators for research. Finally, we conclude that major IoT data solutions are in the
early stages of development. In the future, a real-time analytics solution will be needed that can provide quick
information.

THANK YOU
Divyansh Srivastava
Student-VIT, Vellore
divyanshsrivastava115@gmail.com

IOT DATA AND BIG DATA

Recommended

More Related Content

What's hot (20)

Similar to IOT DATA AND BIG DATA (20)

Recently uploaded (20)

IOT DATA AND BIG DATA