A Survey on Data Mapping Strategy for data stored in the storage cloud 111

VOLUME 9,ISSUE 3,MAY 2016 ISSN-2347-8047
INTERNATIONAL JOURNAL OF COGNITIVE SCIENCE,
ENGINEERING AND TECHNOLOGY
Available online at: http://airnetjournal.org/
A Survey on Data Mapping Strategy for data
stored in the storage cloud
Navneet Kumar
Department Of cse
K S Institute Of
Technology
Bengaluru,India
Naseeruddin V N
Department Of cse
K S Institute Of
Technology
Bengaluru,India
Murali Krishna V
Department Of cse
K S Institute Of
Technology
Bengaluru,India
S K Manu
Department Of cse
K S Institute Of
Technology
Bengaluru,India
Swathi K
Department Of cse
K S Institute Of
Technology
Bengaluru,India
Abstract— In the recent past the data being processed over the
internet is increasing exponentially so it’s difficult to store such
huge amount of data and It becomes computationally inefficient
to analyze such huge data. There is currently considerable
enthusiasm around the Map Reduce paradigm for large-scale
data analysis. It is inspired by functional programming which
allows expressing distributed computation massive amounts of
data. It is designed for large-scale data processing as it allows to
run on clusters of commodity hardware. A prominent parallel
data processing tool Map Reduce is gaining significant
momentum from both industry and academia as the volume of
data to analyze grows rapidly. In this paper we propose a method
to process huge amount of data over the internet. This method
involves storing the data to be processed on the cloud and
processing the data on hadoop multicluster environment.
Keywords— Storage Cloud, Hadoop cluster, Hadoop,
Distributed File System, Parallel Processing, MapReduce
I.Introduction
The very challenging problem is to analyze big data. For the
effective handling of such massive data or applications, the
use of MapReduce framework has been widely came into
focus. Over the last few years, MapReduce has emerged as the
most popular computing paradigm for parallel, batch-style and
analysis of large amount of data. Many areas where massive
data analysis is required, MapReduce is used. There are
evolving numbers of applications that handle big data but to
handle such huge collection of data is a very
challengingproblem today. Here, we got the MapReduce or its
opensource equivalent Hadoop which is a powerful tool for
building such applications. Data-intensive processing is fast
and currently becoming a necessity to handle the large
databases efficiently. It is required to design algorithms that
must be capable of scaling to real-world datasets. There is
currently considerable enthusiasm around the MapReduce
paradigm for large-scale data analysis. It is inspired by
functional programming which allows expressing distributed
computations on massive amounts of data. It is designed for
large-scale data processing as it allows running on clusters of
commodity hardware. MapReduce is used in the areas where
the volume of data to analyze grows speedily. Though, it
comprise of such abilities, still there are argument on its
concert, effectiveness, and simple concept. At the present time
there is outburst of data, so to process such a massive volume
of data in a timely manner, parallel processing is important.
MapReduce gained its popularity when used successfully by
Google. In real, it is a scalable and fault-tolerant data
processing tool which provides the ability to process huge
voluminous data in parallel with many low-end computing
nodes. By virtue of its simplicity, scalability, and fault
tolerance, MapReduce is becoming ubiquitous, gaining
significant momentum from both industry and academia.
However, MapReduce has inherent limitations on its
performance and efficiency. Therefore, many studies have
endeavoured to overcome the limitations of the MapReduce
framework. The goal of this analysis is to provide a timely
remark on the status of MapReduce studies and related work
focusing on the current research aimed at improving and
enhancing the MapReduce framework. This paper is brought
into consideration to assist the database in understanding
various technical aspects of the MapReduce framework. In
this paper, we focus on the working of MapReduce framework

and examine its in-built advantages and drawbacks. We then
introduce application and effective ways to improve its
properties so that we can get the optimized result. We also
brought into focus the issues and challenges raised on
MapReduce. It is well known for its simplicity, effectiveness
and capability to handle “Big Data” in a timely manner. With
all these valuable features still it consist of some limitations
which is required to be sorted out.
II.Working
MapReduce is a programming model and an associated
implementation for processing and generating large datasets
that is amenable to a broad variety of real-world tasks [3]. The
MapReduce paradigm of parallel programming provides
simplicity, while at the same time offering load balancing and
fault tolerance The Google File System (GFS) that typically
underlies a MapReduce system provides the efficient and
reliable distributed data storage needed for applications
involving large databases [10]. MapReduce is inspired by the
map and reduces primitives present in functional languages. In
its pure form, various implementations of the MapReduce
interface are possible, depending on the desired context. Some
currently available implementations are: shared-memory
multi-core system [11][12], asymmetric multi-core
processors[13], graphic processors, and cluster of networked
machines[4]. The most popular implementation is probably
the one introduced by Google, which utilizes large clusters of
commodity computers connected with switched Ethernet. In
essence, the Google’s MapReduce technique simplifies the
development and lowers the cost of large-scale distributed
applications on clusters of commodity machines. MapReduce
framework executes its tasks based on runtime scheduling
scheme. It means that MapReduce does not build any
execution plan that specifies which tasks will run on which
nodes before execution [14]. The MapReduce model is
capable of parallelly processing large data sets distributed
across many nodes. The main goal is to simplify large data
processing by using inexpensive cluster computers and to
make this easy for users while achieving both load balancing
and fault tolerance. Map-Reduce have two primary functions:
the Map function and the Reduce function. These functions are
defined by the user to meet the specific requirements. The
original Map-Reduce software is a proprietary system of
Google, and therefore, not available for public use [15].
Although the distributed computing is largely simplified with
the notions of Map and Reduce primitives, the underlying
infrastructure is non-trivial in order to achieve the desired
performance [2]. A key infrastructure in Google’s MapReduce
is the underlying distributed file system to ensure data locality
and availability [3]. Combining the MapReduce programming
technique and an efficient distributed file system, one can
easily achieve the goal of distributed computing with data
parallelism over thousands of computing nodes.
III.Methodlogy
The Architecture above illustrates the layout of the project.
User uploads the data to the cloud over the internet, then
selects the operation to be carried out. the controller present as
a middleware interprets the request and forwards the request to
hadoop master. hadoop master starts the jobtracker and
connects the cloud as the data node and the mapreduce
algorithm is run, which maps the data and reduces according
to the algorithm implempted. the result is collected and
concated and stored back onto the cloud for the use r to
download the result.
The use of a storage cloud allows the user to upload the data
and download the data from places connected to the internet
without known where the actual processing is done.
IV.Design
The goals of application is to provide an easy to use interface
so that a user with even little knowledge about using website
can use the browser . We have designed few models and
structures to explain the design and structure of the application
under discussion.
Data Flow Model
A data flow diagram (DFD) is a graphical representation of
the "flow" of data through an information system, modeling
its process aspects. Often they are a preliminary step used to
create an overview of the system which can later be
elaborated. DFDs can also be used for the visualization of
data processing (structured design).
A DFD shows what kinds of information will be input to and
output from the system, where the data will come from and
go to, and where the data will be stored. It does not show
information about the timing of processes, or information
about whether processes will operate in sequence or in
parallel.
Data flow: A data flow shows the flow of information from

its source to its destination. A data flow is represented by a
line, with arrowheads showing the direction of flow.
Data Store: A data store is a holding place for information
within the system. It is represented by an open-ended narrow
rectangle.
External Entities: It is normal for all information represented
within a system to have been obtained from and/or to be
passed on to external source recipient.
Processes: When naming processes, avoid glossing over
them, without really understanding their role. It is descriptive
title area – like ‘process’ or ‘update’.
Data Flows: Double-headed arrows can be used on all but
bottom-level diagrams. Furthermore, in common with most of
the other symbol used, a data flow at a particular level of
diagram may be decomposed to multiple data flows.
V.Snapshots
Description: Admin logins using admin as Admin name and
Password. If admin name or password not matches it will
display message as Wrong admin or wrong password, If
matches it will display message as Login Success.
Description: Website contains options for user, through

which he can do the necessary task.
Description: User can select the containers and can upload the
data to the cloud.
Description: User can select the output container where the
result is stored and can download the data from the cloud.
Acknowledgment
The satisfaction and euphoria that accompany the successful
completion of any task will be incomplete without the mention
of the individuals, we are greatly indebted to, who through
guidance and providing facilities have served as a beacon of
light and crowned our efforts with success .we are thankful to
Mrs. Swathi K , Assistant Professor, CSE,KSIT for being our
Project Guide, under whose able guidance this project work
has been carried out and completed successfully.
We thank the management, principal, Department of computer
science and engineering, KSIT. We thank VGST(Vision
Group on Science and Technology) Government of Karnataka,
India for providing infrastructure facilities through the K-FIST
Level II project at KSIT,CSE R&D Department Bengaluru.
References
[1] Maitrey S, Jha. An Integrated Approach for CURE Clustering using Map-
Reduce Technique. In Proceedings of Elsevier, ISBN 978-81- 910691-6-3,2nd
August 2013.
[2] Kyuseok Shim. MapReduce Algorithms for Big Data Analysis. In
Proceedings of the VLDB Endowment, Vol. 5, No. 12, August 27th 2012,
Istanbul, Turkey.
[3]Jeffrey Dean et al. Mapreduce: Simplified data processing on large
clusters. In Proceedings of the 6th USENIX OSDI, pages 137–150, 2004.
[4] J. Dean et al. MapReduce: Simplified data processing on large clusters.
Communications of the ACM, 51(1):107– 113, 2008.
[5] D. DeWitt and M. Stonebraker. MapReduce: A major step backwards. The
Database Column, 1, 2008.
[6] A. Pavlo et al. A comparison of approaches to large-scale data analysis. In
Proceedings of the ACM SIGMOD, pages 165– 178, 2009.
[7] M. Stonebraker et al. MapReduce and parallel DBMSs: friends or foes?
Communications of the ACM, 53(1):64–71, 2010.
[8] A. Thusoo et al. Hive: a warehousing solution over a mapreduce
framework. Proceedings of the VLDB Endowment, (2):1626–1629, 2009.
[9] A.F. Gates et al. Building a high-level dataflow system on top of Map-
Reduce: the Pig experience. Proceedings of the VLDB Endowment,
2(2):1414–1425, 2009.
[10] S. Ghemawat et al. The google file system. ACM SIGOPS Operating
Systems Review, 37(5):29–43, 2003.
[11]OpenStack Installation Guide for Ubuntu 14.04 ,February 26, 2015.
[12]http://www.stackoverflow.com/
[13]https://github.com/

A Survey on Data Mapping Strategy for data stored in the storage cloud 111

Recommended

More Related Content

What's hot (16)

Viewers also liked (12)

Similar to A Survey on Data Mapping Strategy for data stored in the storage cloud 111 (20)

A Survey on Data Mapping Strategy for data stored in the storage cloud 111