Big Data Analytics Module-4 power point presentation vtu

T.Godhandaraman, HKBK College of Engineering ,Bangalore - 560045
Module-4
MapReduce, Hive and Pig

What you get?
LO 4.1 Get understanding of MapReduce, map tasks using the key-value store, grouping by keys, reduce tasks
using combiners, and coping with node failures
LO 4.2 Get knowledge of composing MapReduce programs for calculations, such as counting, summing, and
algorithms for relational algebraic operations, projections, unions, intersections, natural joins, grouping,
aggregation operations and the matrix multiplication
LO 4.3 Get understanding of Hive, architecture, installation, comparison of Hive data store with traditional
databases
LO 4.4 Apply HiveQL for querying, sorting, aggregating, querying scripts, MapReduce Joins and sub-queries
LO 4.5 Get knowledge of Pig architecture, Grunt shell commands, data model, Pig Latin, developing scripts and
extensibility using UDFs.

Hadoop is a software framework for writing distributed applications.
Hadoop processes Big Data (multi-terabyte datasets) in parallel and in a reliable and fault-tolerant
way.
MapReduce functions are an integral part of the Hadoop physical organization.
MapReduce is a programming model for the distributed computing environment.
Applications using MapReduce v2, process huge amounts of data, in parallel, on a large number of
data nodes reliably.
The major ecosystem components of Handoop, are AVRO, Zookeeper, Ambari, HBase, Hive, Pig and
Mahout.
This chapter focuses on details of MapReduce, Hive and Pig programming and their use in Big Data
applications.

4.1.1 INTRODUCTION
Big Data architecture design layers

Data Storage Layer:
Hadoop file system [HDFS] provides the distributed data storage facility.
The data processing layer:
is the application support layer, while the application layer is the data consumption layer in Big-
Data architecture design.
Big Data processing layer includes the APIs of Programs such as MapReduce and Spark.
Data processing layer includes HBase which creates column-family data store using other formats
such as key-value pairs or JSON file.
HBase stores and processes the columnar data after translating into MapReduce tasks to run in
HDFS.

Data processing layer also includes Hive which creates SQL-like tables.
Hive stores and processes table data after translating it into MapReduce tasks to run in HDFS.
Hive creates SQL-like tables in Hive shell.
Hive uses HiveQL processes queries, ad hoc (unstructured) queries, aggregation functions and
summarizing functions, such as functions to compute maximum, minimum, average of selected or
grouped datasets.
HiveQL is a restricted form of SQL.
The support layer also includes Pig.
Pig is a data-flow language and an execution framework.
Pig enables the usage of relational algebra in HDFS.
MapReduce is the processing framework and YARN is the resource managing framework.

Key Terms
MapReduce programming model refers to a programming paradigm for processing Big Data sets
with a parallel and distributed environment using map and reduce tasks.
YARN refers to provisioning of running and scheduling parallel programs for map and reduce
tasks and allocating parallel processing resources for computing sub-tasks running in parallel at the
Hadoop for a user application.
Script refers to a small program (codes up to few thousand lines of code) used for purposes such as
query processing, text processing, or refers to a small code written in a dynamic high-level general-
purpose language, such as Python or PERL.

SQL-like scripting language:
means a language for writing script that processes queries similar to SQL. SQL lets us:
(i) write structured queries for processing in DBMS,
(ii) create and modify schema, and control the data access,
(iii) create client for sending query scripts, and create and manage server databases, and
(iv) view, query and change (update, insert or append or delete) databases.

NoSQL DBs refers to DBs with no prior fixed schema, schema-less models, and databases which
possess increasing flexibility for data manipulation.
NoSQL data model
Relaxation in one or more of the ACID properties (Atomicity, Consistence, Isolation and Durability)
of the database.
A theorem known as CAP (Consistency, Availability and Partitions) states that out of three
properties, at least two must be present.
NoSQL relies upon another model known as the BASE model.
BASE model has three principles: Basic availability (the availability of data even in the presence of
multiple failures), Soft state (data consistency is the developer's problem and should not be handled by
the database), Eventual consistency (when no new changes occur on existing data, eventually all
accesses to that data will return the last updated value).

Data-architecture patterns refer to formats used in NoSQL DBs. The examples are Key-Value Data
Stores, Object Data Stores, Column family Big Data Stores, Tabular Data Stores and Document
Stores, Graph Data Store.
Key-Value Data Store refers to a simplest way to implement a schema-less database.
A string called key maps to values in a large data string hence storage easily scales up and data
retrievals are fast.
Object Data Store refers to a repository which stores the
(i) objects (such as files, images, documents, folders and business reports),
(ii) system metadata which provides information such as filename, creation_date, last_modified,
language_used (such as Java, C, C#, C++, Smalltalk, Python), access_permissions, supported Query
languages, and
(iii) Custom metadata which provides information such as subject, category and sharing permission.

Tabular Data Store refers to table, column-family or BigTable like Data Store.
Column family Big Data store refers to a storage in logical groups of column families. The storage
may be similar to columns of sparse matrix. They use a pair of row and column keys to access the
column fields.
BigTable Data Store is a popular column-family based Data Store. Row key, column key and
timestamp uniquely identify a value. Google BigTable, HBase and Cassandra DBs use the BigTable Data
Store model.
Document Store means a NoSQL DB which stores hierarchical information in a single unit called
document. Document stores data in nested hierarchies; for example in XML document object model,
JSON formats data model.

Tuple means an ordered list of elements. Tuples implement the records.
Collection means a well-defined collection of distinct objects in a set, the objects of a set are called
elements. A collection is similar to a table of RDBMS.
A collection in a database also refers to storage of a number of documents.
A collection may store documents which do not have the same fields.
Thus, documents in the collection are schema-less. Thus, it is possible to store documents of varying
structures in a collection.
Aggregate refers to collection of data sets in the key value, column family or BigTable data stores
which usually require sequential processing.
Aggregation function refers to a function to find counts, sum, maximum, minimum, other statistical
or mathematical function using a collection of datasets, such as column or column-family.

Natural join is where two tables join based on all common columns. Both the tables must have the same
column name and the data type.
Inner join is the default natural join. It refers to two tables that join based on common columns
mentioned using the ON clause. Inner Join returns all rows from both tables if the columns match.
Node refers to a place for storing data, data block or read or write computations.
Data center in a DB refers to a collection of related nodes. Many nodes form a data center or rack.
Cluster refers to a collection of many nodes.
Keyspace means a namespace to group multiple column families or in general name given to group of
tables.

Indexing to a field means providing reference to a field in a document of collections.
Indexing is used to query and perform operations on that document.
A DB creates an index on the _id field of every collection
Projection refers to a unary operation (single input or operand) written as
Projection returns a set obtained by selecting only the n attributes.

4.2.1 MAPREDUCE MAP TASKS, REDUCE TASKS AND
MAPREDUCE EXECUTION
Big data processing employs the MapReduce programming model.
A Job means a MapReduce program.
Each job consists of several smaller units, called MapReduce tasks.
A software execution framework in MapReduce programming defines the parallel tasks.
MapReduce Programming Model

The model defines two important tasks, namely Map and Reduce.
Map takes input data set as pieces of data and maps them on various nodes for parallel processing.
The reduce task, which takes the output from the maps as an input and combines those data
pieces into a smaller set of data.
A reduce task always run after the map task (s).
Many real-world situations are expressible using this model.
This Model describes the essence of MapReduce programming where the programs written are
automatically parallelize and execute on a large cluster.

MapReduce process on client submitting a job
shows MapReduce process when a client submits a job, and the succeeding actions by the JobTracker and TaskTracker.

A user application specifies locations of the input/output data and translates into map and reduce
functions.
The Hadoop job client then submits the job (jar/executable etc.) and configuration to the JobTracker,
which then takes the responsibility of distributing the software/configuration to the slaves by scheduling
tasks, monitoring them, and provides status and diagnostic information to the job-client.
The master is responsible (JobTracker) for scheduling the component tasks in a job onto the slaves,
monitoring them and re-executing the failed tasks. The slaves execute the tasks as directed by the master.
The data for a MapReduce task is typically residing in the HDFS in the form of Files.
The data in files may be in the form of key value pairs, documents, Tabular data, line-based log files,
binary format file, multi-line input records.
These input files are practically very large, hundreds of terabytes or even more than it.

4.2.1 Map-Tasks
Map task means a task that implements a map(), which runs user application
codes for each key-value pair (ki, vi).
The output of map() would be zero (when no values are found) or intermediate key-value pairs
(k2, v2). The value v2 is the information that is later used at reducer for the transformation operation
using aggregation or other reducing functions.
Reduce task refers to a task which takes the output v2 from the map as an input and combines those
data pieces into a smaller set of data using a combiner.

Logical View of map() Functioning

Hadoop Mapper Class
Hadoop Java API includes Mapper class.
An abstract function map() is present in the Mapper class.
Any specific Mapper implementation should be a subclass of this class and overrides the abstract
function, map().
Note: Individual Mappers do not communicate with each other.

Number of Maps
The number of maps depends on the size of the input files, i.e., the total number of blocks of the
input files.
Example:
if the input files are of 1TB in size and the block size is 128 MB, there will be 8192 maps.
The number of map task Nmap, can be explicitly set by using setNumMapTasks(int).

4.2.2 Key-Value Pair
MapReduce uses only key-value pairs as input and output.
Hence available Data should be first converted into key-value pairs before it is passed to the
Mapper, as the Mapper only understands key-value pairs of data.
Key-value pairs in Hadoop MapReduce are generated as follows:
InputSplit - Defines a logical representation of data and presents a Split data for processing at
individual map().
As user we don’t deal with InputSplit in Hadoop directly, as InputFormat (InputFormat is responsible
for creating the Inputsplit and dividing into the records) creates it. FileInputFormat breaks a file into
128MB chunks.

Also, by setting mapred.min.split.size parameter in mapred-site.xml user can change the value of the
split size as per requirement.
Each 128MB chunks consists of a set to data records to be processed by a Map().
RecordReader –
Communicates with the InputSplit and converts the Split into records which are in the form of key-
value pairs in a format suitable for reading by the Mapper.
RecordReader uses TextlnputFormat by default for converting data into key-value pairs.
The splits are passed to createRecordReader(). CreateRecordReader() creates RecordReader for the
split. Then RecordReader generate record (key-value pair), which it passes to the map function.

Steps Involved in MapReduce key-value pairing

Map and Reduce functions uses key value pair at 4 places:
1. map() input,
2. map() output,
3. reduce() input and
4. reduce() output.
map() input map() output
reduce() input
reduce() output

4.2.3 Grouping by Key
Mapper outputs by grouping the key-values, and the value v2 append in a list of values.
A "Group By" operation on intermediate keys creates v2.
Shuffle and Sorting Phase
Shuffling in MapReduce
The process of transferring data from the mappers to reducers is known as shuffling i.e. the
process by which the system performs the sort and transfers the map output to the reducer as input. So,
MapReduce shuffle phase is necessary for the reducers, otherwise, they would not have any input (or
input from every mapper).

Sorting in MapReduce
The keys generated by the mapper are automatically sorted by MapReduce Framework, i.e. Before
starting of reducer, all intermediate key-value pairs in MapReduce that are generated by mapper get
sorted by key and not by value.

4.2.4 Partitioning
A partitioner partitions the key-value pairs of intermediate Map-outputs.
It partitions the data using a user-defined condition, which works like a hash function.
The total number of partitions is same as the number of Reducer tasks for the job. Let us take an
example to understand how the partitioner works.
Write an application to process the input dataset to find the highest salaried employee by gender
in different age groups (for example, below 20, between 21 to 30, above 30).

Method − The operation of this map task is as follows −
Read the value (record data), which comes as input value from the argument list in a string.
Using the split function, separate the gender and store in a string variable.
Input − The key would be a pattern such as “any special key + filename + line number” (example: key =
@input1) and the value would be the data in that line (example: value = 1201 t gopal t 45 t Male t 50000).
String[] str = value.toString().split("t");
String gender=str[3];
Send the gender information and the record data value as output key-value pair from the map task to the partition
task.
context.write(new Text(gender), new Text(value));
Repeat all the above steps for all the records in the text file.
Output − You will get the gender data and the record data value as key-value pairs.

Partitioner Task
The partitioner task accepts the key-value pairs from the map task as its input. Partition implies
dividing the data into segments. According to the given conditional criteria of partitions, the input key-
value paired data can be divided into three parts based on the age criteria.
Input − The whole data in a collection of key-value pairs.
key = Gender field value in the record.
value = Whole record data value of that gender.
Method − The process of partition logic runs as follows.
Read the age field value from the input key-value pair.
String[] str = value.toString().split("t");
int age = Integer.parseInt(str[2]);

Check the age value with the following conditions.
Age less than or equal to 20
Age Greater than 20 and Less than or equal to 30.
Age Greater than 30.
if(age<=20)
{
return 0;
}
else if(age>20 && age<=30)
{
return 1;
}
else
{
return 2;
}
Output − The whole data of key-value pairs are segmented into three collections of key-value pairs. The Reducer works
individually on each collection.

Reduce Tasks
The number of partitioner tasks is equal to the number of reducer tasks. Here we have three partitioner
tasks and hence we have three Reducer tasks to be executed.
Input − The Reducer will execute three times with different collection of key-value pairs.
key = gender field value in the record.
value = the whole record data of that gender.
Method − The following logic will be applied on each collection.
 Read the Salary field value of each record.
String [] str = val.toString().split("t”);
Note: str[4] have the salary field value.

Check the salary with the max variable. If str[4] is the max salary, then assign str[4] to max, otherwise
skip the step.
if(Integer.parseInt(str[4])>max)
{
max=Integer.parseInt(str[4]);
}
Repeat Steps 1 and 2 for each key collection (Male & Female are the key collections). Afterexecuting
these three steps, you will find one max salary from the Male key collection and one max salary from
the Female key collection.

4.2.5 Combiners
A Combiner, also known as a semi-reducer, is an optional class that operates by accepting the inputs
from the Map class and thereafter passing the output key-value pairs to the Reducer class.
The main function of a Combiner is to summarize the map output records with the same key.
The output (key-value collection) of the combiner will be sent over the network to the actual Reducer
task as input.
The Combiner class is used in between the Map class and the Reduce class to reduce the volume
of data transfer between Map and Reduce. Usually, the output of the map task is large and the data
transferred to the reduce task is high.
The following MapReduce task diagram shows the COMBINER PHASE.

MapReduce Combiner Implementation
1. The following example provides a theoretical idea about combiners. Let us assume we have the
following input text file named input.txt for MapReduce.

The important phases of the MapReduce program with
Combiner are discussed below.
Record Reader
• This is the first phase of MapReduce where the Record Reader reads every line from the input text file
as text and yields output as key-value pairs.
Input − Line by line text from the input file.
Output − Forms the key-value pairs. The following is the set of expected key-value pairs.

Map Phase
The Map phase takes input from the Record Reader, processes it, and produces the output as another
set of key-value pairs.
Input − The following key-value pair is the input taken from the Record Reader.
Output − The expected output is as follows −

Combiner Phase
The Combiner phase takes each key-value pair from the Map phase, processes it, and produces the
output as key-value collection pairs.
Input − The following key-value pair is the input taken from the Map phase.
Output − The expected output is as follows −

Reducer Phase
• The Reducer phase takes each key-value collection pair from the Combiner phase, processes it, and
passes the output as key-value pairs. Note that the Combiner functionality is same as the Reducer.
• Input − The following key-value pair is the input taken from the Combiner phase.
• Output − The expected output from the Reducer phase is as follows −

Record Writer
This is the last phase of MapReduce where the Record Writer writes every key-value pair from the
Reducer phase and sends the output as text.
Input − Each key-value pair from the Reducer phase along with the Output format.
Output − It gives you the key-value pairs in text format. Following is the expected output.

Benefits of Combiner
This limits the volume of data transfer between map and reduce tasks, and thus reduces the cost of data
transfer across the network.
Combiners use grouping by key for carrying out this function.
The combiner works as follows:
It does not have its own interface and it must implement the interface at reduce().
It operates on each map output key. It must have the same input and output key-value types as the
Reducer class.
It can produce summary information from a large dataset because it replaces the original Map
output with fewer records or smaller records.

4.2.6 Reduce Tasks
Java API at Hadoop includes Reducer class. An abstract function, reduce() is in the Reducer. Any
specific Reducer implementation should be subclass of this class and override the abstract reduce().
Reduce task implements reduce() that takes the Mapper output (which shuffles and sorts), which is
grouped by key-values (k2, v2) and applies it in parallel to each group.
Intermediate pairs are at input of each Reducer in order after sorting using the key.
Reduce function iterates over the list of values associated with a key and produces outputs such as
aggregations and statistics.
The reduce function sends output zero or another set of key-value pairs (k3, v3) to the final the output
file. Reduce: {(k2, list (v2) -> list (k3, v3)}

4.2.7 Details of MapReduce Processing Steps

Example:
Automotive Components and Predictive Automotive Maintenance Services (ACPAMS). ACPAMS is an
application of (Internet) connected cars which renders services to customers for maintenance and
servicing of (Internet) connected cars.
ACPAMS Company can receive the alerts/messages every hour from several sensors of the automotive
components installed in the number of cars. A server maps the keys for filling fuel, changing of the
coolant, etc. It requires a lot of time to scan the hourly maintenance log sequentially because there are
a million cars registered for the ACPAMS service. Each car is equipped with nearly 50 sensor-based
components sending alerts/message every minute.
The MapReduce programmer can split the application task among multiple subtasks, say one hundred
sub-tasks, and each sub-task processes the data of a selected set of a Service users.

Alert Messages
The log contains information in the following format:
maintenance service Id:(<CarRegistrationNumber>_<alert/messsageld>)

MapReduce Processing Steps in ACPAMS
Application

Pseduocodes for MapReduce algorithm for the
ACPAMS.

4.2.8 Coping with Node Failures
Hadoop achieves fault tolerance is through restarting the tasks.
Each task nodes (TaskTracker) regularly communicates with the master node, JobTracker.
If a TaskTracker fails to communicate with the JobTracker for a pre-defined period (by default, it is set
to 10 minutes), JobTracker assumes node failure.
The JobTracker knows which map and reduce tasks were assigned to each TaskTracker.
If a TaskTracker has already completed nine out of ten reduce tasks assigned to it, only the tenth task
must execute at a different node.
Map tasks are slightly more complicated. A node may have completed ten map tasks but the
Reducers may not have copied all their inputs from the output of those map tasks. Now if a node fails,
then its Mapper outputs are inaccessible. Thus, any complete map tasks must also be re-executed to
make their results available to the remaining reducing nodes. Hadoop handles all of this automatically.

Following points summarize the coping mechanism with distinct Node Failures:
(i) Map TaskTracker failure:
• Map tasks completed or in-progress at TaskTracker, are reset to idle on failure
• Reduce TaskTracker gets a notice when a task is rescheduled on another TaskTracker
(ii) Reduce TaskTracker failure:
- Only in-progress tasks are reset to idle
(iii) Master JobTracker failure:
- Map-Reduce task aborts and notifies the client (in case of one master node).

4.3.1 COMPOSING MAPREDUCE FOR CALCULATIONS AND
ALGORITHMS
Here we understand the use of MapReduce program composition in counting and summing,
algorithms for relational algebraic operations, projections, unions, intersections, natural joins,
grouping and aggregation, matrix multiplication and other computations.

4.3.1 Composing Map-Reduce for Calculations
Some calculations are:
Counting and Summing: Counting is used in the data querying application. For example,
count of messages generated, word count in a file, number of cars sold, and analysis of the
logs, such as number of tweets per month. Application is also in business analytics field.
Sorting Many applications need sorted values in a certain order by some rule or process.
Mappers just emit all items as values associated with the sorting keys which assemble as a
function of items. Reducers combine all emitted parts into a final list.

Finding Distinct Values (Counting unique values) Applications such as web log analysis need
counting of unique users.
Two solutions are possible:
(i)The Mapper emits the dummy counters for each pair of field and groupId, and the Reducer
calculates the total number of occurrences for each such pair.
(ii)The Mapper emits the values and groupId, and the Reducer excludes the duplicates from the
list of groups for each value and increments the counter for each group. The final step is to sum
all the counters emitted at the Reducer. This requires only one MapReduce job but the process is
not scalable, and hence has limited applicability in large data sets.

Collating: Collating is a way to collect all items which have the same value for a function in one
document or file, or a way to process items with the same value of the function together.
Examples of applications are producing inverted indexes
Indexing
Indexing allows you to traverse from index to Topic
Inverted Indexing
Inverted Indexing Allows you to traverse from
words to documents

Filtering or Parsing: Filtering or parsing collects only those items which satisfy some
condition or transform each item into some other representation. Filtering/parsing include tasks
such as text parsing, value extraction and conversion from one format to another.
Examples of applications of filtering are found in data validation, log analysis and querying
of datasets.

4.3.2 Matrix-Vector Multiplication by MapReduce

Algorithm for using MapReduce:
The Mapper operates on A and emits row-wise multiplication of each matrix
element and vector element (aij x bj i). The Reducer executes sum() for
summing all values associated with each i and emits the element ci. Application of
the algorithm is found in linear transformation.

4.3.3 Relational—Algebra Operations
Relational algebraic operations on large datasets using MapRed:
4.3.3.1 Selection
Example of Selection in relational algebra is as follows: Consider the attribute names
(ACVM_ID, Date, chocolate_flavour, daily_sales). Consider relation

The Mapper calls test() for each tuple in a row. When test satisfies the selection criterion
then emits the tuple. The Reducer transfers the received input tuple as the output.

4.3.3.2 Projection
Example of Projection in relational algebra is as follows:
The test() tests the presence of attribute (s) used for projection and the factor by an attribute needs
projection.
The Mapper calls test() for each tuple in a row. When the test satisfies, then emits the tuple (same as
in selection). The Reducer transfers the received input tuples after eliminating other attributes. Such
operations are used in analytics.

4.3.3.3 Union
Example of Union in relations is as follows: Consider,
The Mapper executes all tuples of two sets for union and emits all the resultant tuples.
The Reducer class object transfers the received input tuples after eliminating the possible
duplicates.

4.3.3.4 Intersection and Difference
Intersection Example of Interaction in relations is as follows: Consider,
The Mapper executes all tuples of two sets for intersection and emits all the resultant
tuples. The Reducer transfers only tuples that occurred twice.
R1 = {(524, 12122017, Oreo, 72)}
R2 = {(525, 12122017, KitKat, 82)}
and
R3 = {(526, 12122017, KitKat, 82), (524, 12122017, Oreo, 72)}
Result of Intersection operation between R1 and R3 are
R1 R3 = {(524, 12122017, Oreo, 72)}

The Mapper emits all the tuples and tag. A tag is the name of the set (say, set_1 or set_2 to which a tuple
belongs to).
The Reducer transfers only tuples that belong to set_1.

Symmetric Difference
Symmetric difference (notation is A B (or A B)] is another relational entity. It
means the set of elements in exactly one of the two relations A or B.
R3 R1= {(12122017, Oreo, 25), (12122017, Oreo, 72)}
The Mapper emits all the tuples and tag. A tag is the name of the set (say,
set_l or set_2 this tuple belongs to).
The Reducer transfers only tuples that belong to neither set _l or set_2.

4.4.1 HIVE
Hive was created by Facebook.
Hive is a data warehousing tool and is also a data store on the top of Hadoop.
Enterprises uses a data warehouse as large data repositories that are designed to enable the
Searching, managing, and analyzing the data.
Hive processes structured data, integrates well heterogeneous sources.
Additionally, also manages the volumes of data.

HIVE Features

Hive Characteristics
1. Has the capability to translate queries into MapReduce jobs. This makes Hive scalable,
able to handle data warehouse applications, and therefore, suitable for the analysis of static
data of an extremely large size data and application.
2. Supports web interfaces as well. Application APIs as well as web-browser clients, can
access the Hive DB server.
3. Provides an SQL dialect (Hive Query Language, abbreviated HiveQL or HQL).

HIVE Limitations
1.Not a full database. Main disadvantage is that Hive does not provide update, alter and
deletion of records in the database.
2.Not developed for unstructured data.
3.Not designed for real-time queries.
4.Performs the partition always from the last column.

4.4.1 Hive Architecture

Hive architecture components are:
• Hive Server - An optional service that allows a remote client to submit requests to Hive and
retrieve results. Requests can use a variety of programming languages. Hive Server exposes a
very simple client API to execute HiveQL statements.
• Hive CLI (Command Line Interface) - Popular interface to interact with Hive. Hive runs in
local mode that uses local storage when running the CLI on a Hadoop cluster instead of HDFS.
• Web Interface - Hive can be accessed using a web browser as well. This requires a HWI
Server running on some designated code. The URL http:// hadoop:<port no.> / hwi command
can be used to access Hive through the web.

• Metastore - It is the system catalog. All other components of Hive interact with the
Metastore. It stores the schema or metadata of tables, databases, columns in a table, their data
types and HDFS mapping.
• Hive Driver -It manages the life cycle of a HiveQL statement during compilation,
optimization and execution.

4.4.2 Hive Installation
Hive can be installed on Windows 10, Ubuntu 16.04 and MySQL. It requires
three software packages:
• Java Development kit for Java compiler (Javac) and interpreter
• Hadoop
• Compatible version of Hive with Java- Hive 1.2 onward supports Java
1.7 or newer.

Steps for installation of Hive in a Linux based
OS are as follows:
1. Install Javac and Java from Oracle Java download site. Download jdk 7 or a later version
from
http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-
1880260.html, and extract the compressed file.
All users can access Java by Making Java available to all users.
The user has to move it to the location "/usr/local/" using the required commands.
2. Set the path by the commands for jdk1.7.0_71,
export JAVA_HOME=usr/local/jdk1.7.0_71,
export PATH=$PATH: $JAVA_HOME/bin
(Can use alternative install /usr/bin/java usr/local/java/bin/java 2)

3. Install Hadoop http://apache.claz.org/hadoop/common/hadoop-
2.4.1/
4. Make shared HADOOP, MAPRED, COMMON, HDFS and all related files,
configure HADOOP and set property such as replication parameter.
5. Name the yarn.nodemanager.aux-services. Assign value to
mapreduce_shuffle. Set namenode and datanode paths.
6. Download http://apache.petsads.us/hive/hive-0.14.0/. Use is command to
verify the files $ tar zxvf apache-hive-0.14.0-bin.tar.gz, $ ls
7. Use an external database server. Configure metastore for the server.

4.4.3 Comparison with RDBMS (Traditional
Database)

4.4.4 Hive Data Types and File Formats

Hive has three Collection data types

HIVE file formats and their descriptions

4.4.5 Hive Data Model

4.4.6 Hive Integration and Workflow Steps
Hive integrates with the MapReduce and HDFS. Figure below shows the dataflow
sequences and workflow steps between Hive and Hadoop.

Steps 1 to 11 are as follows:

4.4.7 Hive Built-in Functions

Examples
Following are the examples of the returned output:
SELECT floor(10.5) from marks; Output = 10.0
SELECT ceil(10.5) from marks; Output = 11.0

4.51 HIVEQL
Hive Query Language (abbreviated HiveQL) is for querying the large datasets
which reside in the HDFS environment.
HiveQL script commands enable data definition, data manipulation and query
processing.
HiveQL supports a large base of SQL users who are using SQL to extract
information from data warehouse.

4.5.1 HiveQL Data Definition Language (DDL)
HiveQL database commands for data definition for DBs and Tables are
CREATE DATABASE,
SHOW DATABASE (list of all DBs),
CREATE SCHEMA,
CREATE TABLE.

Following are HiveQL commands which create
a table:

External Table

Example: on Creating a Table

Creating a Database

Showing Database

Dropping a Database

Example
Usages of database commands for CREATE, SHOW and DROP.
CREATE DATABASE IF NOT EXISTS toys_companyDB;
SHOW DATABASES;
default toys_companyDB //Default database is test.
//Delete dtabase using the command:
Drop Database toys_companyDB.

4.5.2 HiveQL Data Manipulation Language
(DML)
HiveQL commands for data manipulation are
USE <database name>,
DROP DATABASE,
DROP SCHEMA,
ALTER TABLE,
DROP TABLE, and
LOAD DATA.

Loading Data into HIVE DB

Example for usages of data manipulation
commands, INSERT, ALTER, and DROP.

Solution (i): Loading Data

Solution (ii): Alter

Solution (iii): Delete

4.5.3 HiveQL For Querying the Data
• For any data analysis application there is a need for partitioning and storing the
data.
• A data warehouse should have a large number of partitions where the tables,
files and databases store.
• Querying then requires sorting, aggregating and joining functions.

4.5.3.1 Partitioning
• Table partitioning refers to dividing the table data into some parts based on the
values of particular set of columns.
• Hive organizes tables into partitions.
• Partition makes querying easy and fast.
• This is because SELECT is then from the smaller number of column fields.
• The following example explains the concept of partitioning, columnar and file
records formats.

Table Partitioning

Renaming the Partition

Add a partition to the Existing Table

Drop a partition

Example

Partitioning: Query Fast Processing
The following example shows how querying is processed fast by using partitioning
of a table.
A query processes faster when using partition.
Selection of a product of a specific category from a table during query processing
takes lesser time when the table has a partition based on a category.

Advantages of Partition
1. Distributes execution load horizontally.
2. Query response time becomes faster when processing a small part of the
data instead of searching the entire dataset.
Limitations of Partition
1. Creating a large number of partitions in a table leads to a large number of
files and directories in HDFS, which is an overhead to NameNode, since it
must keep all metadata for the file system in memory only.
2. Partitions may optimize some queries based on Where clauses, but they
may be less responsive for other important queries on grouping clauses.

3. A large number of partitions will lead to a large number of tasks (which will
run in separate JVM) in each MapReduce job, thus creating a lot of
overhead in maintaining JVM start up and tear down (A separate task will be
used for each file). The overhead of JVM start up and tear down can exceed
the actual processing time in the worst case.

4.5.3.2 Bucketing
A partition itself may have a large number of columns when tables are very large.
Tables or partitions can be sub-divided into buckets.
Division is based on the hash of a column in the table.
CLUSTERED BY clause divides a table into buckets. A coding example on Buckets is
given below:

4.5.3.3 Views
Views provide ease of programming.
Complex queries simplify using reusable Views.
A View provisions the following:
• Saves the query and reduces the query complexity
• Use a View like a table but a View does not store data like a table
• Hides the complexity by dividing the query into smaller, more manageable
pieces
• The Hive executes the View and then the planner combines the information
in View definition with the remaining actions on the query

4.5.3.4 Sub-Queries (Using Views)

4.5.4 Aggregation
Hive supports the following built-in aggregation functions. The usage of these
functions is same as the SQL aggregate functions.

4.5.5 Join
• A JOIN clause combines columns of two or more tables, based on a relation
between them.
• HiveQL Join is more or less similar to SQL JOINS.
• Example:

Join Example

Left outer Join Example

Right Outer Join Example

Full Outer Join Example

4.5.6 Group By Clause

4.6.1 PIG
Apache developed Pig, which:
• Is an abstraction over MapReduce
• Is an execution framework for parallel processing
• Reduces the complexities of writing a MapReduce program
• Is a high-level dataflow language. Dataflow language means that a Pig
operation node takes the inputs and generates the output for the next node
• Is mostly used in HDFS environment
• Performs data manipulation operations at files at data nodes in Hadoop.

Applications of Apache Pig
Applications of Pig are:
• Analyzing large datasets
• Executing tasks involving adhoc processing
• Processing large data sources such as web logs and streaming online data
• Data processing for search platforms. Pig processes different types of data
• Processing time sensitive data loads; data extracts and analyzes quickly. For
example, analysis of data from twitter to find patterns for user behavior and
recommendations.

Features
(i) Apache PIG helps programmers write complex data transformations using scripts
(without using Java). Pig Latin language is very similar to SQL and possess a rich set of
built-in operators, such as group, join, filter, limit, order by, parallel, sort and split.
It provides an interactive shell known as Grunt to write Pig Latin scripts.
(ii) Creates user defined functions (UDFs) to write custom functions which are not available
in Pig. A UDF can be in other programming languages, such as Java, Python, Ruby,
Jython, JRuby. They easily embed into Pig scripts written in Pig Latin. UDFs provide
extensibility to the Pig.
(iii) Process any kind of data, structured, semi-structured or unstructured data, coming from
various sources.
(iv) Reduces the length of codes using multi-query approach. Pig code of 10 lines is equal to
MapReduce code of 200 lines. Thus, the processing is very fast.

(v) Extracts the data, performs operations on that data and dumps the data in
the required format in HDFS. The operation is called ETL (Extract Transform
Load).
(vi) Performs automatic optimization of tasks before execution.
(vii) Programmers and developers can concentrate on the whole operation
without a need to create mapper and reducer tasks separately.

(viii) Reads the input data files from HDFS or the data files from other sources
such as local file system, stores the intermediate data and writes back the
output in HDFS.
(ix) Pig characteristics are data reading, processing, programming the UDFs in
multiple languages and programming multiple queries by fewer codes. This
causes fast processing.
(x) Pig derives guidance from four philosophies, live anywhere, take anything,
domestic and run as if flying. This justifies the name Pig, as the animal pig
also has these characteristics.

Differences between Pig and MapReduce

Differences between Pig and SQL

Differences between Pig and Hive

Pig Architecture

The three ways to execute scripts are:
1. Grunt Shell: An interactive shell of Pig that executes the scripts.
2. Script File: Pig commands written in a script file that execute at Pig Server.
3. Embedded Script: Create UDFs for the functions unavailable as Pig built-in
operators. UDF can be in other programming languages. The UDFs can
embed in Pig Latin Script file.

1. Parser
A parser handles Pig scripts after passing through Grunt or Pig Server.
The Parser performs type checking and checks the script syntax.
The output is a Directed Acyclic Graph (DAG).
DAG represents the Pig Latin statements and logical operators. Nodes represent
the logical operators. Edges between sequentially traversed nodes represent the
dataflows.

2. Optimizer
The DAG is submitted to the logical optimizer.
The optimization activities, such as split, merge, transform and reorder operators
execute in this phase.
The optimization is an automatic feature.
The optimizer reduces the amount of data in the pipeline at any instant of time,
while processing the extracted data.
It executes certain functions for carrying out this task, as explained as follows:

PushUpFilter: If there are multiple conditions in the filter and the filter can be split,
Pig splits the conditions and pushes up each condition separately. Selecting these
conditions at an early stage helps in reducing the number of records remaining in
the pipeline.
PushDownForEachFlatten: Applying flatten, which produces a cross product
between a complex type such as a tuple, bag or other fields in the record, as late
as possible in the plan. This keeps the number of records low in the pipeline.
ColumnPruner: Omitts never used columns or the ones no longer needed, reducing
the size of the record. This can be applied after each operator, so that the fields
can be pruned as aggressively as possible.
MapKeyPruner: Omitts never used map keys, reducing the size of the record.

LimitOptimizer: If the limit operator is immediately applied after load or sort
operator, Pig converts the load or sort into a limit-sensitive implementation, which
does not require processing the whole dataset. Applying the limit earlier reduces
the number of records.
Compiler The compiler compiles after the optimization process. The optimized
codes are a series of MapReduce jobs.
Execution Engine Finally, the MapReduce jobs submit for execution to the
engine. The MapReduce jobs execute and it outputs the final result.

Apache Pig — Grunt Shell

Installing Pig

Pig Latin Data Model

Big Data Analytics Module-4 power point presentation vtu

Recommended

More Related Content

Similar to Big Data Analytics Module-4 power point presentation vtu (20)

Recently uploaded (20)

Big Data Analytics Module-4 power point presentation vtu