Scribd
Upload
117 views

Data Ingestion From The RDS To HDFS Using Sqoop

Sqoop is used to import data from an RDS MySQL database into HDFS. An external Hive table is then created over this data and queries are run to verify the data import. The data is then transformed into optimized file formats like ORC and stored in a Hive-HBase table for analysis. Various analyses are performed including finding child mortality rates by state, highest populations per household by district, and lowest sex ratios by district. Results are viewed in Hue and performance is compared across file formats.

Uploaded by

Ram Guggul
Copyright
© © All Rights Reserved
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
117 views

Data Ingestion From The RDS To HDFS Using Sqoop

Sqoop is used to import data from an RDS MySQL database into HDFS. An external Hive table is then created over this data and queries are run to verify the data import. The data is then transformed into optimized file formats like ORC and stored in a Hive-HBase table for analysis. Various analyses are performed including finding child mortality rates by state, highest populations per household by district, and lowest sex ratios by district. Results are viewed in Hue and performance is compared across file formats.

Uploaded by

Ram Guggul
Copyright
© © All Rights Reserved
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
You are on page 1/ 5

Data Ingestion from the RDS to HDFS using Sqoop

1. Sqoop Import command


sqoop import --connect jdbc:mysql://upgradawsrds.cpclxrkdvwmz.us-east-
1.rds.amazonaws.com:3306/indiaahs2012_13 --username upgraduser --password upgraduser --table
Key_indicator_districtwise
2. Command to see the list of imported data

[root@ip-10-0-0-157 ~]# hadoop fs -ls


Found 7 items
drwx------ - root supergroup 0 2019-06-03 10:00 .Trash
drwx------ - root supergroup 0 2019-07-07 19:56 .staging
drwxr-xr-x - root supergroup 0 2019-05-18 18:06 Categories
drwxr-xr-x - root supergroup 0 2019-07-07 19:56 Key_indicator_districtwise
-rw-r--r-- 3 root supergroup 15081 2019-03-17 17:48 dropbox-policy.txt
[root@ip-10-0-0-157 ~]# hadoop fs -cat Key_indicator_districtwise/part-m-*

External table creation in Hive and loading the ingested data into it. Data ingestion
verification.
1. Command to create the external table
hive> create external table if not exists indiaahs_extdata (
> `ID` int,
> `State_Name` string,
> `State_District_Name` string,
> `AA_Sample_Units_Total` int,
> `AA_Sample_Units_Rural` int,
> `AA_Sample_Units_Urban` int,
-
-
-
> `AA_Children_12_23_Months_Urban` int,
> `BB_Average_Household_Size_Sc_Total` float,
> `BB_Average_Household_Size_Sc_Rural` float,
> `BB_Average_Household_Size_Sc_Urban` float,
> `BB_Average_Household_Size_St_Total` float,
-
-
-
> `ZZ_Sex_Ratio_At_Birth_Total_Lower_Limit` int,
> `ZZ_Sex_Ratio_At_Birth_Total_Upper_Limit` int,
> `ZZ_Sex_Ratio_At_Birth_Rural_Lower_Limit` int,
> `ZZ_Sex_Ratio_At_Birth_Rural_Upper_Limit` int,
> `ZZ_Sex_Ratio_At_Birth_Urban_Lower_Limit` int,
> `ZZ_Sex_Ratio_At_Birth_Urban_Upper_Limit` int)
> row format delimited fields terminated by ','
> location '/hive/data/indiaahs_data';
OK
Time taken: 1.902 seconds

2. Command to load the ingested data into the external table


hive> load data local inpath 'hdfs:/user/root/Key_indicator_districtwise/part-m-*' into table
indiaahs_data;

3. Queries to verify that the ingestion is correctly accomplished


a. Query to count the total number of rows along with the screenshots of the data
fetched by the query on MySQL Workbench and Hue
Query:
select count(*) from Key_indicator_districtwise;
MySQL Workbench:

Hue:
Hue is not working in my hadoop cluster

b. Query to select the top 10 rows and first 8 columns along with the screenshots
of the data fetched by the query on MySQL Workbench and Hue
Query:
select ID, State_Name, State_District_Name, AA_Sample_Units_Total, AA_Sample_Units_Rural,
AA_Sample_Units_Urban, AA_Households_Total, AA_Households_Rural
from Key_indicator_districtwise limit 10;
MySQL Workbench:

Hue:
Hue is not working in my hadoop cluster

Subset schema creation in Hive to support the analyses


1. Columns used in the subset schema
CREATE TABLE Indiaahs2012 AS
> SELECT ID, State_Name, State_District_Name FROM indiaahs_data
> WHERE (ID = 7 AND State_Name = Assam);
2. Storage format used
ORC
3. Create and insert command for the default format
<Create command>
<Insert command>
4. Create and insert command for the formats such as ORC
<Create command>
<Insert command>
5. Create and insert command for the Hive-HBase integrated table
<Create command>
<Insert command>
6. Screenshot of runtimes against each query given above for the default format, formats
such as ORC format as well as Hive-Hbase integration
For default format:
<Screenshot of run time for query 1>
<Screenshot of run time for query 2>
<Screenshot of run time for query 3>
For formats such as ORC:
<Screenshot of run time for query 1>
<Screenshot of run time for query 2>
<Screenshot of run time for query 3>
For Hive-Hbase Integrated Table:
<Screenshot of run time for query 1>
<Screenshot of run time for query 2>
<Screenshot of run time for query 3>
6. Create and insert command for the partition table for analyses 1 & 2.
<Create command>
<Insert command>
The result of each analysis along with the query and the corresponding chart generated in
Hue. Keep optimizations in mind
1. The child mortality rate of Uttar Pradesh
<Query on the table with the chosen format such as orc>
<Screenshot of the result>
<Query on the Hive-Hbase integrated table>
<Screenshot of the result>
2. The fertility rate of Bihar
<Query on the table with the chosen format such as orc>
<Screenshot of the result>
<Query on the Hive-Hbase integrated table>
<Screenshot of the result>
3. State wise child mortality rate and state wise fertility rate and does high fertility
correlate with high child mortality?
<Query on the table with the chosen format such as orc>
<Screenshot of the result>
<Query on the Hive-Hbase integrated table>
<Screenshot of the result>
<Chart>
<Query on the table with the chosen format such as orc>
<Screenshot of the result>
<Query on the Hive-Hbase integrated table>
<Screenshot of the result>
<Chart>
<Query on the table with the chosen format such as orc>
<Screenshot of the result>
<Query on the Hive-Hbase integrated table>
<Screenshot of the result>
<Chart>
4. Find top 2 districts per state with the highest population per household
<Query on the table with the chosen format such as orc>
<Screenshot of the result>
<Query on the Hive-Hbase integrated table>
<Screenshot of the result>
<Chart>
5. Find top 2 districts per state with the lowest sex ratios
<Query on the table with the chosen format such as orc>
<Screenshot of the result>
<Query on the Hive-Hbase integrated table>
<Screenshot of the result>
<Chart>

You might also like