Scribd
Upload
9 views

HIVE Avro Data File and Avsc Schema File

The document discusses storing data from an RDBMS table into an Avro data file and schema file using Sqoop and Hive. It shows using Sqoop to import an RDBMS table into an Avro file, creating an external Hive table with the Avro schema, and modifying the Avro schema file to remove a field.

Uploaded by

Srinivas H
Copyright
© © All Rights Reserved
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
9 views

HIVE Avro Data File and Avsc Schema File

The document discusses storing data from an RDBMS table into an Avro data file and schema file using Sqoop and Hive. It shows using Sqoop to import an RDBMS table into an Avro file, creating an external Hive table with the Avro schema, and modifying the Avro schema file to remove a field.

Uploaded by

Srinivas H
Copyright
© © All Rights Reserved
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
You are on page 1/ 3

HIVE avro data file and avsc schema file

24 Oct 2021
Hands On

RDBMS

create database acheck;

use acheck;

create table atab(id int,name varchar(100),amount int);

insert into atab values(1,'zeyo',40);

insert into atab values(2,'sai',50);

insert into atab values(3,'hema',60);

Edge Node

mkdir /home/cloudera/avscdir

cd /home/cloudera/avscdir

sqoop import --connect jdbc:mysql://localhost/acheck --username root --password cloudera --table


atab -m 1 --target-dir /user/cloudera/atab --as-avrodatafile

ls /home/cloudera/avscdir =>>> check whether you have avsc file

hadoop fs -put /home/cloudera/avscdir/atab.avsc /user/cloudera/


Hive -location

create external table atab ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe'


STORED AS AVRO LOCATION '/user/cloudera/atab' TBLPROPERTIES
('avro.schema.url'='/user/cloudera/atab.avsc');

select * from atab; ===> u will see the data

Go to Hue ---> edit the avsc file and remove

,{

"name" : "name",

"type" : [ "null", "string" ],

"default" : null,

"columnName" : "name",

"sqlType" : "12"

}
Come to hive and query the table again

select * from atab; --- You should see only two columns

You might also like