Apache Pig Relational Operators - II
0 likes153 views
The document provides an overview of various Pig relational operators, including foreach, filter, join, co-group, and union. It explains how these operators can be used to manipulate and query datasets in Pig, showcasing examples for each operator type such as self-join, inner join, and outer join. Additionally, it discusses handling different schemas in datasets and introduces the rank operator for assigning ranks to tuples.
Apache Pig Relational Operators - II
- 1. PIG Relational Operators - II Foreach, Filter, Join, Co- Group, Union
- 2. Relational operator: foreach foreach the name itself describes for each record do something. It is similar to For-Loop for specifying the iteration that is executed repeatedly. Example: select few columns grunt> a =foreach dataTransaction Generate $0,$1,$2 ; It can also be used for various arithmetic operations such as grunt> A= FOREACH dataTransaction Generate $0,($3+$4) as S; or grunt> a =foreach dataTransaction Generate $0, (TransAmt1+TransaAmt2) as S; Rupak Roy
- 3. grunt > B= FOREACH A GENERATE $1/100; or grunt> b = foreach A GENERATE ($1/100) as D C= FOREACH B GENERATE ( (D >50)?’above’ : ‘below’); or C= foreach B generate ( (D==50)?’Equal’ : ((D>50)?’above’:’down’)); Rupak Roy
- 4. Relational Operators: filter It is used to select the required tuple based on conditions. Or simply we can say filter helps to remove unwanted data/records based on requirements. Example such as: grunt> F = Filter dataTransaction by TransAmt1 > 500; Or grunt> F1 = filter dataTransaction by (($4+$5)/100) > 2 ; Or grunt> F2 = filter dataTransaction by $6 == ‘Nunavut’; Or grunt> F3 = filter data Transaction by $1 MATCHES ‘ Car.*’; #it will give all the names that starts with CA…. Or grunt> F4 = filter dataTRansaction by NOT $1 MATCHES ‘Car.*’; #it will give all the names that doesnot starts with CA Rupak Roy
- 5. Relational Operators: filter Or grunt>F5 = filter dataTransaction by CustomerName MATCHES ‘Ca.*s’; #it will filter the records based on names starting with ‘Ca’ and ends with ‘s ’ . To represent any number of characters we use * and in this case we want any number of characters before ‘s’but after Ca Or grunt> F5 = filter dataTransaction by CustomerName MATCHES ‘ .*(nica|los) .* ‘ #now here the dot start ( .* ) means it can have any number of characters before and after .*(nica or los) .* nica = MONICA Federle los = Carlos Daly Rupak Roy
- 6. Relational operators: Join Join Operator is used when we have to combine two or more datasets. Joining the two or more datasets is done based on a common key from the datasets. Joins can be of 3 types 1. Self-join 2. Inner-join 3. Outer-join – left join, right join and full join Rupak Roy
- 7. Self – join Self join is used for joining a table itself. Let’s understand this with the help of an example: #Load the same dataset under different Alias name: grunt> join1= LOAD ‘/home/hduser/datasets/join1.csv’ using PigStorage(‘,’) as ( CustomerNAme:chararray, Transaction_ID:bytearray, ProductName: chararray); grunt> join11= LOAD ‘/home/hduser/datasets/join1.csv’ using PigStorage(‘,’) as ( CustomerNAme:chararray, Transaction_ID:bytearray, ProductName: chararray); Rupak Roy
- 8. #perform Self-join using JOIN operator grunt> selfjoin = JOIN join1 by Transaction_ID, join11 by Transaction_ID; grunt> dump selfjoin; Rupak Roy
- 9. Inner-join Is also known as equijoin. Inner join returns rows when there is a match in both tables based on a common key or a value. #Load data2 grunt> join2= LOAD ‘/home/hduser/datasets/join2.csv’ using PigStorage(‘,’) as ( CustomerNAme:chararray, Transaction_ID:bytearray, Department: chararray); grunt> innerjoin = JOIN join1 by Transaction_ID, join2 by Transaction_ID; grunt> dump innerjoin; Rupak Roy
- 10. Outer Join Left Outer Join returns all rows from the left table, even if there is no match in the right table and it will take only the values from the right table that matches with the left table. grunt> leftouter = JOIN join1 by Transaction_ID LEFT OUTER, join2 BY Transaction_ID; Right Outer Join: is the opposite of Left Outer Join. It returns all the rows from the right table even if there are no matches in the left table and it will take only the values from the left table that matches with the right table grunt> rightouter =JOIN join1 by Transaction_ID RIGHT OUTER , join2 by Transaction_ID; Rupak Roy
- 11. Outer Join Full Outer Join: returns all the rows from both the tables when there is a match in one of the relations. grunt> fullouter = JOIN join1 by Transaction_ID FULL OUTER, join2 BY Transaction_ID; Rupak Roy
- 12. Joins are one of the important operators Rupak Roy
- 13. CO-Group: which essentially performs a join and a group at the same time. COGROUP on multiple datasets results in a record with a key dataset. To perform COGROUP type: grunt> COGROUP join1 on Transaction_ID, join2 on Transaction_ID; Rupak Roy
- 14. Relational Operator: UNION Is to merge the contents of two and more datasets. grunt> U = UNION join1, join2; dump U; What if we want to merge two datasets that has different schemas exampe: join1= LOAD ‘/home/hduser/datasets/join1.csv’ using PigStorage(‘,’) as ( CustomerNAme:chararray, Transaction_ID:chararray, Department: chararray); join1u= LOAD ‘/home/hduser/datasets/join1.csv’ using PigStorage(‘,’) as ( CustomerNAme:chararray, Transaction_ID:int, Department: chararray); join2= LOAD ‘/home/hduser/datasets/join2.csv’ using PigStorage(‘,’) as ( CustomerNAme:chararray, Transaction_ID:chararray, Department: chararray); Unioned= UNION join1u,join2 ; Describe Unioned; it will through an error ‘cannot cast to byte array ‘ due to different data types of transaction ID. Rupak Roy
- 15. It will be very tedious and time consuming to go back and forth and load the data to change the schema. We can also explicitly define the schema while using relational queries without disturbing the original schema. grunt> joinM= FOREACH join2 generate $0,(int)$1,$2; unioned = UNION joinM, join1u; describe unioned; Alternatively to perform UNION for incompatible data type using ONSCHEMA; grunt>U= UNION ONSCHEMA join1u, join2; Rupak Roy
- 16. Relational Operator: RANK Returns rank to each tuple with a relation; Example: grunt> vi names Zara,1,F David,2,F David,2,T Alan,2,M Calvin,3,M Alan,5,M Chris,8,M Ellie ,7,F Bob,8,M Carlos,2,M Then press ‘ ESC’ key then type ‘ :wq! ‘ to save grunt> names = load ‘/home/hduser/datasets/names’ using PigStorage (‘,’) as ( n1:charrray,n2:int,n3:chararray); grunt> DUMP names; Rupak Roy
- 17. grunt> ranked = RANK names; grunt> dump ranked; (1, Zara,1,F) (2, David,2,F) (3 David,2,T) (4 Alan,2,M) (5, Calvin,2,M) (6, Alan,5,M) (7, Chris,8,M) (8, Ellie ,7,F) (9, Bob,8,M) (10,Carlos,2,F) We can also implement rank using two fields, each one with different sorting order. grunt> ranked2 = RANK names by N1 ASC, N2 DESC; grunt> dump ranked2; Rupak Roy
- 18. Sometimes we might encounter the RANK has been assigned to 2 fields or 2 records with a same rank. To overcome the issue we have a small function call DENSE grunt> rankedG = RANK names by N1 DESC, N2 ASC DENSE; (1,Zara,1,F) (2,Elie,7,F) (3,David,2,F) (3,David,2,T) (4,Chris,8,M) (5,Carlos,2,F) (6,Calvin,3,M) (7,bob,8,M) (8,Alan,2,M) (9,Alan,5,M) Rupak Roy
- 19. Next We will learn UDF (User Define Function). Rupak Roy