Interactive query using hadoop
Download as PPTX, PDF0 likes384 views
Proof of Concept that details differences between Tez, Hive, Impala as it pertains to different complex queries being executed by business.
Ad
More Related Content
What's hot (20)
Similar to Interactive query using hadoop (20)
Ad
Recently uploaded (20)
Ad
Interactive query using hadoop
- 1. Interactive Query in Big Data Ecosystem ARVIND RADHAKRISHNEN
- 2. Why do we need Big Data Systems? • Enterprise business needs are changing • Provisioning data & reporting on near real time are becoming important • Increasing focus on creating “Data Lake” in enterprises • Enabling Data as Service for Business Analytics • Intelligent Decision Making through combining various data sources
- 3. Use Case #1 • Enable Client Centric Analysis through combining • Consumer to Business (C2B) transactions across lines of businesses that will help understand “Consumer Cash Flows” • Capture Business to Business (B2B) transactions • Analyze spending patterns • Get a better view of consumer ability to pay back on obligations • Create personalized offerings for cross selling Business Analytics
- 4. Use Case #2 • Improve Credit Risk Management • Enable full spectrum view of borrower • In most credit reports, data is refreshed only once in 60-90 days • Access additional details pertinent to property, borrower’s undisclosed liens, property & tax liens, judgments & child-support obligations. • Additional Appraisal related details available via external sources like Corescore-FICO. Business Analytics
- 5. How do we enable Business Analytics? • Big DataTechnology Solutions for accomplishing BusinessAnalytics: • Hive (Map Reduce as Execution Engine) • Tez as an Execution Engine over Hive (Open Sourced by Hortonworks) • Impala (Open Sourced by Cloudera) • Drill (Provided by MapR) • Presto (Open sourced by Facebook) • All these solutions allow ANSI SQL queries to be executed with minimal modifications • Most of the Oracle/Netezza/Teradata/SQL Server Datatypes are now supported • They help create table/view over data residing on HDFS, with schema metadata stored on Hive Metastore or alternate DBs
- 6. How does Tez enable Interactive querying? • Built onYARN – resource management framework for Hadoop • Enables pre-hot containers for executing queries • Run time optimization for task scheduling & concurrency • Enables Cache for Hot data records • Eliminates HDFS read/write during intermediate stages as used in MR
- 7. How does Impala enable Interactive querying? • Custom Execution Engine written in C++ & circumvents MapReduce • Enables In Memory Execution (aggregation & right hand inputs of joins are cached) • Query/Data Flow: • Uses distributed service (impalad) that runs on every data node • Query is received by Query Planner, which then sends plan fragments to different data nodes for execution • Query Coordinator initiates execution on impalad nodes. • Intermediate results are streamed between impalad nodes • Final result is streamed back to client by Query Co-ordinator
- 8. Case Study • We performed a POC to determine: • Can Big Data Ecosystem support the interactive queries that are being performed by business teams in production environment for a large Fortune 50 company? • What would be the scalability, performance metrics of Big Data Solutions compared with an Industry standard MPP?
- 9. Case Study Customer Transaction History HDFS Files (CustomerTransaction) Impala Query 1 Query n Cloudera Cluster HDFS Files (CustomerTransaction) Hive/Tez Query 1 Query n Hortonworks Cluster Steps Involved: • Customer Transaction History File (25 Million records) is generated and ingested into HDFS • External Tables are created on Impala over HDFS file • Tez is enabled on Hortonworks cluster and external table is created over HDFS file containing same data as cloudera cluster • Query is executed on same data structure on both clusters using Hue and Hive Shell Scope of POC: Model Business Transaction Records (25 million records) in Lab environment, setup Hadoop Cluster and verify Query compatibility and performance. What did we do? We setup 2 different clusters – 1 with Cloudera (for Impala) and 1 with Hortonworks for Tez/Hive
- 10. Queries that were run for POC • The below were the different types of Queries that were run against the MPP Database and Hadoop Ecosystem (Tez/Hive, Impala) • Count(*) • Count(*) by a column • Count of Distinct SubQuery • Cartesian Joins, Aggregation, Subquery and union all (With Filter) • Cartesian Joins, Aggregation, Subquery and union all(without Filter) • SubQuery & Aggregation • Cartesian inner join with a date filter on one table • Cartesian left outer join with date filter on one table • Cartesian right outer join with date filter on one table • Cartesian full outer join with date filter on one table
- 11. What did we observe? • The interactive queries that were run in business environment were executed in both Impala &Tez/Hive without any modifications • We saw that after enabling compression, the performance improved in Impala • The Query Response time in Big Data solutions was almost similar or better to MPP database
- 12. Performance Stats *This view should not be used for benchmarking.This was done on a cluster configuration not provided or approved by Hortonworks or Cloudera.We have not completed the comparison with compressed files on Hive/Tez.
- 13. Thank you! Reach me at: Twitter: @arvindrkrishnen