Accelerating Big Data Analytics with Apache Kylin
- 1. Accelerate big data analytics with Apache Kylin Shaofeng Shi, 史少锋 Apache Kylin committer & PMC
- 2. • Shaofeng Shi， 史少锋 [email protected], [email protected] • Apache Kylin committer & PMC, joined Kylin project since 2014 in eBay; • Now chief architect at Kyligence Inc. About me
- 3. • Apache Kylin background • Why OLAP cube is needed for big data • How Kylin build/persist/query cube on Hadoop • Performance benchmark • Use cases Agenda
- 4. • Huge amount of data be collected today. • Transactions, server/app logs, mobile/IoT, etc; • Hadoop is the de facto standard for big data. • Hadoop was not designed for low-latency SQL query. • MapReduce, Hive, Pig, Spark… are for batch processing; • Query performance decreases dramatically as data volume increases. User experience is bad. • Challenge: how to keep high performance data analytics as data grows? Background 0 2 4 6 8 10 12 1 2 3 4 5 6 7 8 Query performance Data volume
- 5. • Massive Parallel Processing (MPP) and SQL on Hadoop technologies tries to solve the problem. • Amazon Redshift, Greenplum, Presto, Impala, Spark SQL, etc; • MPP accelerate queries in the following ways: • Distribute data into multiple nodes, processing in parallel; • Optimize I/O with column-oriented storage, compression encoding, etc; • In-memory processing; MPP and SQL on Hadoop
- 6. • Performance • Handling massive data costs much resource; When resource is insufficient, performance is downgraded. Latency from tens of seconds to tens of minutes. • Concurrency • CPU/memory/network intensive, hard to serve multiple concurrent users; • Scalability • Data need be redistributed when scale out, taking hours; • Master node becomes a bottleneck as cluster size increase. Cluster size is limited to 1-2 hundreds; MPP’s limitation
- 7. A real OLAP solution for big data High performance High scalability Standard complaint High concurrency + Ease of use
- 8. What is Apache Kylin Data analytics Hive / Kafka / RDBMS Apache Kylin MR/Spark HBase Interactive Reporting Dashboard OLAP ／ Data mart Hadoop - Build OLAP cube on Hadoop - Support TB to PB data - Sub-second query latency - ANSI-SQL - JDBC / ODBC / REST API - BI integration - Web GUI - LDAP/SSO Apache Kylin: Extreme OLAP Engine for Big Data https://kylin.apache.org
- 9. OLAP cube • OLAP cube is a data structure optimized for very quick data analysis. • OLAP cube has been adopted by traditional Data Warehouses for decades. • For analyst, the cube concept is easier to be accepted than others. • Why not use cube for big data OLAP? Picture is from https://www.guru99.com/online-analytical- processing.html#4
- 10. Benefits from cube for big data OLAP Cube Classification, aggregation, and sorting ü Performance: querying cube can be 1000x faster than querying raw data; ü Ease of use: cube is topic oriented, analysts can self-serve; ü Cost efficiency: build once, use many times. Much computing resources can be saved. ü Feasibility: with Hadoop, building cube for a large dataset becomes easy. Source data
- 11. Apache Kylin is built with mainstream technologies Apache Kylin Data Analyst, BI Tools, Web App… 4. Send SQL query Online data flow Offline data flow 6. Scan & filter cube 1. Extract data to Hadoop 2. Build the cube 3. Load 5. Parse, optimize & rewrite query
- 12. Kylin can build all cuboids your want • One cube has 2^N cuboids; (N = dimension number) • Cuboid = Materialized view; • All cuboids are consistent (updated atomically); • User can define partial cube with rules.
- 13. • Kylin automatically generate the cubing jobs, and then execute/track them. How Kylin build cube on Hadoop
- 14. • Overall cubing steps: Ø Extract source data from Hive/Kafka/RDBMS to HDFS; Ø Build dimension dictionaries for encoding; Ø Build cuboids with MapReduce or Spark; Ø Convert cube to HBase format; Ø Load into HBase. How Kylin build cube on Hadoop (cont.) By-layer cubing
- 15. • Cube is composed by: • Cuboids • Dimensions • Metrics (measures) • Cuboid ID + Dimension values à row key • Metrics à HBase column value How Kylin persist cube into HBase Logical table for cuboid 00011111
- 16. How Kylin query cube with SQL select l_returnflag, o_orderstatus, sum(l_quantity) as sum_qty, sum(l_extendedprice) as sum_base_price from v_lineitem inner join v_orders on l_orderkey = o_orderkey where l_shipdate <= '1998-09-16' group by l_returnflag, o_orderstatus order by l_returnflag, o_orderstatus; Sort Aggr. Filter TablesO(N) Join Parse SQL to an execution plan • Kylin uses Apache Calcite as the SQL parser and optimizer;
- 17. How Kylin query cube with SQL (cont.) Sort Aggr. Filter Tables Join Sort Cube Filter Pick the best matched cuboid O(1) Rewrite toThese steps have already been completed in the cube build. • Kylin optimize and adapt the plan to OLAP cube. • With less processing, return result instantly.
- 18. • Translate cube query into HBase table scan; • Group by, Filter -> Identify the best cuboid • Filters -> HBase scan ranges (row key start/end), fuzzy row filter • Aggregations -> Measure columns • Scan HBase table and translate HBase result into cube result; • HBase Coprocessor is used for condition pushdown and region side aggregation. • HBase Result (row key + col value) decodes to cube result (dimensions + measures) • Let Calcite do the final round processing (filtering, sorting, grouping, window, etc). How Kylin query cube with SQL (cont.)
- 19. Performance benchmark Tested on SSB dataset, 200X to 1700X faster than Apache Hive As data size increase, the response time keeps stable KAP was the name of Kyligence release.
- 20. Ø Build a partial cube by manual (with rules) or automatically (cube planner); Ø Support UHC dimension (e.g., cell number, cardinality > 100 million); Ø Support precisely count distinct on UHC (with bitmap), and roll up at any level; Ø Support fully and incrementally data load, also allow refreshing history data; Ø Support Kafka and RDBMS (MySQL, Redshift etc) as data source; Ø Multi-nodes deployment for HA; Ø Read-write separation deployment (dedicated HBase) for high performance; Ø The Real-time OLAP (v3.0) is at alpha now; Advanced features
- 21. ü Dashboard, reporting and business intelligence on big data (support Tableau, Cognos, MSTR, Qlik, Superset, Zeppelin…) ü Multi-dimensional data exploration ü Traditional data warehouse offload/migration to Hadoop ü User behavior analysis (PV, UV, retention, funnel, etc) ü Transparent query acceleration Scenarios for Apache Kylin
- 22. • The biggest O2O service provider in China • Catering takeaway, rating, hotel, travel, bike (mobike), etc; • More than 300 millions active users and 3 million merchants; • Apache Kylin was selected as the central OLAP platform since 2015, serving thousands of business analysts from all business lines. • Till Aug, 2018, total data row in Kylin is 8.9 trillion，cube storage is 971 TB. • 3.8 million queries per day; 50% queries < 200ms; 90% queries < 1.2s Use case: Meituan & Dianping
- 23. • Yahoo! Japan’s online shop data analysis. • Use Kylin to replace Impala, fulfill the low latency requirement to business analysts; • Query latency reduced from 1 minute to <1s; • With Kylin’s cross-region deployment, cube instead of raw data is pushed to the DC that nearby the analysts, saved the bandwidth； • Read more: https://techblog.yahoo.co.jp/oss/apache-kylin/ Use case: Yahoo! Japan
- 24. • About Telecoming • Mobile payments (Direct Carrier Billing) and digital marketing company in Spain. • To meet the demand of analytics over the large volumes of data, deployed Hadoop and Kylin to support interactive analytical queries. • “Thanks to this new architecture, Telecoming has improved the quality and productivity of its decision-making processes, which translates into better results for their business.” • Read more: http://www.stratebi.com/-/big-data- marketing-telecoming Use case: Telecoming
- 25. • About OLX Group • A global online marketplace, headquartered in Amsterdam, owned by Naspers, operating in 45 countries; • Built DW with Amazon Redshift in the past; Redshift couldn’t scale, moving to data lake on S3; • Need a solution to enable analyst users doing interactive analysis with Tableau. • Tried several solutions; finally, Kylin + EMR well solved the challenge! • “Kylin is the game changer with its extreme fast performance and seamless integration with Tableau.” Use case: OLX Group Picture from https://www.olx.co.id/
- 26. • Apache Kylin home: https://kylin.apache.org/ • Apache Kylin source code: https://github.com/apache/kylin • Join Kylin mailing list: [email protected] and [email protected] • Kylin meetup: https://www.meetup.com/Apache-Kylin-Meetup/ • Need enterprise solution? contact Kyligence: https://kyligence.io/ or [email protected] Useful links
- 27. See a demo?
- 28. Thank you! 谢谢！