How Auto Microcubes Work with Indexing & Caching to Deliver a Consistently Fast Business Intelligence Experience on Hadoop
Download as PPTX, PDF1 like454 views
Jethro CTO Boaz Raufman and Jethro CEO Eli Singer discuss the performance benefits of adding auto microcubes to the processing framework in Jethro 2.0. They discuss how the auto microcubes working in tandem with full indexing and a smart caching engine deliver a consistently interactive-speed business intelligence experience across most scenarios and use cases. The main use case they discuss is querying data on Hadoop directly from a BI tool such as Tableau or Qlik.
Ad
More Related Content
What's hot (20)
Viewers also liked (12)
Ad
Similar to How Auto Microcubes Work with Indexing & Caching to Deliver a Consistently Fast Business Intelligence Experience on Hadoop (20)
Ad
Recently uploaded (20)
![Download Wondershare Filmora Crack [2025] With Latest](https://cdn.slidesharecdn.com/ss_thumbnails/neo4j-howkgsareshapingthefutureofgenerativeaiatawssummitlondonapril2024-240426125209-2d9db05d-250419-250428115407-a04afffa-thumbnail.jpg?width=560&fit=bounds)
![Pixologic ZBrush Crack Plus Activation Key [Latest 2025] New Version](https://cdn.slidesharecdn.com/ss_thumbnails/fashionevolution2-250322112409-f76abaa7-250428124909-b51264ff-250504160528-fc2bb1c5-thumbnail.jpg?width=560&fit=bounds)
![Get & Download Wondershare Filmora Crack Latest [2025]](https://cdn.slidesharecdn.com/ss_thumbnails/revolutionizingresidentialwi-fi-250422112639-60fb726f-250429170801-59e1b240-thumbnail.jpg?width=560&fit=bounds)
How Auto Microcubes Work with Indexing & Caching to Deliver a Consistently Fast Business Intelligence Experience on Hadoop
- 2. About Jethro SQL Data • What Does Jethro Do? – BI on Big Data acceleration – Reporting, dashboards, discovery, ad-hoc • How It Works? – Indexing and caching server – Combines columnar SQL DB design with search-indexing technology • Partnerships – BI: Tableau, Qlik – Hadoop: Cloudera, MapR, Hortonworks
- 3. SQL on Hadoop – Complimentary Approaches • Hive / Tez • Impala • Presto • SparkSQL • Drill • HAWQ • IBM/Big SQL • Actian • Tajo • … SQL-on-Hadoop Solutions Full-Scan: Read all rows • JethroData JethroData Index-Access: Read ONLY needed rows Comparison: Full-Scan: Optimal for predictive & reporting Index-Access: Optimal for interactive BI
- 4. What Is Jethro for BI Tools? An indexing & caching server • BI tool uses live DB access – Sends SQL queries via ODBC / JDBC • Jethro key performance features 1. Full indexing – every column is indexed 2. Result cache – every query is cached 3. Auto Cubes – every repeatable pattern • Everything stored in Hadoop – Cache, aggregations, index & column files, … • Incrementally updated – Every day / hour / min Live Access HDFS BI Tool
- 5. Jethro: Enabling Unlimited Interactive BI for Big Data Unlimited Interactive Big Data
- 6. Low Conccurency Interactive Slow MPP speed = more resources Jethro: Hi Speed Low resources Hi Conccurency Interactive BI requires both speed and conncurency Faster • Indexes • Hi Performance Execution • Results cache • Auto Cubes
- 7. Indexes – boosting filtered queries • Indexes everything – every column, every value • Filtering (where clause expression) is done against the indexes • The more you filter the faster you get the results – execution time depends on size of scanned data set • Resources required per query are order or manganite lower which enable high concurrency
- 8. What to do when indexes are not enough? • The Challenge: How to provide interactive response time (seconds) for use cases that include wide queries with little or no filtering • Our Approach: Add CUBES technology which is complimentary to INDEXES • Jethro rule: Make this absolutely seamless to the user
- 9. Traditional OLAP Cubes – The Short Story Cube: Select City, Item, Year, sum(sold_price) from Sales group by City, Item, Year Queries that can use the cube: Select City, Item, sum(sold_price) from Sales where Year=2016 group by City, Item Select Item, sum(sold_price) from Sales group by Item City Item Year sum(sold_price) NY iPhone7 2016 $50,000 NY Samsung7 2016 $40,000 NY iPhone6 2015 $42,500 LA iPhone7 2016 $70,000 LA Samsung6 2015 $35,000
- 10. Traditional OLAP Cubes • Performance: Fast response time for queries that hit the cube • Concurrency: Low resource footprint per query enabling high concurrency • Use Case: Works great for static query environment Not suitable for dynamic environments that support self service and complex dashboards
- 11. Traditional OLAP Cubes: Challenges • Hard to implement: Manually pre-defined, requires specialized tools and expertize • Resources consuming: Heavy processing on cubes creation that can effect global system performance • Operational overhead: keeping cube up to data with source data is time and resource consuming • Use case limitations: Size limitation and operational limitations that make it practically impossible to use for many use cases, Such as: – Large number of dimensions – High cardinality dimension – Count distinct aggregators – Complex expressions – Many different queries
- 12. How to have your cake and eat it too • Auto generated cubes – Cubes are automatically generated in the background based on actual user interaction – No expertise, no specialize tools, no pre design – Unlimited access to the data • Micro Cubes – Many Micro cubes instead few gigantic cubes – Easily support many different queries • Incremental – Auto cubes are incremental and automatically updated – Zero operational overheads – Stable performance unaffected by ongoing new data streaming
- 13. How to have your cake and eat it too, cont. • Complex queries normalization – Rewrite complex queries to reuse simplified common query blocks – Increase cubes reusability • Optimized for count distinct – Handling for count distinct using values bitmaps – Handle count distinct without hitting cube size limitation • Complementary to indexes – Use indexes for large number of filters or hi cardinality dimensions: – Maintain stable interactive performance by utilizing complementary index and cubes
- 14. Jethro Query Processing Flow Query Arrives Query Match ? Cube Match ? Process Query (Indexes, Columns, MT execution) Optimal for Cube? Cache Results Generate Cube Response from Results Cache Response from Cubes reply reply reply No No Yes Yes
- 16. DEMO • TCP-DS data set • Single table: 1.2 billion rows • Multi tables: 1.6 billion rows fact • 2 Jethro nodes (AWS r3.4xl) over EFS • BI: Tableau