Using SQL-MapReduce for Advanced Analytics
Download as PPTX, PDF7 likes3,976 views
This document discusses using SQL-MapReduce (SQL-MR) for advanced analytical queries. SQL-MR allows parallelization of complex SQL queries using MapReduce functions. It simplifies architectures by eliminating the need for separate data warehouses, datamarts and cubes. SQL-MR enables new forms of analytics like deep, complex, operational and self-service analytics without restricting queries. The marriage of SQL and MapReduce offers significant potential by parallelizing analytical logic processing.
Ad
More Related Content
Viewers also liked (14)
Similar to Using SQL-MapReduce for Advanced Analytics (20)
Ad
More from Teradata Aster (20)
Ad
Recently uploaded (20)
Using SQL-MapReduce for Advanced Analytics
- 1. Using SQL-MapReduce for Advanced Analytical QueriesbyRick F. van der LansR20/Consultancy BV
- 2. What Did the Users Want?BI reportsProduction databases
- 3. But What Did We Create?ODSdatawarehousedatamartproductiondatabasecube
- 4. Problems with Current DW Platforms45%40%39%37%33%29%23%23%21%20%19%16%16%15%14%13%11%4%3%Poor query responseCan’t support advanced analyticsInadequate data load speedCan’t scale to large data volumesCost of scaling up is too expensivePoorly suited to real-time or on demand workloadsCurrent platform is a legacy we must phase outCan’t support data modeling we needWe need platform that supports mixed workloadsCan’t support large concurrent user countInadequate high availabilityInadequate support for in-memory processingInadequate support for web services and SOACurrent platform is 32-bit, and we need 64-bitCurrent platform is SMP, and we need MPPWe need platform better suited to cloud or virtualizationCan’t secure the data properlyOtherNo problemsSource: P. Russom, ‘NextGeneration Data WarehousePlatforms’, TDWI BestPractices Report, fourthquarter 2009.
- 5. 49%8%20%12%8%1%1%3%current DW platform2009201020112012201320142015 or laterNeed for More Powerful Data Warehouse Platformsno plans to replaceSource: P. Russom, ‘NextGeneration Data WarehousePlatforms’, TDWI BestPractices Report, fourthquarter 2009.
- 6. New Forms of AnalyticsAdvanced AnalyticsOperational AnalyticsDeep AnalyticsSelf-Service AnalyticsComplex AnalyticsAutomated Analytics
- 7. Positioning of Advanced Analyticscomplexityof analyticalquerieshighcomplex queries onsmall to medium size databasesadvancedanalyticssimple queries onsmall to medium size databasessimple queries onlarge to ultra large databaseslowdatabase sizelowhigh
- 8. Parallellization of SQLWorkerWorkerWorkerSELECT *FROM CUSTOMERSWHERE LOCATION = 'New York'DatabaseServerMaster
- 9. How Easy Is Parallelizing SQL Queries? (1)Example 1:SELECT ID, SALES_DATE, PRICEFROM SALES_RECORDSWHERE PRICE > 100Example 2:SELECT REGION_ID, SUM(PRICE)FROM SALES_RECORDSWHERE PRICE > 100GROUP BY REGION_ID
- 10. How Easy Is Parallelizing SQL Queries? (2)Example 3: Get all the flights to London for which another flight exists to London that leaves within an hour on the same day.SELECT *FROM DEPARTURES AS D1WHERE DESTINATION = 'London'AND DEPARTURE_TIME + 60 MINUTES >= (SELECT MIN(DEPARTURE_TIME) FROM DEPARTURES AS D2 WHERE DESTINATION = 'London' AND D2.DEPARTURE_TIME > D1.DEPARTURE_TIME AND D2.DEPARTURE_DAY = D1.DEPARTURE_DAY)ORDER BY DEPARTURE_TIME
- 11. How Easy Is Parallelizing SQL Queries? (3)SELECTA.PROD_DESC AS ITEM1,B.PROD_DESC AS ITEM2,C.PROD_DESC AS ITEM3,COUNT (*) AS CNTFROM(SELECT SF.STORE_ID, SF.REG_ID, SF.TRAN_NO, SF.ITEM_ID, SF.DT, PD.PROD_DESC, PD.PRICE FROM SALES_FACT SF INNER JOIN PRODUCT_DIM PD WHERE SF.ITEM_ID=PD.ITEM_ID) AS TRANSACTIONS A, (SELECT SF.STORE_ID, SF.REG_ID, SF.TRAN_NO, SF.ITEM_ID, SF.DT, PD.PROD_DESC, PD.PRICE FROM SALES_FACT SF INNER JOIN PRODUCT_DIM PD WHERE SF.ITEM_ID=PD.ITEM_ID) AS TRANSACTIONS B,(SELECT SF.STORE_ID, SF.REG_ID, SF.TRAN_NO, SF.ITEM_ID, SF.DT, PD.PROD_DESC, PD.PRICE FROM SALES_FACT SF ,,, PRODUCT_DIM PD WHERE SF.ITEM_ID=PD.ITEM_ID) AS TRANSACTIONS C WHERE A.STORE_ID=B.STORE_ID AND B.STORE_ID=C.STORE_ID AND A.STORE_ID=C.STORE_ID AND A.REG_ID=B.REG_ID AND B.REG_ID=C.REG_ID AND A.REG_ID=C.REG_ID AND A.TRAN_NO=B.TRAN_NO AND B.TRAN_NO=C.TRAN_NO AND A.TRAN_NO=C.TRAN_NO AND A.DT=B.DT AND B.DT=C.DT AND A.DT=C.DT AND A.ITEM_ID<>B.ITEM_ID AND A.ITEM_ID<>C.ITEM_ID AND B.ITEM_ID<>C.ITEM_IDGROUP BY A.PROD_DESC, B.PROD_DESC, C.PROD_DESCHAVING COUNT(*)>1000ORDER BY COUNT(*) DESC;Example 4: Market basket analysis:
- 12. Declarativeness and Storage IndependencyDeclarativeness: The developer has only to program what has to be done, and not how it should be done.Storage independency: The language should hide how data is physically stored and how it is accessed.
- 13. Advantages of Two PropertiesProductivity increaseless code has to be writtenMaintainability: less code means having to maintain less codeFlexibility: changes to the storage layer can be made without the need to change the SQL code in the reports
- 14. Different Types of SQL FunctionsBuilt-in or User-definedSELECT FLIGHT, TRUNCATE(DEPARTURE_TIME, MINUTES)FROM DEPARTURES AS D1WHERE BANK_HOLIDAY(DEPARTURE_TIME) = 1Scalar or TableSELECT AVG(DURATION)FROM LAST_FIVE_ROWS(DEPARTURES)Pure SQL, Procedural, or ExternalSimple or Complex
- 15. MapReduceMapReduce is a programming model introduced by GoogleAimed at processing requests on large data sets where the processing can be distributed over a high number of nodes using parallel capabilities Two steps Map and ReduceMap is like SelectReduce is like Group-by
- 16. Aster Data’s SQL-MapReduce (1)SQL-MR is a set of built-in and user-defined external table functionsExample:SELECT *FROM GET_NEXT_FLIGHT_1HR (ON DEPARTURES PARTITION BY DESTINATION)WHERE DESTINATION = 'London'ORDER BY DEPARTURE_TIMEAll the SQL-MR function processing is parallelizedIncluding complex group-by operations and time-series analytics
- 17. Aster Data’s SQL-MapReduce (2)An SQL-MR function can contain the most complex analytical logicProgrammers of SQL don’t need to learn a new language, Java, C++, Python, and many more can be usedThe SQL statements invoking SQL-MR functions are still declarative and storage-independentThe functions themselves are notUsable by any BI tools supporting SQL
- 19. SQL-MR Technical AdvantagesTechnical DisadvantagesParallelization of complex operations
- 21. Efficiency of low-level programming language
- 26. Polymorphism of the functions
- 27. Nesting of the functions
- 28. Small group of developers have to learn a new language (possibly)
- 29. Low-level language is not declarative
- 30. Non-portable functions Market Basket Analysis using SQL-MRSELECT PROD_DESC1, PROD_DESC2, PROD_DESC3, COUNT(*) AS CNTFROM BASKET_GENERATOR( ON ((SELECT SF.STORE_ID, SF.REG_ID, SF.TRAN_NO, SF.ITEM_ID, SF.DT, PD.PROD_DESC, PD.PRICE FROM SALES_FACT SF INNER JOIN PRODUCT_DIM PD WHERE SF.ITEM_ID=PD.ITEM_ID) AS TRANSACTIONS A PARTITION BY STORE_ID, REG_ID, TRAN_NO, DT BASKET_ITEM(‘PROD_DESC') BASKET_SIZE('3') )GROUP BY PROD_DESC1, PROD_DESC2, PROD_DESC3HAVING COUNT(*)>1000ORDER BY COUNT(*) DESC;
- 31. Business Advantages of SQL-MRSimplification of architectureDeep analyticsComplex analyticsOperational analyticsSelf-service analyticsNo forbidden queries
- 33. ConclusionsThe analytical and reporting demands are increasingMost environments already have problems with performanceThe marriage of SQL and MapReduce offers an enormous potentialParallelizing the processing of analytical logic
- 34. Business Advantages of SQL-MRSimplification of architectureDeep analyticsComplex analyticsOperational analyticsSelf-service analyticsNo forbidden queries
- 35. Questions & AnswersRick van der LansR20 Consultancye-mail: [email protected] website: http://www.r20.nlStephanie McReynoldsDirector of Product Marketing, Aster Datae-mail: [email protected] More Information on Aster Data:http: //www.asterdata.com