Hybrid Data Warehouse Hadoop Implementations
26 likes7,730 views
The document discusses the evolving relationship between data warehouse (DW) and Hadoop implementations. It notes that DW vendors are incorporating Hadoop capabilities while the Hadoop ecosystem is growing to include more DW-like functions. Major DW vendors will likely continue playing a key role by acquiring successful new entrants or incorporating their technologies. The optimal approach involves a hybrid model that leverages the strengths of DWs and Hadoop, with queries determining where data resides and processing occurs. SQL-on-Hadoop architectures aim to bridge the two worlds by bringing SQL and DW tools to Hadoop.
Ad
More Related Content
What's hot (20)
Viewers also liked (20)
Ad
Similar to Hybrid Data Warehouse Hadoop Implementations (20)
Ad
More from David Portnoy (7)
Hybrid Data Warehouse Hadoop Implementations
- 1. The future of hybrid Data Warehouse-Hadoop implementations - - David Portnoy Datalytx, Inc. 312.970.9740 http://LinkedIn.com/in/DavidPortnoy - - - © Copyright 2013 David Portnoy and Datalytx, Inc. - - -
- 2. Why this topic? Note on terminology used In this context RDBMS (relational databases) are synonymous with DW (data warehouses) Data Warehouse vendors are evolving to incorporate the best Hadoop has to offer. Similarly, the Hadoop ecosystem is growing to include capabilities previously available only to large scale (MPP) DW platforms. Understanding the trends and alternatives helps your organization identify the most effective long term solution Launched in TDWI forum on LinkedIn (See http://bit.ly/Hybrid-DW-Hadoop) Who are the winners in the race for the ultimate hybrid DBMS-Hadoop implementation? As described in http://bit.ly/Hybrid-RDBMS-Hadoop, the industry is going to a hybrid DBMS-Hadoop model that leverages the best of both worlds. (Microsoft for example is building its Polybase with cost based query optimization that decides whether to push processing to the Hadoop data nodes or the PDW compute nodes.) Which vendors do you see as the current leaders in this race? And for the visionaries and philosophers among you... How do you see it ultimately shaking out? This group extends the TDWI community online and is designed to foster peer network and discussion of key issues relevant to business intelligence and data warehousing managers. TDWI (The Data Warehousing Institute ™) provides education, training, certification, news, and research for executives and information technology (IT) professionals worldwide. Founded in 1995, TDWI is the premier educational institute for business intelligence and data warehousing. Our Web site is www.tdwi.org.
- 3. Relationship between Hadoop & DW implementations To leverage the strengths in each platform, traditionally... Hadoop is used for storage & transformation (specifically, ELT) of vast volumes of raw data ...while DW is used for analytics on a subset of the processed data Extract from source systems Load to DW Transform in place DW Reporting & OLAP using traditional tools
- 4. Why do DW vendors care about Hadoop? ...And why not just ignore it as a special use case solution? 1. Compelling price point to store high volume data, especially if it’s not needed for real-time access 2. Has become the de facto standard for ambitious big data projects 3. HDFS and MapReduce are becoming mature and stable technologies (in development since 2005), despite the fact that the rest of the ecosystem is still rapidly evolving 4. DBMS vendors have been missing a scalable distributed file system, such as HDFS, to provide capability to store and manipulate variable and unstructured data
- 5. Do companies with an MPP DW still need Hadoop? For companies that have an MPP (Massively Parallel Processing) data warehouse, such as Teradata or Netezza... Couldn’t the DW platform do everything Hadoop could do? Yes and no. 1. Yes, you can store a lot of unstructured data in text fields, simulate operations of key-value pairs, and scale processing capacity horizontally, just like Hadoop 2. But the types of processing that can be done against the DW is more limited than with Hadoop. (Although some DW vendors now allow open source tools, including MapReduce and R, to crunch the data.) 3. And the cost of storing high volumes of data – especially for low usage frequency and high latency operations – is much lower for Hadoop. So adding Hadoop to the mix can keep the size of the more expensive DW platform in check.
- 6. What paths are DW vendors taking? DW vendors can choose from 3 typical paths for responding to competing technologies 1. Ignore: Hope it’s a fad that goes away None of the major vendors see this as a viable option 2. Reactionary: Interoperate with existing Hadoop products This seems to be the most common path for established commercial vendors 3. Proactive: Embrace Hadoop and contribute to extending the ecosystem This seems to be the approach for new entrants competing in a targeted niche
- 7. What will happen to major DW vendors? It’s safe to say that major commercial vendors like IBM, Oracle, Teradata and Microsoft will continue to be key players. 1. Each one already has a product roadmap involving some way of responding to or incorporating the Hadoop ecosystem. 2. Many of the most successful new entrants will be acquired by these vendors and incorporated into their product lines. 3. This pattern can me seen in the historical evolution of revolutionary or niche technologies, such as columnar databases, in-memory databases, and self-service BI capabilities.
- 8. Partnerships & Alliances Most large commercial DBMS vendors have partnerships with specific Hadoop distribution developers. Oracle (Big Data Appliance) IBM Netezza Microsoft (HDInsight) Teradata & AsterData appliance Greenplum (Prior to GreenplumHD)
- 9. Possible phases of a DW platform to evolve into a hybrid There are many ways to get to the end goal, but here’s a possible evolution path for commercial DW vendors to hybrid DBMS-Hadoop solutions Phase Description Independent DW and Hadoop operate independently, storing completely different data sets depending on size and structure and processing them in completely different ways. Batch data movement There’s an efficient method to shuttle data back and forth between DW and Hadoop. Focused primarily on loading a subset of data into the DW for analytics. Transformations typically happen in context of ELT, rather than ETL. Integrated storage Queries are issued against the system, which in turn determines where the requested data resides – DW or Hadoop. If data resides on Hadoop, it pulls data into DW prior to executing a query against it. Optimized processing in place Queries are issued against the system, which in turn determines where the requested data resides – DW or Hadoop. If data resides on Hadoop, the query is executed in place within Hadoop, possibly after converting the logic to MapReduce. The result is brought back to the DW.
- 10. How about Hadoop distribution vendors? Hadoop distribution vendors like Cloudera, Hortonworks and MapR are developing products to add DW capabilities and bridge the gap between the two worlds Their solution: Improve on limitations of MapReduce Improve on data silos and overhead of moving data between DW and Hadoop
- 11. What’s wrong with MapReduce? MapReduce is the legacy and most pure processing environment for Hadoop. But it’s not ideal for a number of reasons Performance: Long lag between query and results make it difficult for interactive analytics Requires high degree of skill with Java for processing data. Existing resources with SQL skills would be underutilized. Doesn’t leverage existing company investment in ETL, reporting, analytics tools
- 12. Why not use DW-Hadoop connectors? The original approach of integrating DW with Hadoop using connectors to move data back and forth is not ideal. It introduces costs and inefficiencies of dealing with data silos. To solve this problem, a more practical “SQL-onHadoop” architecture is being adopted. Typically, SQL-on-Hadoop capabilities include: Interactive analytical queries (readonly) Parallelism / distributed processing Efficient joins across multiple tables ANSI SQL Compliance Query caching Ability to use existing ETL, OLAP and reporting tools from commercial vendors SQL-on-Hadoop players: Cloudera’s Impala Hortonworks’ Stinger (faster Hive via ORCfile & Tez) Apache Drill (supported by MapR) Hadapt / HadoopDP Greenplum’s HAWQ (on Pivotal HD) Teradata’s SQL-H (on Aster/PostgreSQL)
- 13. DW & Hadoop vendors approach from opposite directions Relational DBMS Add interactivity Hybrid DW vendors start with their relational DBMS platform... and add interactivity to Hadoop Add DBMS features Initially all processing might occur on DBMS, with Hadoop being used for storage Ultimately evolves to pushing processing to Hadoop cluster Examples: Microsoft PDW / Polybase, Greenplum / Pivotal HD Hadoop vendors start with their Hadoop distribution... and add DBMS features This is also known as “SQL-onHadoop” Add query optimizer, real-time capabilities, etc. Examples: Cloudera Impala, Hortonworks Stinger, Hadapt In the end both of these approaches might end up with very similar of hybrid solutions.
- 14. Utopian vision Ultimately, ideally the user doesn’t know (and doesn’t care) where the data is stored and how it’s processed, as made possible by using... Single toolset: Use single set of ETL, reporting and analytics tools, regardless where data resides Automated optimization: DW automatically decides between RDBMS and Hadoop... Where to store data, based on its structure and directives on how it’s to be used Where to push processing, based on query cost optimization
- 15. Evolution of SQL-on-Hadoop: Operational data store Eventually, Hadoop implementations might evolve into supporting “operational” transactions Ability to handle workloads to power websites and applications Transaction-orient write capability, rather than read-only of analytical queries “ACID” database capabilities, including concurrency, distributed transactional support, and guarantees of data consistency
- 16. Microsoft's Approach Microsoft's strategy involves the Polybase initiative and ability to leverage its extensive range of BI tools Polybase is the hybrid RDBMS-Hadoop platform, which spans queries across HDFS on HDInsight (Hortonworks’ distribution that runs on Windows) and Microsoft’s MPP DW platform, SQL Server PDW (Parallel Data Warehouse) Polybase development is phased: Phase 1: Parallel data transfer between SQL Server Compute Nodes and Hadoop Data Nodes, but all processing is done on DBMS Phase 2: Use query optimizer to decide where to process jobs. Selectively push work to Hadoop by converting queries to MapReduce Integration with Microsoft’s BI stack, including: Reporting (SQL Server Reporting Services) OLAP (SQL Server Analysis Services) Self-service BI (Excel, PowerPivot, Power View)
- 17. Microsoft's Approach (cont.) The advantages of going the Microsoft route are numerous... Make use of Hadoop using familiar high productivity self-service BI tools Excel itself can handle data extracts from Hadoop PowerPivot for large-scale data exploration using xVelocity in-memory analytics engine Power View for ad-hoc visualization in SSRS, accessible via SharePoint or Excel In Office 2013, PowerPivot and Power View are natively integrated with Excel Leverage existing and widely available .NET developers Management of a Hadoop cluster (using Apache Ambari) is integrated with Microsoft System Center, already used by IT Operators for database management Deliver tighter security through integration with Windows Server Active Directory Cloud-based Hadoop available through Windows Azure HDInsight Service Interactive access to Hadoop through Hortonworks' Hive ODBC driver
- 18. Microsoft's Approach (cont.) And some of the disadvantages include... Licensing costs for both Windows Server that run HDInsight nodes and SQL Server nodes associated with Polybase Uncertain performance and adoption of HDInsight distribution Many of the advantages in integration and leveraging resources don’t apply for non-Microsoft shops
- 19. Greenplum’s Pivotal HD Greenplum’s Pivotal HD implements But... Greenplum on top of HDFS Proprietary technology means Yet still capable of running MapReduce vendor lock-in and jobs if needed inability to take advantage of a vibrant developer community More mature than many of its rivals Capabilities extend well beyond those of Licenses are expensive and... open source distributions Open source alternatives (Impala, Drill, Shark) are becoming available
- 20. Example SQL-on-Hadoop: HadoopDB / Hadapt Architecture of Hadapt Hadapt is a commercialized version of Daniel Abadi's HadoopDB project For structured data, each Data Node uses DBMS (Postgres or VectorWise) instead of HDFS Load balancing and performance optimization on nodes Hive
- 21. Example SQL-on-Hadoop: BigSQL BigSQL is PostgreSQL implemented on Hadoop