Data Warehouse - Incremental Migration to the Cloud
4 likes1,834 views
A data warehouse (DW) migration is no small undertaking, especially when moving from on-premises to the cloud. A typical data warehouse has numerous data sources connecting and loading data into the DW, ETL tools and data integration scripts performing transformations, and reporting, advanced analytics, or ad-hoc query tools accessing the data for insights and analysis. That’s a lot to coordinate and the data warehouse cannot be migrated all at once. Using a data replication technology such as Oracle GoldenGate, the data warehouse migration can be performed incrementally by keeping the data in-sync between the original DW and the new, cloud DW. This session will dive into the steps necessary for this incremental migration approach and walk through a customer use case scenario, leaving attendees with an understanding of how to perform a data warehouse migration to the cloud. Presented at RMOUG Training Days 2019
Ad
More Related Content
What's hot (20)
Similar to Data Warehouse - Incremental Migration to the Cloud (20)
Ad
More from Michael Rainey (20)
Ad
Recently uploaded (20)
![PRE-NATAL GRnnnmnnnnmmOWTH seminar[1].pptx](https://cdn.slidesharecdn.com/ss_thumbnails/pre-natalgrowthseminar1-250427093235-de04befc-thumbnail.jpg?width=560&fit=bounds)
Data Warehouse - Incremental Migration to the Cloud
- 1. © 2019 Snowflake Computing Inc. All Rights Reserved DATA WAREHOUSE: INCREMENTAL MIGRATION TO THE CLOUD Michael Rainey | RMOUG Training Days February 2019
- 2. © 2019 Snowflake Computing Inc. All Rights Reserved ABOUT ME Michael Rainey Senior Solutions Architect, Professional Services at Snowflake Computing Oracle ACE Director Twitter: @mRainey Email: [email protected]
- 3. © 2019 Snowflake Computing Inc. All Rights Reserved 3 YEARS IN STEALTH + 3 YEARS GA 1000+ employees Over 2000 customers today Over $850M in venture funding from leading investors First customers 2014, general availability 2015 Founded 2012 by industry veterans with over 120 database patents Queries processed in Snowflake per day: 100 million Largest single table: 68 trillion rows Largest number of tables single DB: 200,000 Single customer most data: > 40PB Single customer most users: > 10,000 Fun facts:
- 4. © 2019 Snowflake Computing Inc. All Rights Reserved CLOUD DATA WAREHOUSING FOR DUMMIES Grab a copy at the Snowflake booth!
- 5. © 2019 Snowflake Computing Inc. All Rights Reserved WHAT IS DATA WAREHOUSING?
- 6. © 2018 Snowflake Computing Inc. All Rights Reserved THE EVOLUTION OF DATA PLATFORMS Data warehousing has evolved over four decades 1980’s 1990’s 2000’s 2010’s Relational database Data warehouse appliance Data warehouse & platform software Built-for-the cloud data warehouse
- 7. © 2019 Snowflake Computing Inc. All Rights Reserved DATA WAREHOUSE DEPLOYMENT OPTIONS Three categories of data warehousing options Packaged Software Appliance A true SaaS data warehouse • Customer installs - purchases hardware separately • Need IT expertise to build and manage the data warehouse • Data warehouse is hosted on hardware • Need to specify disk space and compute resources • Pay for compute and storage as you go • Zero management required
- 8. © 2019 Snowflake Computing Inc. All Rights Reserved TRADITIONAL SOLUTIONS CANNOT SCALE Decentralized, local storage Single cluster Shared-nothing Shared storage Single cluster Shared-disk
- 9. © 2019 Snowflake Computing Inc. All Rights Reserved Source: Kerry Osborne’s keynote at RMOUG Training Days 2019 WHERE WE SPEND TIME
- 10. © 2019 Snowflake Computing Inc. All Rights Reserved WHAT’S NEEDED? A CLOUD DATA WAREHOUSE Current technologies are insufficient Easy Management All Data in One System Skilled staff misutilized on database management; difficult to add new users, workloads or data Data locked in disparate systems for different data types High upfront and ongoing costs, along with complexity, prohibit scaling Many query tools and noSQL systems require specialized knowledge Scale On-Demand Use Standard Skills
- 11. © 2019 Snowflake Computing Inc. All Rights Reserved WE’VE DECIDED TO GO CLOUD...NOW WHAT? EDW Data Sources Data Lake or Hadoop Datamarts ETL BI / Analytics OLTP databases Enterprise applications Web apps Third-party Other
- 12. © 2019 Snowflake Computing Inc. All Rights Reserved WE’VE DECIDED TO GO CLOUD...NOW WHAT? Logical Datamarts BI / Analytics EDW Data Lake ETL or ELT OLTP databases Enterprise applications Web apps Third-party Other Data Sources
- 13. © 2019 Snowflake Computing Inc. All Rights Reserved What to think about: ● SQL syntax - differences and gaps ● Technologies and tools involved ○ ETL/ELT tool ○ BI, analytics, reporting tools ○ Excel, direct query access, etc. ● Security! Users and roles, data security at rest/in flight ● Planning administration and operations changes ○ How to monitor the DW ○ Updated roles / responsibilities ● Downtime - keep it to a minimum (as in, zero downtime) ● Training on the new cloud data warehouse platform ○ SQL syntax differences ○ Connecting to the DW GETTING READY
- 14. © 2019 Snowflake Computing Inc. All Rights Reserved MIGRATION APPROACH - LIFT AND SHIFT
- 15. © 2019 Snowflake Computing Inc. All Rights Reserved MIGRATION APPROACH - DATA VIRTUALIZATION
- 16. © 2019 Snowflake Computing Inc. All Rights Reserved MIGRATION APPROACH - INCREMENTAL USING CONTINUOUS REPLICATION
- 17. © 2019 Snowflake Computing Inc. All Rights Reserved Move the data and begin taking realizing benefits of the cloud data warehouse immediately ● Connect business intelligence/reporting/analytics tools first ● Gain champions of the cloud data warehouse Applications continue to work against the legacy data warehouse ● ETL still runs as-is - migrate/rebuild at your own pace ● Additional sources such as master data management (MDM), streaming, etc can be migrated based on priority - not all at once Minimize migration risk ● ETL rebuilding / modifications WHY REPLICATION FOR MIGRATION?
- 18. © 2019 Snowflake Computing Inc. All Rights Reserved REPLICATION OPTION - ORACLE GOLDENGATE
- 19. © 2019 Snowflake Computing Inc. All Rights Reserved CONTINUOUS REPLICATION TO CLOUD DW
- 20. © 2019 Snowflake Computing Inc. All Rights Reserved CONTINUOUS REPLICATION TO CLOUD DW Snowflake Database External S3 Snowpipe Service Server-less Loader S3 notification File data
- 21. © 2019 Snowflake Computing Inc. All Rights Reserved Begin data replication from source ● Build up transactions during initial load time Perform initial load of data from current DW to cloud DW ● Capture SCN or equivalent as-of initial load Begin replication to target after initial load SCN ● Transactions will “catch up” and sync Repoint reporting / analytics tools at cloud data warehouse ● Begin with focus on single application or subject area ● Work through initial challenges (SQL syntax, data type conversions, etc) Migrate ETL processes ● Work through migration/modification in DEV environment Once ETL ready, switch off data replication and switch on migrated ETL processes INCREMENTAL MIGRATION USING DATA REPLICATION EXAMPLE STEPS
- 22. © 2019 Snowflake Computing Inc. All Rights Reserved Cost! Availability of data replication software ● Does the company have a license already? ● Which tools that can be used for micro-batch loading? Latency requirements ● For example, if ETL batch process runs twice daily, can the cloud data warehouse be updated 15 minutes later? Project timeline Not a one-size fits all approach CONSIDERATIONS
- 23. © 2019 Snowflake Computing Inc. All Rights Reserved Oracle GoldenGate docs: https://docs.oracle.com/goldengate/c1221/gg-winux/index.html GoldenGate for Big Data docs: https://docs.oracle.com/goldengate/bd123210/gg-bd/index.html Oracle Data Integration blog: https://blogs.oracle.com/dataintegration/data-integration Continuous Data Replication into Snowflake with Oracle Goldengate blog post: https://www.snowflake.com/blog/continuous-data-replication-into-snowflake-with-oracle-goldengate/ Replicating Data to Oracle Autonomous Data Warehouse Cloud: https://docs.oracle.com/goldengate/c1230/gg-winux/GGODB/replicating-data-oracle-autonomous-data-war ehouse-cloud.htm MORE INFORMATION
- 24. THANK YOU © 2019 Snowflake Computing Inc. All Rights Reserved
- 25. © 2019 Snowflake Computing Inc. All Rights Reserved DISCOVER THE PERFORMANCE, CONCURRENCY, AND SIMPLICITY OF SNOWFLAKE As easy as 1-2-3! 01 Visit Snowflake.com 02 Click “Try for Free” 03 Sign up & register Snowflake is the only data warehouse built for the cloud. You can automatically scale compute up, out, or down—independent of storage. Plus, you have the power of a complete SQL database, with zero management, that can grow with you to support all of your data and all of your users. With Snowflake On Demand™, pay only for what you use. Sign up and receive $400 worth of free usage for 30 days!