Data Warehousing in the Cloud: Practical Migration Strategies
Download as PPTX, PDF6 likes7,694 views
Dave Wells of Eckerson Group discusses why cloud data warehousing has become popular, the many benefits, and the corresponding challenges. Migrating an existing data warehouse to the cloud is a complex process of moving schema, data, and ETL. The complexity increases when architectural modernization, restructuring of database schema, or rebuilding of data pipelines is needed.
Ad
More Related Content
What's hot (20)
Similar to Data Warehousing in the Cloud: Practical Migration Strategies (20)
Ad
More from SnapLogic (20)
Ad
Recently uploaded (20)
Data Warehousing in the Cloud: Practical Migration Strategies
- 1. © David L Wells Data Warehousing in the Cloud Practical Migration Strategies Dave Wells [email protected]
- 2. © David L Wells 2 Dave Wells Director, Data Management Practice Eckerson Group www.eckerson.com • Advisory consultant • Educator • Industry analyst • Business Intelligence • Analytics • Data Management
- 3. © David L Wells Cloud Data Warehousing – What and Why? 3 Challenges of Conventional Data Warehousing Growth Management Workload Fluctuation Data Center Management Data Center & Operations Costs Processing Bottlenecks & Delays Projects Wait for Infrastructure Business Critical with Risks Security & Governance Challenged Complex Database Management
- 4. © David L Wells Benefits of Cloud Data Warehousing 4 Overcoming the Challenges Growth Management Workload Fluctuation Data Center Management Data Center & Operations Costs Processing Bottlenecks & Delays Projects Wait for Infrastructure Business Critical with Risks Security & Governance Challenged Complex Database Management Scalability: growth in data, processing & users Elasticity: adapt to workload peaks and valleys Managed Infrastructure: reduce data center overhead Cost Savings: cut cost of hardware, staffing, etc. Processing Speed: fast data pipelines, no bottlenecks Deployment Speed: agility, instant infrastructure Disaster Recovery: benefits of virtualization Security & Governance: service provider features + VPC RDBMS in the Cloud: gracefully accepts existing schema
- 5. © David L Wells Technologies for Cloud Data Warehousing 5 Cloud Data Warehouse Platforms
- 6. © David L Wells Technologies for Cloud Data Warehousing 6 Migration Tools – Integration Platform as a Service (iPaaS)
- 7. © David L Wells Technologies for Cloud Data Warehousing 7 Migration Tools – Data Warehouse Automation
- 8. © David L Wells Technologies for Cloud Data Warehousing 8 Migration Tools – Data Virtualization
- 9. © David L Wells Step-by-Step Data Warehouse Migration 9 The Big Picture Migration Technology Selection Migration Strategy Architectural Assessment Business Case Planning Testing and Operationalization incrementalmigration Scope, Timing, Resources, Schedule, User Transparency, Testing Plan Drivers, Costs, Benefits, Risk of Migrating, Risk of Not Migrating Reliability, Availability, Performance, Scalability, Adaptability, Maintainability Lift and Shift or Incremental by Workload, Workload Breakdown and Priorities Cloud Data Warehousing Platform, Migration Tools Schema, Data, Process, Metadata, Users and Applications Function Test, Performance Test, DQ Audit, Scheduling, Monitoring, Support
- 10. © David L Wells Step-by-Step Data Warehouse Migration 10 Business Case Agility Performance Growth Cost Savings Labor Savings Business Case What are the drivers to move to the cloud? What are the business benefits? Who cares about them? What are the technical benefits? Who cares about them? What are the business disadvantages of not migrating? Who feels the pain? What are the technical disadvantages of not migrating? Who feels the pain?
- 11. © David L Wells Step-by-Step Data Warehouse Migration 11 Architectural Assessment Agility Performance Growth Cost Savings Labor Savings Business Case Architecture Current State Assessment of Data Warehouse Architecture Good Okay Flawed Business Architecture Organization Architecture Data Architecture Integration Architecture Technology Architecture Reliability Availability Performance Scalability Adaptability Maintainability Suited to purpose Fits gracefully into the environment Structurally sound Compliant with codes and regulations Sustainable through expected lifespan Aesthetically pleasing
- 12. © David L Wells Step-by-Step Data Warehouse Migration 12 Architectural Assessment Change Warehouse Positioning Change Data Flow Change Data Models Change Data Stores Change Technology Change Architectural Concept Beside data lake, inside data lake … Landing, staging, warehouse, data marts … Normalized, denormalized, dimensional … Divide, combine, partition, retire … Automation, virtualization ... Hub-and-spoke, bus, hybrid … Agility Performance Growth Cost Savings Labor Savings Business Case Current State Assessment of Data Warehouse Architecture Good Okay Flawed Migrate as is Review and refine Redesign Architecture
- 13. © David L Wells Step-by-Step Data Warehouse Migration 13 Migration Strategy COMPLEX MEGA-PROJECT Agility Performance Growth Cost Savings Labor Savings Business Case Current State Assessment of Data Warehouse Architecture Good Okay Flawed Migrate as is Review and refine Redesign Lift and shift • By pain points • By subject area • By data source • By user groups Migration StrategyArchitecture
- 14. © David L Wells Step-by-Step Data Warehouse Migration 14 Migration Strategy 1 2 3 Incremental migration of individual workloads on a case-by-case basis. • When to migrate? • Migrate as is? • Modify and migrate? • Replace with new data mart? Agility Performance Growth Cost Savings Labor Savings Business Case Current State Assessment of Data Warehouse Architecture Good Okay Flawed Migrate as is Review and refine Redesign Lift and shift • By pain points • By subject area • By data source • By user groups Migration StrategyArchitecture
- 15. © David L Wells Step-by-Step Data Warehouse Migration 15 Technology Selection Agility Performance Growth Cost Savings Labor Savings Business Case Current State Assessment of Data Warehouse Architecture Good Okay Flawed Migrate as is Review and refine Redesign Lift and shift • By pain points • By subject area • By data source • By user groups Migration Strategy Cloud platform Migration tools Technology Selection Architecture
- 16. © David L Wells Step-by-Step Data Warehouse Migration 16 Migration Plan Migrate Test Operationalize Agility Performance Growth Cost Savings Labor Savings Business Case Current State Assessment of Data Warehouse Architecture Good Okay Flawed Migrate as is Review and refine Redesign Lift and shift • By pain points • By subject area • By data source • By user groups Migration Strategy Cloud platform Migration tools Technology Selection Migration pipelines ETL meta data dataschema users & apps Architecture
- 17. © David L Wells Step-by-Step Data Warehouse Migration 17 Schema Migration pipelines ETL meta data dataschema users & apps Do you need to change … • Structure • Indexing • Partitioning • Optimization • Pre-Joining • Derivation • Aggregation
- 18. © David L Wells Step-by-Step Data Warehouse Migration 18 Data Migration pipelines ETL meta data dataschema users & apps How much data are you moving? What is your network capacity and what else uses the network? How long will it take to migrate and what can you do to accelerate? Do you need to transform data due to schema adjustment? Should you transform in stream or pre-process?
- 19. © David L Wells Step-by-Step Data Warehouse Migration 19 ETL pipelines ETL meta data dataschema users & apps Change the code base to optimize for platform performance? Change data transformations to sync with data restructuring? Reorganize data flows? Reduce data latency? Migrate ETL processing to the cloud?
- 20. © David L Wells Step-by-Step Data Warehouse Migration 20 Data Pipelines pipelines ETL meta data dataschema users & apps Rebuild pipelines instead of migrating existing ETL? Package individual transform actions as executable objects? Assemble objects as modules? Configure modules for workflow and dataflow? Gain performance, agility, or maintainability?
- 21. © David L Wells Step-by-Step Data Warehouse Migration 21 Metadata pipelines ETL meta data dataschema users & apps Source-to-target mappings and tracing data lineage? Can metadata be readily moved to cloud platform? Can you export and import metadata? Reverse engineer or rebuild from scratch?
- 22. © David L Wells Step-by-Step Data Warehouse Migration 22 Users & Applications pipelines ETL meta data dataschema users & apps Uninterrupted business operations Security and access authorizations Communication and coordination Connecting BI and analytics tools and applications
- 23. © David L Wells Bringing it all Together 23 Cloud Data Warehouse (CDW) - Many good solutions are available. Select the one that is the best fit with your business model, your budget and your existing systems. Integration Platform as a Service (iPaaS) – Use to connect and migrate data from multiple, different endpoints. Make sure your iPaaS is strong not just for application integration but also cloud DW, big data & data lake integration. Data Warehouse Automation (DWA) – Ideal for schema replication from the legacy data store to the new CDW and for schema management. Select a DWA that can best automate routine developer tasks. Data Virtualization (DV) – Use to provide easier access to data by your “customers” using a modern interface. Ideal for incremental data migration.
- 24. © David L Wells Getting Started with Cloud Data Warehousing 24 What’s Next? Should your data warehouse move to the cloud? What benefits would you get from cloud data warehousing? What challenges and risks will you face? How would you approach migrating your data warehouse? What people, tools, and resources will you need?
- 25. © David L Wells Questions …
- 26. © David L Wells Get the Whitepaper at SnapLogic.com/resources Email me at [email protected] Follow my blogs at eckerson.com/blogs/data-management Thank You! Free Demo available on SnapLogic.com