Modernizing Data Architecture using Data Virtualization for Agile Data Delivery
1 like364 views
This document discusses modernizing a company's data architecture using data virtualization to improve agile data delivery. It describes the company's current federated model and need to respond more quickly. The proposal is to implement a data virtualization layer to land data from various sources, create a conformed data warehouse, and provide a single data access point. This would virtualize data, reduce costs, improve flexibility and throughput, and accelerate change cycles compared to the current environment. A specific example of using this approach to enable daily and intra-day reporting from a new system is also described.
Modernizing Data Architecture using Data Virtualization for Agile Data Delivery
- 1. INTERNAL USE ONLY Modernizing Data Architecture using Data Virtualization for Agile Data Delivery Dave Kay, Data Consultant UKGI Data, Analytics and Architecture
- 2. INTERNAL USE ONLY 2
- 3. INTERNAL USE ONLY 3 55,000Employees (approximately) 170+ Countries USD 4.6 bn Business operating profit
- 4. INTERNAL USE ONLY 4 The Zurich Group Operates a Federated Model for Data Deliveries. Group Function – Delivers Data Framework Satellite Function – ZNA Satellite Function – UKGI & UK Life Satellite Function – Zurich Switzerland BU Etc…
- 5. INTERNAL USE ONLY 5
- 6. INTERNAL USE ONLY 6 Big change coming in Insurance – the ability to respond at pace is key to success. • • • • • •
- 7. INTERNAL USE ONLY 7
- 8. INTERNAL USE ONLY 8 Learn about and build plans for: • Driving toward a near future that encompasses: • With ambition for a longer term where: •
- 9. INTERNAL USE ONLY 9
- 10. INTERNAL USE ONLY 10 SchedulingDataSources ETLProcessing SolutionDelivery UserAccess
- 11. INTERNAL USE ONLY 11 Reduce Cost Flex Resource Raise throughput Accelerate change • Cost of change prohibitive to generating value from data • Slower time to market inhibiting impact of good analytics • Complexity of solution can cause delivery bottlenecks • Bespoke coded solution requires unique resource to maintain
- 12. INTERNAL USE ONLY 12 Data Landing Conformed Warehouse Data Hub Interface Layer • Data landed into infrastructure from disparate sources • Conformed enterprise data warehouse provides central storage • Data hub providing single ‘user facing’ data source • Simplified Interface layer landscape – single point of entry
- 13. INTERNAL USE ONLY 13
- 14. INTERNAL USE ONLY 14 Business Need: Provide daily MI and intra-day Operational Reporting for a community of business analysts, salesmen, underwriters, managers and actuarial staff – off the back of a brand new system. • XML Source with over 13,000 elements, 10+ products • XML Schema not available • Data items and schema differ dramatically per product type • Fast pace of change
- 15. INTERNAL USE ONLY 15 • Subset of data ‘shredded’ via XSLT – immediately reduces scope (250 items) • SSIS transforms at multiple points generate maintenance difficulties • Data effectively replicated multiple times • Change Request process lengthy and expensive • Specialist development skills required
- 16. INTERNAL USE ONLY 16 • XML made available into new DAA architecture for ingestion • Denodo used as a virtual staging wrapper • Views over XML representing core entities for reporting (Relational View) virtualised in Denodo – cached to SQL Server • Relational, Reporting and Analytical stores published as consumable data sets via Denodo • Two tier interface: • Rapid interactive access to quickly changing data for ad-hoc queries via Alteryx, JDBC, REST etc • Stable, conformed relational marts virtualised & cached for exploration, analysis and visualisation in Qlik etc
- 17. INTERNAL USE ONLY 17 • • • • • • • • ••
- 18. INTERNAL USE ONLY 18
- 19. INTERNAL USE ONLY 19 DataSources VirtualStaging VirtualSemantic AtomicWarehouse VisualisationAnalytics
- 20. INTERNAL USE ONLY 20 Reduce Cost Flex Resource Raise throughput Accelerate change • Cost model reduced • Significantly quicker time to market • Greater throughput • Resource pool expanded