Why Data Vault?
13 likes10,985 views
Given at Oracle Open World 2011: Not to be confused with Oracle Database Vault (a commercial db security product), Data Vault Modeling is a specific data modeling technique for designing highly flexible, scalable, and adaptable data structures for enterprise data warehouse repositories. It has been in use globally for over 10 years now but is not widely known. The purpose of this presentation is to provide an overview of the features of a Data Vault modeled EDW that distinguish it from the more traditional third normal form (3NF) or dimensional (i.e., star schema) modeling approaches used in most shops today. Topics will include dealing with evolving data requirements in an EDW (i.e., model agility), partitioning of data elements based on rate of change (and how that affects load speed and storage requirements), and where it fits in a typical Oracle EDW architecture. See more content like this by following my blog http://kentgraziano.com or follow me on twitter @kentgraziano.
![More Notables…
“This enables organizations to take control of their
data warehousing destiny, supporting better and
more relevant data warehouses in less time than
before.” Howard Dresner
“[The Data Vault] captures a practical body of
knowledge for data warehouse development which
both agile and traditional practitioners will benefit
from..” Scott Ambler](https://image.slidesharecdn.com/whydatavaultoow2011-111011132258-phpapp02/85/Why-Data-Vault-33-320.jpg)
Ad
More Related Content
What's hot (20)
Viewers also liked (20)
Ad
Similar to Why Data Vault? (20)
Ad
More from Kent Graziano (10)
Recently uploaded (20)
Why Data Vault?
- 1. Why Data Vault? Kent Graziano Data Vault Master and Oracle ACE TrueBridge Resources OOW 2011 Session #28782
- 2. My Bio • Kent Graziano – Certified Data Vault Master – Oracle ACE (BI/DW) – Data Architecture and Data Warehouse Specialist • 30 years in IT • 20 years of Oracle-related work • 15+ years of data warehousing experience – Co-Author of • The Business of Data Vault Modeling (2008) • The Data Model Resource Book (1st Edition) • Oracle Designer: A Template for Developing an Enterprise Standards Document – Past-President of Oracle Development Tools User Group (ODTUG) and Rocky Mountain Oracle User Group – Co-Chair BIDW SIG for ODTUG
- 3. Data Vault Definition The Data Vault is a detail oriented, historical tracking and uniquely linked set of normalized tables that support one or more functional areas of business. It is a hybrid approach encompassing the best of breed between 3rd normal form (3NF) and star schema. The design is flexible, scalable, consistent, and adaptable to the needs of the enterprise. It is a data model that is architected specifically to meet the needs of today’s enterprise data warehouses. Dan Linstedt: Defining the Data Vault TDAN.com Article (C) TeachDataVault.com
- 4. Where does a Data Vault Fit? (C) TeachDataVault.com
- 5. Where does a Data Vault Fit? Oracle’s Next Generation Data Warehouse Reference Architecture Data Vault goes here (C) Oracle Corp
- 6. Why Bother With Something New? Old Chinese proverb: 'Unless you change direction, you're apt to end up where you're headed.' (C) TeachDataVault.com
- 7. Why do we need it? • We have seen issues in constructing (and managing) an enterprise data warehouse model using 3rd normal form, or Star Schema. – 3NF – Complex PKs with cascading snapshot dates (time-driven PKs) – Star – difficult to re-engineer fact tables for granularity changes • These issues lead to break downs in flexibility, adaptability, and even scalability (C) Kent Graziano
- 8. Data Vault Time Line E.F. Codd invented 1976 Dr Peter Chen 1990 – Dan Linstedt relational modeling Created E-R Begins R&D on Data Vault Diagramming Modeling Chris Date and Hugh Darwen Maintained Mid 70’s AC Nielsen and Refined Popularized Modeling Dimension & Fact Terms 1960 1970 1980 1990 2000 Late 80’s – Barry Devlin and Early 70’s Bill Inmon Dr Kimball Release “Business Began Discussing Data Data Warehouse” Warehousing Mid 80’s Bill Inmon Popularizes Data Mid 60’s Dimension & Fact Modeling Warehousing presented by General Mills and 2000 – Dan Linstedt Dartmouth University Mid – Late 80’s Dr Kimball releases first 5 articles on Popularizes Star Schema Data Vault Modeling (C) TeachDataVault.com
- 9. Data Vault Modeling… (C) TeachDataVault.com
- 10. What Are the Issues? This is NOT what you want happening to your project! (C) TeachDataVault.com THE GAP!!
- 11. What Are the Foundational Keys? Flexibility Scalability Productivity (C) TeachDataVault.com
- 12. Key: Flexibility (Agility) Enabling rapid change on a massive scale without downstream impacts! (C) TeachDataVault.com
- 13. Key: Scalability Providing no foreseeable barrier to increased size and scope People, Process, & Architecture! (C) TeachDataVault.com
- 14. Key: Productivity Enabling low complexity systems with high value output at a rapid pace (C) TeachDataVault.com
- 15. Bringing the Data Vault to Your Project HOW DOES IT WORK? (C) TeachDataVault.com
- 16. Key: Flexibility (Agility) • Goes beyond standard 3NF • Hyper normalized • Hubs and Links only holds keys and meta data • Satellites split by rate of change and/or source • Enables Agile data modeling • Easy to add to model without having to change existing structures and load routines • Relationships (links) can be dropped and created on-demand. • No more reloading history because of a missed requirement • Based on natural business keys • Not system surrogate keys • Allows for integrating data across functions and source systems more easily • All data relationships are key driven. (C) TeachDataVault.com
- 17. Key: Flexibility (Agility) Adding new components to the EDW has NEAR ZERO impact to: • Existing Loading Processes • Existing Data Model • Existing Reporting & BI Functions • Existing Source Systems • Existing Star Schemas and Data Marts (C) TeachDataVault.com
- 18. Split and Merge ON DEMAND! 2 weeks from now 6 months from now (C) TeachDataVault.com
- 19. Case In Point: Result of flexibility of the Data Vault Model allowed them to merge 3 companies in 90 days – that is ALL systems, ALL DATA! (C) TeachDataVault.com
- 20. Key: Scalability in Architecture Scaling is easy, its based on the following principles • Hub and spoke design • MPP Shared-Nothing Architecture • Scale Free Networks • Can be partitioned vertically and horizontally to meet performance demands (C) TeachDataVault.com
- 21. Perhaps You Wish To Split For Performance Reasons? FROM THIS TO THIS! (C) TeachDataVault.com
- 22. Case In Point: Result of scalability was to produce a Data Vault model that scaled to 3 Petabytes in size, and is still growing today! (C) TeachDataVault.com
- 23. Key: Scalability in Team Size You should be able to SCALE your TEAM as well! With the Data Vault methodology, you can: Scale your team when desired, at different points in the project! (C) TeachDataVault.com
- 24. Case In Point: (Dutch Tax Authority) Result of scalability was to increase ETL developers for each new source system, and reassign them when the system was completely loaded to the Data Vault (C) TeachDataVault.com
- 25. Key: Productivity Increasing Productivity requires a reduction in complexity. The Data Vault Model simplifies all of the following: • ETL Loading Routines • Real-Time Ingestion of Data • Data Modeling for the EDW • Enhancing and Adapting for Change to the Model • Ease of Monitoring, managing and optimizing processes (C) TeachDataVault.com
- 26. Key: Productivity • Standardized modeling rules • Highly repeatable and learnable modeling technique • Can standardize load routines • Delta Driven process • Re-startable, consistent loading patterns. • Can standardize extract routines • Rapid build of new or revised Data Marts • Can be automated • RapidACE (www.rapidace.com) (C) Kent Graziano
- 27. Key: Productivity • The Data Vault holds granular historical relationships. • Holds all history for all time, allowing any source system feeds to be reconstructed on-demand • Easy generation of Audit Trails for data lineage and compliance. • Data Mining can discover new relationships between elements • Patterns of change emerge from the historical pictures and linkages. • The Data Vault can be accessed by power-users (C) Kent Graziano
- 28. Case in Point: Result of Productivity was: 2 people in 2 weeks merged 3 systems, built a full Data Vault EDW, 5 star schemas and 3 reports. These individuals generated: • 90% of the ETL code for moving the data set • 100% of the Staging Data Model • 75% of the finished EDW data Model • 75% of the star schema data model (C) TeachDataVault.com
- 29. The Competing Bid? The competition bid this with 15 people and 3 months to completion, at a cost of $250k! (they bid a Very complex system) Actual total cost? $30k and 2 weeks! (C) TeachDataVault.com
- 30. Other Benefits of a Data Vault • Modeling it as a DV forces integration of the Business Keys upfront. • Good for organizational alignment. • An integrated data set with raw data extends it’s value beyond BI: • Source for data quality projects • Source for master data • Source for data mining • Source for Data as a Service (DaaS) in an SOA (Service Oriented Architecture). • Upfront Hub integration simplifies the data integration routines required to load data marts. • Helps divide the work a bit. • It is much easier to implement security on these granular pieces. • Granular, re-startable processes enable pin-point failure correction. • It is designed and optimized for real-time loading in its core architecture (without any tweaks or mods). (C) Kent Graziano
- 31. Conclusion? Changing the direction of the river takes less effort than stopping the flow of water (C) TeachDataVault.com
- 32. The Experts Say… “The Data Vault is the optimal choice for modeling the EDW in the DW 2.0 framework.” Bill Inmon “The Data Vault is foundationally strong and exceptionally scalable architecture.” Stephen Brobst “The Data Vault is a technique which some industry experts have predicted may spark a revolution as the next big thing in data modeling for enterprise warehousing....” Doug Laney
- 33. More Notables… “This enables organizations to take control of their data warehousing destiny, supporting better and more relevant data warehouses in less time than before.” Howard Dresner “[The Data Vault] captures a practical body of knowledge for data warehouse development which both agile and traditional practitioners will benefit from..” Scott Ambler
- 34. Who’s Using It?
- 35. Growing Adoption… • The number of Data Vault users in the US surpassed 500 in 2010 and grows rapidly (http://danlinstedt.com/about/dv- customers/) (C) Kent Graziano
- 36. In Review… • Data Vault provides you with the tools you need to succeed in your DW/BI projects • Flexibility • Enabling rapid change on a massive scale without downstream impacts! • Scalability • Providing no foreseeable barrier to increased size and scope • Productivity • Enabling low complexity systems with high value output at a rapid pace (C) TeachDataVault.com
- 38. Where To Learn More The Technical Modeling Book: http://LearnDataVault.com On YouTube: http://www.youtube.com/LearnDataVault On Facebook: www.facebook.com/learndatavault Dan’s Blog: www.danlinstedt.com The Discussion Forums: http://LinkedIn.com – Data Vault Discussions World wide User Group (Free): http://dvusergroup.com The Business of Data Vault Modeling by Dan Linstedt, Kent Graziano, Hans Hultgren (available at www.lulu.com ) 38
- 39. 10/11/2011 (C) TeachDataVault.com 39
- 40. Contact Information Kent Graziano [email protected] Want more Data Vault? Session # 05923: Introduction to Data Vault Modeling Thursday, 4:00 PM, Moscone South Rm 303