CWIN 17 / sessions data vault modeling - f2-f - nishat gupta

Data Vault Modeling – An Insight
Nishant Gupta
Bangalore, September 27th
#CWIN17

Session’s Title | Date
Copyright © 2017 Capgemini and Sogeti. All rights reserved. 2
Table of Contents
 Introduction
 Data Vault Modeling Components
 Building Data Vault model
 DVM – An Answer
 DVM 2.0 – An Agile way

Introduction

What is Data Vault Modeling ?
The Data Vault is a detail oriented, historical tracking and uniquely linked set of
normalized tables that support one or more functional areas of business.
It is a hybrid approach encompassing the best of breed between 3rd normal form (3NF)
and star schema. The design is flexible, scalable, consistent and adaptable to the needs
of the enterprise.
It is a data model that is architected specifically to meet the needs of enterprise data
warehouses.
--Dan Linstedt
The Data Vault is functionally based, not subject oriented as defined
by Bill Inmon

Why Data Vault Modeling ?
 3 NF has complex Pks when cascading snapshot dates (time driven PKs)
 Real time loading is a challenge
 Drill Down Analysis & Queries are complex and tedious
 Top down approach results in unavoidable Flexibility and Scalability Issues
 Star Schema – difficult to implement /re engineer fact tables for granularity changes
 Data Redundancy & Helper Tables isolates subject related information
 Inconsistent Query linkages due to incompatible grains
 Synchronization issues during Real Time Data Load
 Limited Enterprise views and Data Mining capabilities
Challenges with DW Data Modeling Architectures

Data Vault Modeling Place in Evolution
 1960s - Codd, Date et. al Normal Forms
 1970s - Peter Chen created E-R diagramming
 1980s - Normal Forms adapted to DWs courtesy Bill Inmon
 1985+ - Star Schema for OLAP by Ralph Kimball
 1990s - Data Vault concept developed Dan Linstedt
 2000+ - Data Vault concept published by Dan Linstedt

Data Vault Modeling Evolution

Where does a Data Vault Fit ?

Data Vault Modeling Components

Data Vault Modeling - Components
 Hub Entities
 Candidate Keys + Load Time + Source
 Link Entities
 FKs from Hub + Load Time + Source
 Satellite Entities
 Descriptive Data + Load Time + Source + End Time
• Dimension = Hub+ Satellite
• Fact = Satellite + Link [+ Hub ]
Data Vault Modeling has 3 Keys Components summarized below with their attributes

DVM Components - Hubs
 Hub Entities
 Hub Entities, or Hubs, are a single table carrying at a minimum a unique
list of business keys
 For example, Invoice number, Employee number, Customer Number, Part
number and VIN etc.
 Hubs allows to integrate multiple source systems, hence, should be
source system agnostic
 It contains:
• A Hub PK The Business Key column(s)
• The Load Date (LOAD_DTS) The Source for the record (REC_SRC)
 New in DV 2.0, the Hub PK is a calculated field includes MD5 to link with
Hadoop/No Sql
Hub = Business Key
This is the most important aspect of Data Vault modeling.
It has to identified it correctly yo build an integrated enterprise data warehouse for an organization.

DVM Components – Hubs contd…

DVM Components - Links
 Link Entities
 Link Entities or Links, are a physical representation of a many-to-many
3NF relationship
 A Link is therefore an intersection of business keys . A Link must have at
least two Hubs, but they may be composed of many Hubs
 A Link table’s grain is defined by the number of parent keys it contains
similar to Fact Table in Dimensional Modeling
 It contains:
• A Link PK (Hash Key) The PKs from the parent Hubs – used for lookups
• The Load Date (LOAD_DTS) The Source for the record (REC_SRC)
 New in DV 2.0, The Business Key column(s) – for faster query
Links = Associations
Link captures and records the relationship of business elements at the lowest possible grain that shouldn’t
change over time . It includes transactions and hierarchies.

DVM Components – Links contd…
 Many-to-Many relationships
benefits :
 Flexibility
 Granularity
 Dynamic adaptability
 Scalability
 No need to change the EDW
structure
 Existing Data is fine
 New Data is added
Modeling Links – 1:M or M:M With Link in the Data Vault
If a Link structure is compromised, then the flexibility
of the data vault model is immediately compromised
.

DVM Components – Links contd…

DVM Components - Satellites
 Satellite Entities
 Satellite Entities or Satellites, are Hub key context (descriptive) information
 It is a time-dimensional table housing detailed information about the Hub’s
or Link’s business keys at a point in time or over a time period
 The Change Data Capture (CDC) is done and History is stored .
 It’s concept and structure is similar to Type 2 Slowly Changing Dimension
 It contains:
• Satellite Primary Key: Hub or Link Primary Key & Load Date Time Stamp
• Satellite Optional Primary Key: Sequence Surrogate Number
• Other Attributes including End Data The Source for the record (REC_SRC)
 New in DV 2.0, HASH_DIFF columns for Change Data Capture (CDC)
Satellite = Descriptors
Satellites are typically arranged by type or classification of data, and rate of change, this results in,
Satellite to split away groups of fields that change more quickly than others.

DVM Components – Satellites contd…

Building Data Vault

Building a Data Vault
 Model the Hubs. This requires an understanding of business keys and their usage across
the designated scope.
 Model the Links. Forming the relationships between the keys – formulating an
understanding of how the business operates today in context to each business key.
 Model the Satellites. Providing context to each of the business keys as well as the
transactions (Links) that connect the Hubs together. This begins to provide the complete
picture of thebusiness.
 Model the point-in-time tables. This is a Satellite derivative, of which the structure and
definition is outside the scope of this document (due to space constraints).
The Data Vault should be built as follows

Reference Rule for Data Vault Modeling
 Hub keys cannot migrate into other Hubs (no
parent/child like Hubs).
 Hubs must be connected through Links.
 More than two Hubs can be connected through
Links.
 Links can be connected to other Links.
 Links must have at least two Hubs associated with
them in order to be instantiated.
 Surrogate keys may be utilized for Hubs and Links.
 Surrogate keys may not be utilized for Satellites.
 Hub keys always migrate outward.
 Hub business keys never change, Hubs primary
keys never change.
 Satellites may be connected to Hubs or Links.
 Satellites always contain either a load date-time
stamp, or a numeric reference to a stand-alone load
date-time stamp sequence table.
 Stand-alone tables such as calendars, time, code
and description tables may be utilized.
 Links may have a surrogate key.
 If a hub has two or more satellites, a point-in-time
table may be constructed for ease of joins.
 Satellites are always delta driven, duplicate rows
should not appear.
 Data is separated into Satellite structures based on:
1) type of information 2) rate of change
- BK with low propensity for Change become Hub key -Transactions and Integrated keys become link tables.
- Descriptive data always fits in a Satellite

Data Vault Loading Sequence
 Hubs for Dimensions
 Links for Dimensions
 Satellites for Dimensions
 Hubs for Fact (if any )
 Links for Fact
 Satellites for Fact
Typically data loading for a Data Vault is in the following sequence :

Sample Data Vault Model

World’s Smallest Data Vault
 The Data Vault doesn’t have to be “BIG”
 An data vault can be built incrementally
 Reverse engineering one component is possible
 Building one section of Data Vault, and the connect it
with mart is the right strategy
 The smallest EDW consists of two tables :
 One Hub
 One Satellite

DVM – An Answer to EDW Architecture Pain

EDW Needs Today !
In today’s world where we are receiving loads of data across multiple sources/channels we
need Data Model that is architected specifically to meet the below stated EDW needs :
 Extensive possibilities for data attribution.
 All data relationships are key driven.
 Relationships can be dropped and created on-the-fly.
 Data Mining can discover new relationships between elements
 Artificial Intelligence can be utilized to rank the relevancy of the relationships to the user configured
outcome.

Data Vault Modeling Benefits – An Answer !
The Data Vault Model is a data integration architecture; a series of standards, and
definitional elements or methods by way information is connected within an RDBMS data
store in order to make sense of it.
Business Benefits:
• Manage and Enforce Compliance to various regulation in your Enterprise Data Warehouse
• Spot business problems that were never visible previously
• Rapidly Reduce business cycle time for implementing changes
• Merge new business units into the organization rapidly
• Rapid ROI and Delivery of information to new Star Schemas
• Consolidate disparate data stores., ie: Master Data Management
• SEI CMM Level 5 compliant (Repeatable, consistent, redundant architecture)
• Trace all data back to the source systems

Data Vault Modeling Benefits – An Answer !
Technical Benefits:
 Near-Real-Time Loads
 Traditional Batch Loads
 In-Database Data Mining
 Terabytes to Petabytes of information (Big Data)
 Incremental Build Out
 Seamless integration of unstructured data (NoSQL)
 Dynamic Model Adaptation – self healing
 Business Rule Changes (with Ease)

DVM 2.0 – An Agile way !

Agile manifesto for DW
 User Stories instead of detailed requirements
 Time-boxed iterations
 Iteration has a standard length
 Select User stories that is doable
 Rework is included in the overall scheme of things
 No missed requirements only
• Not Delivered
• Not Discovered

Agile Data Vault Modeling
 Model Iteratively
 Use DVM to create basic components
 Add more in due course of time
 Virtualize the Access Layer
 No Facts and dimension creation upfront
 ETL and Testing can take long
 Create Database View on Top of pattern based DV Model
 User sees reports with live data
 Plan for performance in later iteration

What’s New in DV2.0 ?
 Modeling Structure Includes…
 No SQL, and Non-Relational DB Systems, Hybrid Systems
 Minor Structure Changes to support NoSQL
 New ETL Implementation Standards
 For True Real time support
 For NoSQL support
 New Architecture Standards
 Support for NoSQL data management systems
 New Methodology components
 CMMI, Six Sigma and TQM
 Project Planning and Tracking
 Agile Delivery Mechanisms
 Standards, and template for Projects
The Model is fully compliant with Hadoop and needs
NO changes to work properly

References
 http://tdan.com/data-vault-series
 https://www.slideshare.net/kgraziano/introduction-to-data-vault-modeling
 Building a Scalable Data Warehouse with Data Vault 2.0 by Michael Olschimke and Dan
Linstedt
 Super Charge your Data warehouse by Kent Graziano and Dan Linstedt

Thank You!
Phone: +91 8884400312
name.nishgupt@capgemini.com
Name Nishant Gupta
Title Manager
Role Technical Manager
@Twitter Account
Speaker 1
Photo
Phone: +XX XXXXXXXX
name.name@capgemini.com
Name NAME
Title
Role
@Twitter Account
Speaker 2
Photo

Appendix

www.capgemini.com
The information contained in this presentation is proprietary.
Copyright © 2017 Capgemini and Sogeti. All rights reserved.
Rightshore® is a trademark belonging to Capgemini.
www.sogeti.com
About Capgemini and Sogeti
With more than 180,000 people in over 40 countries, Capgemini is one of With more
than 190,000 people in over 40 countries, Capgemini is one of the world's foremost
providers of consulting, technology and outsourcing services. The Group reported
2016 global revenues of EUR 12.5 billion. Together with its clients, Capgemini
creates and delivers business, technology and digital solutions that fit their needs,
enabling them to achieve innovation and competitiveness. A deeply multicultural
organization, Capgemini has developed its own way of working, the Collaborative
Business Experience™, and draws on Rightshore®, its worldwide delivery model.
Sogeti is a leading provider of technology and software testing, specializing in
Application, Infrastructure and Engineering Services. Sogeti offers cutting-edge
solutions around Testing, Business Intelligence & Analytics, Mobile, Cloud and
Cyber Security. Sogeti brings together more than 23,000 professionals in 15
countries and has a strong local presence in over 100 locations in Europe, USA
and India. Sogeti is a wholly-owned subsidiary of Cap Gemini S.A., listed on the
Paris Stock Exchange.

CWIN 17 / sessions data vault modeling - f2-f - nishat gupta

Recommended

More Related Content

What's hot (20)

Similar to CWIN 17 / sessions data vault modeling - f2-f - nishat gupta (20)

More from Capgemini (20)

Recently uploaded (20)

CWIN 17 / sessions data vault modeling - f2-f - nishat gupta