HOW TO SAVE PILEs of $$$BY CREATING THE BEST DATA MODEL THE FIRST TIME (Kscope 19)

© 2019 Snowflake Inc. All Rights Reserved
HOW TO SAVE
PILES OF $$$
BY CREATING THE
BEST DATA MODEL
THE FIRST TIME
KENT GRAZIANO, CHIEF TECHNICAL EVANGELIST I @KentGraziano

AGENDA
3
Bio
Assumptions
My Best Practices
Standards
Checklist #1
Checklist #2
Checklist #3
Benefits
Considerations
Is it Agile?
Conclusion

MY BIO
4
Chief Technical Evangelist, Snowflake Computing
Oracle ACE Director (DW/BI)
Blogger: The Data Warrior
Certified Data Vault Master and DV 2.0 Practitioner
Former Member: Boulder BI Brain Trust (#BBBT)
Member: DAMA Houston & DAMA International
Data Architecture and Data Warehouse Specialist
• 30+ years in IT
• 25+ years of Oracle-related work
• 20+ years of data warehousing experience
Author & Co-Author of a bunch of books (Amazon)
Past-President of ODTUG and Rocky Mountain Oracle User Group

3 years in stealth, 3+ years GA
5
1200+ employees
Over 2000
customers today
Over $920M in venture
funding from leading
investors
First customers
2014, general
availability 2015
Founded 2012 by
industry veterans
with over 120
database patents
Queries processed in
Snowflake per day:
100 million
Largest single
table:
68 trillion rows
Largest number of
tables single DB:
200,000
Single customer
most data:
> 40PB
Single customer
most users:
> 10,000
Fun facts:

PRESENTATION BASED ON:
6
http://www.amazon.com/Check-Doing-Design-Reviews-ebook/dp/B008RG9L5E/

Not a data modeling
presentation
• You already understand ERD’s and
database design
Not a project management
presentation
7
ASSUMPTIONS

Who are you?
• Data Modeler or Architect
• Project Managers
• IT Managers
• DBA
• Developer
Experience
• Data Modeling
– Less than 1yr?
– 1-5yrs?
– Over 5yrs?
8
SURVEY

Design the Database BEFORE you start
programming!
• Based on the actual business
requirements, needs, and terminology
Have standards and FOLLOW them!
Develop a check list so you don’t
miss a step
• Really – it is faster and more efficient
The sooner you find a mistake the
cheaper it is to fix!
9
MY BEST PRACTICES
(NOSURPRISESHERE)

Have them!
• Insure consistency
• Easier to review
Pick a tool to model in
• SQL Developer Data Modeler (SDDM)
• Erwin (if you have $$)
• Visio (really?)
Generate code whenever possible!
10
STANDARDS

Object naming
• Entities
• Attributes
• Tables
• Columns
• Views
• Constraints (PK, UK, FK)
• Indexes
Use a tool that helps
support these!
11
STANDARDS

SDDM NAMING TEMPLATES
12

Diagrams
• Shapes
• Colors
– Yellow = lookup
– Red outline = shared
• Crows feet & direction
• Legend
– Diagram Name
– Author
– Last Updated
13
STANDARDS

DIAGRAM LEGEND – SDDM
14

CHECKLIST #1

BASIC DESIGN PROCESS – STEPS
16
Develop the logical data model (i.e., entity relationship diagram)
• If doing data vault or data mart design, skip to build of physical model
Subject the model to one (or more) peer reviews
Make changes from the reviews
Get team “sign off” on the model
Convert the model to a physical database design
Subject the physical design to peer review

BASIC DESIGN PROCESS – CONT’D
17
Make changes
Get team sign off
Generate the DDL to build the model
Review the DDL
Make changes
Execute the changed DDL
Hand off for development of ETL programs

CHECKLIST #2

DEVELOP THE LOGICAL MODEL (DETAILS)
19
Gather and review requirements
Review current naming standards and domains
Review existing models (if any)
Develop new data model objects
Modify existing objects if required/indicated
Review logical model check list

DEVELOP THE LOGICAL MODEL
20
Schedule logical model design review session
Prepare for review session
• Print the ERD (or ERD Subview)
• Print entity and attribute detail reports
– Assuming your modeling tool can produce some
– Otherwise, print whatever meta-data you have
– Email to the reviewers
• BETTER: Save the diagrams and reports to a shared knowledge
management site like SharePoint or Confluence
• BEST: Use SDDM with SVN and share

DEVELOP THE LOGICAL MODEL
21
Conduct the review session
Make changes resulting from review session.
Return to the develop/modify objects steps & repeat
When approved, proceed to the development of
physical model

REVIEW SESSION
CHECKLIST (#2A)

LOGICAL MODEL REVIEW CHECKLIST
23
If you have Oracle SDDM
• Have the standard Design Rules been run to validate the model
completeness?
– If no – then do it!
If not using SDDM
• Get a sanity check from another team member before
scheduling review
• Instead of pair-programming (from XP) do pair data modeling
Saves embarrassment from obvious mistakes

SDDM DESIGN RULES
24

25
Entity Review Questions
• Is each entity in the model in 3rd Normal Form?
• Are any entities similar in nature to or a sub-type of, an entity already
in the corporate repository?
– If so, why was that entity not used?
• Are there detailed definitions for every entity and sub-entity?
– Review definitions.
– For data warehouse models, is the source of the data noted?
• Do all entities conform to existing naming standards and
abbreviations?
– Are any new standards or abbreviations indicated?

26
Relationship Review
• Is the cardinality correct?
• Is the optionality correct?
• Do the relationship names make sense to a business
analyst or end user?
• Do all arc’ed relationships validate?

27
Attribute review
• Do all attributes conform to existing naming standards
and abbreviations?
• Are any new standards or abbreviations indicated?
• Have domains been used appropriately?
• Are there any new domains that can be identified?
• Is there an “attribute comment” for every attribute?

28
Attribute review
• Optional - Is there a note for every attribute?
– For data warehouse models, is the source of the attribute noted?
• Are any of the attributes derived?
– If so, why are they included in the logical model, which should
be 3NF?
• Are all the appropriate audit columns included (created by, creation
date, etc.)?
– If not add them!

AUTOMATICALLY ADD AUDIT COLUMNS IN SDDM
29

30
Unique Identifiers
• Does every entity have at least one unique identifier (UID)?
– Does this UID distinguish each instance as being unique?
• Verify with examples.
– Is this UID a true business key as opposed to a system generated
sequence?
– If it is not a business key, why not?
• If system generated, then it should be noted as an exception.
– Is the business key updatable?
• Are there any alternate unique identifiers?
– Verify with examples.

31
Other
• Does the model answer all the known requirements?
– Assumes you have more than air-specs
– Can it answer all the known questions from the users?
• Has the model been compared to other models for quality and
completeness?
– For example, “Universal Data Models” from The Data
Model Resource Book by Silverston, et al.

SDDM REPORTS
32

LOGICAL MODEL REVIEW
33
This process seems intensive
• An ounce of prevention…
Requires modelers to be thorough
Requires reviewers to actually read all the meta-data to verify:
• The descriptions are understandable to a business user
• The text is grammatically correct with proper punctuation
– It may appear in reports or as meta-data in a business
intelligence tool

THANKS FOR PLAYING!
34
Pass or Fail?
• Reviewers decide
Pass – go on to physical model
Fail – revise the model and try again
• The “crumple” factor
Conditional Pass
• Clean up some meta data and domains,
then proceed w/o further review

CHECKLIST #3

DEVELOP THE PHYSICAL MODEL - STEPS
36
Review logical model with data modeler
• If the database designer is a separate person on the team
Review specific implementation requirements
Forward Engineer logical model to a Relational model in SDDM
• Diagram automatically created

SDDM ENGINEER TO RELATIONAL
37

DEVELOP THE PHYSICAL MODEL
38
Make manual modifications to tables and columns
• Try to AVOID this if possible
• As required by tuning and implementation requirements
• To comply with standards
• Add column and table check constraints
Coordinate with the data modeler to account for apparent
missing data elements

39
Review physical model design check list
Schedule physical design review session
Prepare for review session
• Print the schema diagram and table and column detail reports
– Email to the reviewers
• BETTER: Save the diagrams and reports to a shared knowledge
management site like SharePoint or Confluence
• BEST: Use SDDM with SVN and share
Conduct the review session

40
Make changes resulting from review session
• Return to modify the schema diagram
When approved, proceed to generate and deploy DDL to the
development database
• Hand off to development DBA if there is one
Hand-off to programmers for application or ETL development

REVIEW SESSION
CHECKLIST (#3A)

PHYSICAL MODEL REVIEW CHECK LIST
42
Almost identical to Logical Model list
General
• If you have Oracle SDDM
– Have the standard Design Rules been run to validate the model
completeness?
• If no – then do it!
• Otherwise get a sanity check from another team member before
scheduling review
• Is this an OLTP system or Data Warehouse/Data Mart?
– If the model is a Data Mart, is it in a Star Schema format?
• If not, why not?

43
Table Review
• Are any tables similar in nature to a table already in the corporate
repository?
– If so, can the existing definition be used with modifications?
• Is there a table level comment for every table?
• Are there detailed definitions for every table?
• Are help or reference tables needed to support implementation?
– i.e., Does the normalization look correct?

44
Table Review
• Do any of the tables require journaling (i.e., audit trail for changes)?
– If so will a table API be needed to support it?
– Note: See Table API generator in SQL Developer
• If a table is denormalized, validate the reasoning.
– Are there triggers to keep the denormalization in sync?
• Do all tables conform to existing naming standards and abbreviations?

45
Column Review
• Are appropriate audit columns defined?
• Are there column comments for every column?
• If the column is derived from other columns, have the derivation rules
and formulas been recorded?
• If a column has been denormalized from another table, has the
denormalization information been recorded?
• NEW: Are there security considerations such as PII or PHI?

COLUMN DERIVATIONS IN SDDM (I.E., VIRTUAL)
46
Virtual Column
Derivation
specifications

SDDM DIAGRAM SHOWS DERIVATIONS
47

SDDM COLUMN SECURITY SETTINGS
48

SDDM COLUMN SECURITY DISPLAY (SENSITIVE)
49

50
Column Review
• Do all columns conform to existing naming standards and
abbreviations?
• Have domains been used appropriately?
– Are there any new domains that can be identified?

51
Constraints/Referential Integrity
• Do all tables have appropriate primary key constraints?
• Should any secondary unique constraints be defined?
• Are all the foreign keys correct and enabled?
• Has cascade delete been enabled on any foreign key?
– If so, why?

52
Constraints/Referential Integrity
• Should any of the tables have table level check constraints?
– If so have they been defined correctly?
– Should they be enabled in the database?
• Are there any column level check constraints?
– Should they be enabled in the database?
• If any defined constraints will not be enabled, document the
justification and get consensus.

53
Other
• If a corporate data model is available, have all tables and columns
been mapped to entities and attributes in that model?
• If this is a warehouse model, has the source data been identified?
– Is there a model for the source systems?
– Have the target columns been mapped to a known source?
• Does the model answer all the known requirements?

BENEFITS &
CONSIDERATIONS

BENEFITS OF THE REVIEWS
55
#1 – Higher quality models & databases
Standards enforcement
• The modelers and designers can’t ignore the approved standards if
the review team holds them accountable
Issues are discovered sooner in the lifecycle
• Saves $$$$ and results in a better product, faster

BENEFITS OF THE REVIEWS
56
Knowledge transfer
• Everyone in the review learns the model and the business requirements.
Cross Training
• Jr. modelers and programmers get exposure to best practices and examples
of both good and bad data models
• This experience is often more effective than a data modeling class
A team of professionals that treat their work as business and not personal,
while deriving great personal satisfaction from a job well done

New people may be intimidated by
the process
• Outsider to the team
• Feel like being picked on
• Must accept the process as
valid
• Mention the process in
interview
Strive to hold people accountable
Must have team buy-in to
the process
57
CAUTIONS TO CONSIDER

In-experienced data modelers may
be intimidated
• Be clear up front that the critique
is being done to:
– Improve the quality of the models
– Improve their skill and knowledge
– Should not be taken as a personal
attack on their abilities
58
CAUTIONS TO CONSIDER

Keep the units of work small
• A few entities/tables
• Requires scope control
• Think SCRUM!
• Review sessions less than
one hour
Takes time but reduces
overall effort
Reviewers must be readily
available on short notice
• Get at least 2 people if urgent
59
IS IT AGILE?

CONCLUSION
60
Improving the quality of a model will save lots of $$$
over the course of a project
• And in the future!
Improving the quality of models can be done
• Requires discipline & structure
Suggested approaches
• Set standards
• Define a process
• Develop a check list
• Stick to it!

Kent Graziano
Snowflake Computing
Kent.graziano@snowflake.com
On Twitter @KentGraziano
More info at
http://snowflake.com
Visit my blog at
http://kentgraziano.com

QUESTIONS?

HOW TO SAVE PILEs of $$$BY CREATING THE BEST DATA MODEL THE FIRST TIME (Kscope 19)

Recommended

More Related Content

What's hot (20)

Similar to HOW TO SAVE PILEs of $$$BY CREATING THE BEST DATA MODEL THE FIRST TIME (Kscope 19) (20)

More from Kent Graziano (14)

Recently uploaded (20)