Making Sense of Schema on Read

Making Sense of
Schema-on-Read
Modeling JSON
KENT GRAZIANO, CHIEF TECHNICAL EVANGELIST I NOVEMBER 2018
KentGraziano

• Chief Technical Evangelist, Snowflake Computing
• Oracle ACE Director, Alumni (DW/BI)
• OakTable Network
• Blogger – The Data Warrior
• Certified Data Vault Master and DV 2.0 Practitioner
• Former Member: Boulder BI Brain Trust (#BBBT)
• Member: DAMA Houston & DAMA International
• Data Architecture and Data Warehouse Specialist
• 30+ years in IT
• 25+ years of Oracle-related work
• 20+ years of data warehousing experience
• Author & Co-Author of a bunch of books (Amazon)
• Past-President of ODTUG and Rocky Mountain Oracle
User Group
About me
© 2018 Snowflake Computing Inc. All Rights Reserved

3 years in stealth + 3 years GA
700+ employees
Over 2000
customers today
Over $850M in venture
funding from leading
investors
First customers 2014,
general availability
2015
Founded 2012 by
industry veterans
with over 120
database patents
Queries processed in Snowflake per day:
Largest single table:
Largest number of tables single DB:
Single customer most data:
Single customer most users:
60 Million
68 Trillion Rows
200,000
> 40 PB
> 10,000

• Schema-on-Read vs Schema-on-Write
• Why we still need data modeling
• What is JSON?
• Example JSON #1
• Simple 3NF model
• Simple Data Vault model
• Example JSON #2
• 3NF model
• Data Vault model
AGENDA

• Schema-on-Read
• Popularized in document stores and NoSQL dbs
• No upfront modeling
• No predefined structure
• Called semi-structured or flexible-structure data
• Can change contents and structure over time
• Load & Go
• Agile!
Defining Terms

• Schema-on-Write
• What we do in RDBMS today
• Requires knowing the structure in advance
• Upfront modeling & table design required
• Must map source data to the database tables
• ETL/ELT may break if the source data changes
Defining Terms

40 Zettabytes by 2020
Web ERP3rd party apps Enterprise apps IoTMobile

It’s not the data itself
it’s how you take full advantage of the insight it provides
Web ERP3rd party apps Enterprise apps IoTMobile

• We all do!
• To take advantage of all this data, we have to use it
• Schema-on-Read
• There is a SCHEMA – which means a model!
• To query the data requires knowing the structure
• Which means the MODEL of the data or “document”
• Few reporting or BI tools can infer the schema
• So we have to transform it, somehow
• Load to tables and columns?
• Expose with a SQL view?
Who needs data modeling anyway?

What is JSON?
• Java
• Script
• Object
• Notation
A minimal, readable format
for structuring data.
It is used primarily to
transmit data between a
server and a web
application, as an
alternative to XML

• There is LOTS of it out there
• JavaScript is popular
• REST API’s for IoT & Mobile
• Application and web logs – Social Media
• Self-describing so very portable
• Open datasets published in JSON
• Data.gov
• Datasf.org
• Data.cityofNewYork.us
• Opportunity for analysis!
Why worry about JSON?

JSON Support with SQL
Apple 101.12 250 FIH-2316
Pear 56.22 202 IHO-6912
Orange 98.21 600 WHQ-6090
Structured data Semi-structured data
(e.g. JSON, Avro, XML)
{ "firstName": "John",
"lastName": "Smith",
"height_cm": 167.64,
"address": {
"streetAddress": "21 2nd Street",
"city": "New York",
"state": "NY",
"postalCode": "10021-3100"
},
"phoneNumbers": [
{ "type": "home", "number": "212 555-1234" },
{ "type": "office", "number": "646 555-4567" }
]
}
select v:lastName::string as last_name
from json_demo;
All Your Data!

{
"colors": [
{
"color": "black",
"category": "hue",
"type": "primary",
"code": {"rgba": [255,255,255,1],
"hex": "#000”
}
},
{
"color": "green",
"category": "hue",
"type": "secondary”,
"code": { "rgba": [0,255,0,1],
"hex": "#0F0"
}
},
]
}
This is a JSON Document
Enclosed by { }
Elements are Key-Value Pairs
Key : Value
Elements may have nested Keys
Delineated by more { }
Some Values may be Arrays
Delineated by [ ]

"colors": [
{
"color": "black",
"category": "hue",
"type": "primary",
"code":
{
"rgba": [255,255,255,1],
"hex": "#000”
}
}
]
JSON as 3NF –
Logical Model

"colors": [
{
"color": "black",
"category": "hue",
"type": "primary",
"code":
{
"rgba": [255,255,255,1],
"hex": "#000”
}
}
]
JSON as 3NF –
Schema Model

"colors": [
{
"color": "black",
"category": "hue",
"type": "primary",
"code":
{
"rgba": [255,255,255,1],
"hex": "#000”
}
}
]
JSON as Denormalized –
Relational Model

JSON as
Data Vault
"colors": [
{
"color": "black",
"category": "hue",
"type": "primary",
"code":
{
"rgba": [255,255,255,1],
"hex": "#000”
}
}
]

• That is the point of schema-on-read
• No changes to ingest the data
• NoSQL, Snowflake, Oracle
• Example
• More attributes on Color Category or Color Type
• Like “Description”
• In a 3NF model
• Add new columns to entities/tables
• ALTER TABLE required
• In a Data Vault model
• Add new Sat tables on existing Hubs
• CREATE TABLE required
• No change required to existing tables
What if the JSON changes?

{
"fullName": "Johnny Appleseed",
"age": 42,
"gender": "Male",
"phoneNumber":
{
"areaCode": "415",
"subscriberNumber": "5551234”
},
"children":
[
{
"name": "Jayden",
"gender": "Male",
"age": "10" },
{
"name": "Emma",
"gender": "Female",
"age": "8" },
{
"name": "Madelyn",
"gender": "Female",
"age": "6" }
],
"citiesLived": [
{
"cityName": "London”,
"yearsLived": [ "1989", "1993", "1998", "2002" ]
},
{
"cityName": "San Francisco",
"yearsLived": [ "1990", "1993", "1998", "2008" ]
},
{
"cityName": "Portland",
"yearsLived": [ "1993", "1998", "2003", "2005" ]
},
]
}
Nested Elements
Nested Array of Elements
Nested Array of Values,
Within a Nested Array
Of Elements

JSON as 3NF –
Logical Model
"fullName": "Johnny Appleseed",
"age": 42,
"gender": "Male",

JSON as 3NF –
Logical Model
"phoneNumber":
{
"areaCode": "415",
"subscriberNumber": "5551234”
},

JSON as 3NF –
Logical Model
"children":
[
{
"name": "Jayden",
"gender": "Male",
"age": "10" },
{
"name": "Emma",
"gender": "Female",
"age": "8" },
{
"name": "Madelyn",
"gender": "Female",
"age": "6" }
],

JSON as 3NF –
Logical Model
"cityName": "London”,

JSON as 3NF –
Logical Model {
"cityName": "Portland",
"yearsLived": [ "1993", "1998", "2003", "2005" ]
},

JSON as 3NF -
Schema Model
• Can handle some
JSON schema
changes
• Kids get a phone!
• Kids move out!
• Extensions
• More details on City
• Add columns
• More details on
Children
• Add columns or a
dependent table

JSON as
Data Vault

• Can handle some
JSON schema
changes
• Two parents, same kids
• Kids get a phone!
• Kids move out!
• Easy Extensions
• More details on City
• Add a Sat
• Add Link(s)
• More details on Children
• Add a Sat on Link
JSON as
Data Vault
Special case –
Transaction Link
Year included
in UKs

• We still need data models and data modelers
• Schema-on-Read does not mean there is no model
• To READ the data we must understand the SCHEMA
• In the DB world that means we need a model
• Some model types can be easily extended for JSON changes
• Once the schema is understood
• Can be expressed as any type of model
• 3NF
• Data Vault
• Star
• Denormalized
• Object model
• Etc.
Conclusion

Available on
Amazon.com
https://www.amazon.com/
Better-Data-Modeling-
Enhancing-Developer-
ebook/dp/B00UK75LYI/
SHAMELESS PLUG:

Available on
Amazon.com
http://www.amazon.com
/Better-Data-Modeling-
Introduction-
Engineering-
ebook/dp/B018BREV1C/
SHAMELESS PLUG:

Discover the performance, concurrency, and simplicity of
Snowflake
As easy as 1-2-3!
01 Visit Snowflake.com
02 Click “Try for Free”
03 Sign up & register
Snowflake is the only data warehouse built for the
cloud. You can automatically scale compute up,
out, or down—independent of storage. Plus, you
have the power of a complete SQL database, with
zero management, that can grow with you to
support all of your data and all of your users. With
Snowflake On Demand™, pay only for what you
use.
Sign up and receive
$400 worth of free
usage for 30 days!

Kent Graziano
Snowflake Computing
Kent.graziano@snowflake.com
On Twitter @KentGraziano
More info at
http://snowflake.com
Visit my blog at
http://kentgraziano.com
Contact Information

THANK YOU

Making Sense of Schema on Read

Recommended

More Related Content

What's hot (20)

Similar to Making Sense of Schema on Read (20)

More from Kent Graziano (20)

Recently uploaded (20)

Making Sense of Schema on Read