Melbourne: Certus Data 2.0 Vault Meetup with Snowflake - Data Vault In The Cloud

1© 2018 Snowflake Computing Inc. All Rights Reserved.
© 2018 Snowflake Computing Inc. All Rights Reserved
Data Vault and
Snowflake
Agility with a true Cloud Datawarehouse
Aamer Mushtaq
Sales Engineering, APJ
aamer.mushtaq@snowflake.com

• 640 employees
• 1500+ customers
• $923M in funding
US
2014
Founded 2012 EMEA
2016
APAC
2017
Snowflake History

Rapidly expanding
data
New users and
use cases
Diverse tools
and applications
ORGANIZATIONS ARE
STRUGLLING TO
SUPPORT THEIR
ANALYTICS NEEDS
Challenges in the world of analytics

LEGACY DATA LANDSCAPE
5
EDW
Data Sources
Data Lake
or Hadoop
Datamarts
ETL
BI / Analytics
OLTP
databases
Enterprise
applications
Web apps
Third-party
Other

IT’S JUST NOT WORKING…
6
Complexity
Manage both
infrastructure
and data
Limited
Scalability
Can’t support
all data, users
and workloads
Diversity
Unable to
consolidate
siloed datasets
Inadequate
Elasticity
Stuck with
rigid, inflexible
architectures
Rigid
Cost
Forced to
keep the lights
on 24/7

How does Snowflake
make things easier?

9© 2018 Snowflake Computing Inc. All Rights Reserved.© 2018 Snowflake Computing Inc. All Rights Reserved.
All-new SQL data
warehouse
Fully relational, ANSI SQL,
ACID compliant
Built for the cloud
Instantly scale up and
down as needed
Delivered as a service
No infrastructure, knobs,
or tuning to worry about
What is Snowflake?

A NEW ARCHITECTURE FOR DATA WAREHOUSING
Multi-cluster, shared data, in the cloud
10
Traditional Architectures Snowflake
Shared storage
Single cluster
Shared-disk
Decentralised, local storage
Single cluster
Shared-nothing Multi-cluster, shared data
Centralised, scale-out storage
Multiple, independent compute clusters

SNOWFLAKE ARCHITECTURE:
STORAGE AND COMPUTE
11
Storage separated from compute
Automatically grows without adding
nodes. Never run out of space.
Resize compute instantly
Scale up/down depending on the
business needs right now or turn off
when not in use.
Multiple clusters access data
without contention
ETL, reporting, data science, and
applications all running at the same time
without performance impact.

Management Optimisation Security Availability Transactions Metadata
12
SNOWFLAKE ARCHITECTURE:
GLOBAL SERVICES
Centralised management
Work with database and warehouses,
not servers, buckets and files
Separate metadata from storage
and compute
Use cached results, manage users and
security, all without running warehouses
Full transactional consistency
across entire system (ACID)
As soon as data commits, each cluster
sees the newest version

• Allows loading
several tables from
one source
• In one statement
Multi-Table Insert
INSERT ALL
WHEN (SELECT COUNT(*)
FROM dv_hub_country hc
WHERE hc.hub_country_key= stg.hash_key) = 0
THEN
INTO dv.hub_country (hub_country_key, country_abbrv, hub_load_dts,
hub_rec_src)
VALUES (stg.hash_key, stg.country_abbrv, stg.load_dts, stg.rec_src)
WHEN (SELECT COUNT(*)
FROM dv.sat_countries sc
WHERE sc.hub_country_key = stg.hash_key AND
sc.hash_diff = MD5(country_name)) = 0
THEN
INTO dv.sat_countries (hub_country_key, sat_load_dts,
hash_diff, sat_rec_src, country_name)
VALUES (stg.hash_key, stg.load_dts,
stg.hash_diff, stg.rec_src, stg.country_name)
SELECT MD5(country_abbrv) AS hash_key, country_abbv, country_name,
MD5(country_name) AS hash_diff,
CURRENT_TIMESTAMP AS load_dts,
stage_rec_src AS rec_src
FROM stage.country stg;

Hashing
• Snowflake supports both MD5, SHA1 and SHA2 algorithms in standard,
HEX, and binary formats
• i.e. SHA2(), SHA2_HEX() or SHA2_BINARY()
SELECT SHA2_BINARY('Snowflake', 384);
736BD8A53845348830B1EE63A8CD3972F031F13B111F66FFDEC2271A7AE709662E5
03A0CA305BD50DA8D1CED48CD45D9

Perhaps You Wish to Split for Security Reasons?
From This
DV Physical Partitioning
Single Snowflake Database
Schema layout
In Snowflake – you
can easily split for
security reasons
To THIS!
DV Logical Partitioning
Non-sensitive
data
PII or sensitive data with
a Link to non-sensitive
data
Snowflake DB 1 Snowflake DB 2

© 2018 Snowflake Computing Inc. All Rights Reserved.
ETL/ELT
snowpipe
XS
S
M
M
L
Sales
Data
Science
Global Services
Transactional Control
Security
Query Planning & Optimisation
Logical Model
AWS QuickSight
S
AWS Glue

Instant Elasticity = Maximum Agility

ETL/ELT
snowpipe
XS M
Sales
Data
Science
M…
XLS
Multi-cluster
Global Services
Security
Logical Model
AWS QuickSight
AWS Glue

Data Vault Loading Pattern
- Major Synchronization Points
• All processes within sets done in
parallel
• Process sets ”wait” for previous
set completion
• Sets run as soon as data is ready
• No other waiting needed
• Load dependencies maximally
reduced
Stage Loads Vault Loads

Applying Virtual Warehouses to Data Vault
1. Use a Multi-Cluster
Warehouse for Source to
Stage loads
2. Use a separate Warehouse
for each DV entity type
• Zero contention
• Maximum parallelism
• Maximum flexibility -
each can be
independently sized

• Anticipated surges
• Explicitly increase WH nodes (T-shirt size) when expecting more data
• Explicitly increase MCWH minimum clusters when expecting more queries
• Can do both at once with ALTER WAREHOUSE
• Use cron or other scheduling/orchestration tool
• Unanticipated surges
• Rely on MCWH maximum clusters for some extra headroom
• Maximize Business Agility!
• Responsiveness for users
• Throughput and value extracted from variable compute power
• Minimize
• Cost and administrative overhead
Agile Warehouse Scaling – Best Practices

ETL/ELT
snowpipe
XS M
Sales
Data
Science
M…
XL
Multi-cluster
Structured &
semi-structured
Global Services
Security
Logical Model
AWS QuickSight
AWS Glue

{
"city": {
"coord": {
"lat": 27.716667,
"lon": 85.316666
},
"country": "NP",
"findname": "KATHMANDU",
"id": 1283240,
"name": "Kathmandu",
"zoom": 7
},
"clouds": {
"all": 75
},
"main": {
"humidity": 94,
"pressure": 1012,
"temp": 294.15,
"temp_max": 294.15,
"temp_min": 294.15
},
"time": 1504476371,
"weather": [
{

JSON Support with SQL - Variant
Apple 101.12 250 FIH-2316
Pear 56.22 202 IHO-6912
Orange 98.21 600 WHQ-6090
Structured data
(CSV)
Semi-structured data
(e.g. JSON, Avro, XML)
{ "firstName": "John",
"lastName": "Smith",
"height_cm": 167.64,
"address": {
"streetAddress": "21 2nd Street",
"city": "New York",
"state": "NY",
"postalCode": "10021-3100"
},
"phoneNumbers": [
{ "type": "home", "number": "212 555-1234" },
{ "type": "office", "number": "646 555-4567" }
]
}
select v:lastName::string as last_name
from json_demo;
All Your Data!

• Create Sats using a VARIANT
• Hub Key
• Load DTS
• Rec Source
• VARIANT
• Load JSON doc into VARIANT column
• Create a View on the Sat for Business Vault
• View presents only the JSON attributes needed for
current use case
Combine Data Vault & Schema-on-Read
Benefits
ØNo change to Sat
definitions if structure
changes
ØBusiness Vault view is
unaffected
ØChange or add new BV
views over time as
needed

ETL/ELT
snowpipe
XS
S
M
Sales
Data
Science
M…
Test/Dev
Clone
XL
Multi-cluster
Structured &
semi-structured
Global Services
Security
Logical Model
AWS QuickSight
AWS Glue

ETL/ELT
snowpipe
XS
M
M
L
External
Finance
Sales
Data
Science
M…
Clone
Share
Data protection
& time travel
XL
Multi-cluster
Structured &
semi-structured
Global Services
Security
Logical Model
AWS QuickSight
AWS Glue
S
Test/Dev

What are customers
doing with Snowflake?

Modern Data Landscape
33
Logical Datamarts
BI / Analytics
EDW
Data Lake
or Hadoop
ETL or ELT
OLTP
databases
Enterprise
applications
Web apps
Third-party
Other
Data Sources

Melbourne: Certus Data 2.0 Vault Meetup with Snowflake - Data Vault In The Cloud

Story Time

Founded in 2008, Aptus Health connects health and life
sciences companies with healthcare professionals,
healthcare consumers, and other members of the
healthcare ecosystem.
The company creates multichannel marketing campaigns
aimed at physicians and healthcare consumers through the
many digital channels the company owns and operates.
Aptus Health

Where They Started
Mobile App
Campaign
Management
Sample
Requests
Daily ETL
Apply Business Rules
Data Warehouse
- User Activity
- Campaign Activity
- Sample Request Activity
- User Data
- Mobile App Activity
Cube
Reports
Web & Mobile
Analytics
Extracts
Fill
Stage
Load F
D
D
D
D
D
D
D
D
Engagement
Platform

• A business-driven roadmap in collaboration with Engineering to drive priorities
• Central availability of data related to Healthcare Providers across all platforms
• Ability to efficiently answer business questions: new or previous
• New types of analysis should not require weeks of data preparation
• Do not lose data that could have future analytical value
• Ability to analyze historic events
• Clear documentation of all calculations/derivations of data for complete
transparency in meaning of information & to instill confidence
• Master Data Management: stop passing around spreadsheets
• Framework that will be extensible to both US and international data
• Timely data: minutes, not days
• Business reporting enabled with limited impact to data management projects
What did the business need?

• Breaking down data silos
• Integrate physically and logically disparate data
• Acceleration of integrating new data sets
• Enable broad use of data sets
• Ease integration of datasets across AH avoiding point-to-point integrations
• Scale, Elasticity, and Cost-effectiveness
What was IT asking for?

Diversity
One place for
all your data
Scalability
Any scale of
data, users and
workloads
Flexible
Cost
Pay for what
you use, when
you use it
Simplicity
Simple,
serverless,
plug-and-play
Elasticity
Size for what
you need
right now
Snowflake Business Benefits

THANK YOU

Melbourne: Certus Data 2.0 Vault Meetup with Snowflake - Data Vault In The Cloud

Recommended

More Related Content

What's hot (20)

Similar to Melbourne: Certus Data 2.0 Vault Meetup with Snowflake - Data Vault In The Cloud (20)

More from Certus Solutions (20)

Recently uploaded (20)

Melbourne: Certus Data 2.0 Vault Meetup with Snowflake - Data Vault In The Cloud