Cutting to the chase for Machine Learning Analytics Ecosystem & AWS Lake Formation

1© 2019 Amazon Web Services, Inc. or its affiliates. All rights reserved | 1© 2019 Amazon Web Services, Inc. or its affiliates. All rights reserved |
Asif Abbasi – Specialist SA Analytics
03.Nov.2019
Cutting to the chase for Machine Learning
Analytics Ecosystem & AWS Lake Formation
@masifabbasi

2© 2019 Amazon Web Services, Inc. or its affiliates. All rights reserved |
Agenda
• Analytics Ecosystem
• Why did we build AWS Lake Formation?
• What is AWS Lake Formation?
• How does AWS Lake Formation help you?

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Analytics Value Stream
idea insight
COLLECT STORE
PROCESS/
ANALYZE
CONSUME
time to first answer

Agile Analytics
• Experiment
• Invest in and scale up success
• Fail fast
• Adapt and evolve rapidly

A Modern Data Platform
• Treat data as a reusable asset
(while keeping the cost of reuse low)
• Apply an open set of processing
strategies in parallel

Data Lake
Store all your data, forever,
at every stage of its lifecycle
Apply it using the right tool for the job

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Storage Foundations: Amazon S3

Events and Lifecycle Management
Standard
Active data Archive dataInfrequently accessed data
Standard - Infrequent
Access
Amazon Glacier
Create
Delete

Benefits of Using S3 as Your Data Lake Foundation
• Unlimited number of objects and
volume
• 99.99% availability
• 99.999999999% durability
• Versioning
• Tiered storage via lifecycle policies
• SSL, client/server-side encryption
at rest
• Low cost (~ $2700/month/100 TB)
• Natively supported by big data
frameworks (eg. Spark, Hive,
Presto)
• Decouples storage and compute
• Run transient compute clusters
(with Amazon EC2 Spot
Instances)
• Multiple, heterogeneous
clusters can use same data

Ingest
Kinesis Firehose

Data Catalog: Discover and Govern Your Data
Kinesis Firehose

Glue Data Catalog
Hive metastore-compatible, highly-available
metadata repository:
• Search metadata for data discovery
• Connection info – JDBC URLs, credentials
• Classification for identifying and parsing
files
• Versioning of table metadata as schemas
evolve and other metadata are updated
• Table definitions – usable by Redshift,
Athena, Glue, EMR
Populate using Hive DDL, bulk import, or
automatically through crawlers.

Catalog and Query Multiple Sources

Access Control and Auditing
Kinesis Firehose

Access Control and Auditing
IAM
Amazon
S3
Amazon
DynamoDB
Amazon EMR Amazon
Kinesis
Amazon
Athena
Service API Access
AWS
CloudTrail
Identity and Access Management
• Security at the API and data level
3rd party ecosystem security tools
• Blue Talon, Apache Ranger, Sentry, etc
Storage-level access logging and audit
• AWS CloudTrail logs API calls
• S3 access logging
• Log analytics with Athena
Encryption
• AWS Server-Side Encryption
• AWS Key Management Service
• AWS CloudHSM
• Custom materials providers

Process and Analyze
Kinesis Firehose
Athena
Query Service

Job Authoring with AWS Glue
• Python/Scala code
generated by AWS Glue
• Connect a notebook or
IDE to AWS Glue
• Existing code brought into
AWS Glue

Job Execution with AWS Glue
• Schedule-based
• Event-based
• On demand

Serverless Queries: Amazon Athena
• Interactive queries
• ANSI SQL
• No infrastructure or administration
• Zero spin up time
• Query data in its raw format
• AVRO, Text, CSV, JSON, weblogs, AWS
service logs
• Convert to an optimized form like ORC
or Parquet for the best performance
and lowest cost
• No loading of data, no ETL required
• Stream data from directly from Amazon S3
• Take advantage of Amazon S3 durability
and availability

Data Warehousing: Amazon Redshift
Managed, massively parallel, petabyte-scale, relational data warehouse
Scale from 160GB to 2PB online
Automatic streaming backup/restore to S3,
Automatic failover and recovery
ANSI SQL interface
Load data from S3, DynamoDB and EMR

Query Exabytes of Data: Redshift Spectrum
Amazon
Redshift
JDBC/ODBC
...
1 2 3 4 N
Amazon S3
Exabyte-scale object
storage
Data
Catalog
Apache Hive
Metastore
Run Amazon Redshift SQL
queries against exabytes
of data in Amazon S3

Managed Hadoop Framework: Amazon EMR
Scalable compute clusters as a
service
• Create ephemeral clusters –
only pay for the compute
you need
• Integrated with S3 (EMRFS)
• Use spot instances to
reduce processing time
AND costs
• Dynamically scale up and
down
• Data encryption at-rest and
in-transit

Business Analytics Service: Amazon QuickSight
Deep Integration with
AWS Data Sources
Amazon RDS,
Aurora
Amazon
Redshift
Amazon
Athena
Amazon S3
Flat Files
• Fully managed
• 1/10 cost of traditional BI solutions
• Integrated with AWS data sources
and third-party sources
• SPICE – Super-fast, Parallel, In-
memory, Calculation Engine
• Collaborate, share and publish data
sets and analyses

Building a Data Strategy on AWS
Kinesis Firehose
Athena
Query Service

An Open Ecosystem of Applications
Processing &
Analytics
BI & Data Visualization
Kinesis Streams
& Firehose
Batch
EMR
Hadoop, Spark,
Presto
Redshift
Data Warehouse
Athena
Query Service
AWS Batch
Predictive
Real-time
AWS Lambda
Apache Storm
on EMR
Apache
Flink on
EMR
Spark
Streaming on
EMR
Elasticsearch
Service
Kinesis Analytics,
Kinesis Streams
EastiCache DAX
Online/Transactional
DynamoDB
(NoSQL)
Aurora
(Relational)
Neptun
e
(Graph)

26© 2019 Amazon Web Services, Inc. or its affiliates. All rights reserved | 26© 2019 Amazon Web Services, Inc. or its affiliates. All rights reserved |
Data Lakes

A data lake is a centralized repository that allows
you to store all your structured and unstructured
data at any scale

Why data lakes?
Data Lakes provide:
Relational and non-relational data
Scale-out to EBs
Diverse set of analytics and machine learning tools
Work on data without any data movement
Designed for low cost storage and analytics
OLTP ERP CRM LOB
Data Warehouse
Business
Intelligence
Data Lake
1001100001001010111
0010101011100101010
0001011111011010
0011110010110010110
0100011000010
Devices Web Sensors Social
Catalog
Machine
Learning
DW
Queries
Big data
processing
Interactive Real-time

Any analytic
workload, any scale,
at the lowest possible
cost
AWS Direct Connect
AWS Snowball
AWS Snowmobile
AWS Database Migration Service
AWS IoT Core
Amazon Kinesis Data Firehose
Amazon Kinesis Data Streams
Amazon Kinesis Video Streams
On-premises
Data Movement
Amazon SageMaker
AWS Deep Learning AMIs
Amazon Rekognition
Amazon Lex
AWS DeepLens
Amazon Comprehend
Amazon Translate
Amazon Transcribe
Amazon Polly
Amazon Athena
Amazon EMR
Amazon Redshift
Amazon Elasticsearch Service
Amazon Kinesis
Amazon QuickSight
Analytics
Machine Learning
Real-time
Data Movement

Typical steps of building a data lake
Setup storage1
Move data2
Cleanse, prep, and
catalog data
3
Configure and enforce
security and compliance
policies
4
Make data available
for analytics
5

Building data lakes can still take months

Data preparation accounts for ~80% of the work
Building training sets
Cleaning and organizing data
Collecting data sets
Mining data for patterns
Refining algorithms
Other

Sample of steps required Find sources
Create Amazon Simple Storage Service (Amazon S3) locations
Configure access policies
Map tables to Amazon S3 locations
ETL jobs to load and clean data
Create metadata access policies
Configure access from analytics services
Rinse and repeat for other:
data sets, users, and end-services
And more:
manage and monitor ETL jobs
update metadata catalog as data changes
update policies across services as users and permissions change
manually maintain cleansing scripts
create audit processes for compliance
…
Manual | Error-prone | Time consuming

Enforce security policies
across multiple services
Gain and manage new
insights
Identify, ingest, clean, and
transform data
Build a secure data lake in days
AWS Lake Formation

How it works

Register existing data or import new
Amazon S3 forms the storage layer for
Lake Formation
Register existing S3 buckets that
contain your data
Ask Lake Formation to create required
S3 buckets and import data into them
Data is stored in your account. You
have direct access to it. No lock-in.
Data Lake Storage
Data
Catalog
Access
Control
Data
import
Lake Formation
Crawlers ML-based
data prep

Easily load data to your data lake
logs
DBs
Blueprints
Data Lake Storage
Data
Catalog
Access
Control
Data
import
Lake Formation
Crawlers ML-based
data prep
one-shot
incremental

With blueprints
You
1. Point us to the source
2. Tell us the location to load
to in your data lake
3. Specify how often you want
to load the data
Blueprints
1. Discover the source table(s)
schema
2. Automatically convert to
the target data format
3. Automatically partition the
data based on the
partitioning schema
4. Keep track of data that was
already processed
5. You can customize any of
the above

Blueprints build on AWS Glue

Easily de-duplicate your data with ML transforms

Fuzzy de-duplication – under the hood

Fuzzy de-duplication – Innovations
400M+
7.5B+
2.5

Secure once, access in multiple ways
Data Lake Storage
Data
Catalog
Access
Control
Lake Formation
Admin

Security permissions in Lake Formation
Control data access with simple
grant and revoke permissions
Specify permissions on tables and
columns rather than on buckets
and objects
Easily view policies granted to a
particular user
Audit all data access at one place

Security permissions in Lake Formation
Search and view permissions
granted to a user, role, or group in
one place
Verify permissions granted to a
user
Easily revoke policies for a user

Grant table and column-level permissions
User 1
User 2

Security – deep dive
User
IAM users, roles
Active Directory Amazon S3

Search and collaborate across multiple users
Text-based, faceted search
across all metadata
Add attributes like Data
owners, stewards, and other as
table properties
Add data sensitivity level,
column definitions, and others
as column properties
Text-based search and filtering
Query data in Amazon Athena

Audit and monitor in real time
See detailed alerts in the
console
Download audit logs for
further analytics
Data ingest and catalog
notifications also published to
Amazon CloudWatch events

Example: a data lake in 3 easy steps

Step 1: Blueprints to ingest data

Monitor the import
1

Imported data as table in the data lake

Step 2: Grant permissions to securely share data

Step 3: Run query in Amazon Athena

AWS Lake Formation Pricing
No additional charges – Only pay for the
underlying services used.

Customer interest
“We are very excited about the launch of AWS Lake
Formation, which provides a central point of control to
easily load, clean, secure, and catalog data from thousands
of clients to our AWS-based data lake, dramatically
reducing our operational load. … Additionally, AWS Lake
Formation will be HIPAA compliant from day one …”
Aaron Symanski, CTO, Change Healthcare
“I can’t wait for my team to get our hands on AWS Lake
Formation. With an enterprise-ready option like Lake
Formation, we will be able to spend more time deriving
value from our data rather than doing the heavy lifting
involved in manually setting up and managing our data
lake.”
Joshua Couch, VP Engineering, Fender Digital

Thank you!

Cutting to the chase for Machine Learning Analytics Ecosystem & AWS Lake Formation

More Related Content

Similar to Cutting to the chase for Machine Learning Analytics Ecosystem & AWS Lake Formation (20)

More from AWS Riyadh User Group (20)

Recently uploaded (20)

Cutting to the chase for Machine Learning Analytics Ecosystem & AWS Lake Formation