Amazon Redshift

Analyze Data Across Your Lake House with Amazon Redshift

Rajesh Francis
Sr. Analytics Specialist Solutions Architect

© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Discussion Topics

• Data Trends and Lake House architecture Bradley Todd

Liberty Mutual, Technology Architect

• Amazon Redshift Architecture Redshift allows us to quickly spin up

clusters and provide our data scientists
with a fast and easy method to access
• Analyze all your data data and generate insights

• Reference Architectures

• Performance and Scale

• Price for performance

© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
 Amazon Redshift
Data Trends Large-Scale Data Warehousing Service
Used by more
customers for
their data
warehouse
workloads than
anyone else

© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Challenges of data analytics at scale
VARIETY PERFORMANCE COST

Variety of sources Slow Increasing and

and data types performance unpredictable cost

Multiple analytics needs

Difficult to Inflexible tools
manage systems

Data volume and velocity Complex Security,

to scale compliance
© 2021, Amazon Web Services, Inc. or its Affiliates.
Traditional architectures lead to dark data
Traditional architectures & on-prem data warehousing lead to dark data –
data that is collected but challenging to extract insights from that data.

• Can’t scale easily or on-demand

Scale
• Long lead times for hardware procurement & upgrades

• High operational costs

Cost
• Compute tied to storage

• Proprietary formats
Machine
Anti- • Data silos learning

democratization • Need to ingest, transform data before analysis Data lakes BI +

analytics
• Limits on users and data Data Open formats
Central catalog
warehousing

Architecture • One size fits all approach

© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
 Business use cases for Lake House architecture
Capture all my Customer Orders, Income/Expenses, Credit/Debit,
Clickstream data
Run Supply Chain Optimization process
Route Planning… Store information about Customer, Product, Supplier details
Analyze weather patterns…...
What are my open orders to fulfill?

Has the product been delivered to my customer?

Collect large volume of raw
structured, semi-structured or What is my current balance or inventory?
unstructured data

Establish centralized governance

models
Predict customer behavior to….
manage inventory, target… Build a central repository of structured curated data for
medications. analysis to serve multiple business groups and users
Customer Retention, Risk…
Profit analysis: Analyze all customer purchases over the
Management.
last three years and compare against my costs
Predict weather patterns
Risk assessment: Analyze historical 7 years of data for credit
risks. Analyze 20+ years of data for natural disaster

© 2021, Amazon Web Services, Inc. or its Affiliates.

From traditional Data Warehouse to Lake House approach

Relational
databases

Business Business
Intelligence Intelligence
Non-
Big data
relational
processing
databases

Data silos to Data Lake

DW Silo 1 DW Silo 2 Log Machine

analytics learning

Data
warehousing
OLTP ERP CRM LO Devices Web Sensors Socia
B l

© 2021, Amazon Web Services, Inc. or its Affiliates.

 Lake House architecture on AWS
Amazon
Aurora
Scalable data lakes

Amazon Amazon Purpose-built

EMR DynamoDB data services

Amazon
Athena
Seamless
Amazon data movement
S3

Amazon Amazon
Elasticsearch SageMaker Unified governance
Service

Performant and
Amazon cost-effective
Redshift
© 2021, Amazon Web Services, Inc. or its Affiliates.
 Amazon Redshift
Analyze all your data with the fastest and most widely used cloud data warehouse

Amazon
Aurora Analyze all your data
Deepest integration with your data lake

Amazon Amazon
EMR DynamoDB

Amazon
Athena

Amazon
S3
Performance at any scale
Up to 3x better price performance than other cloud DW
Amazon
Amazon
Elasticsearch
SageMaker
Service

Amazon

Lower your costs

Redshift

At least 50% less expensive than other cloud DW

© 2021, Amazon Web Services, Inc. or its Affiliates.

 Amazon Redshift innovates to meet your needs

NEW! NEW! NEW! UPDATED! NEW! NEW!

Analyze all your data Amazon Data Super data type Federated AWS Partner Amazon Redshift Data lake
Redshift ML sharing with JSON support query Lambda console Spectrum + export
Lake house with AWS integration UDF integration AWS Lake Formation

UPDATED! NEW! NEW! UPDATED! NEW! NEW!

Performance & scale RA3 nodes & AQUA Performance Materialized 100K tables HyperLogLog Concurrency
managed storage tuning: views scaling
Fast and self-tuning
automated

UPDATED! NEW! NEW!

Low cost & best value Automatic Cross-AZ cluster Data API On-demand Pause Cost controls Built-in
workload recovery and RIs and resume security features
Predictable costs manager

© 2021, Amazon Web Services, Inc. or its Affiliates.

Customers – sample list
Tens of thousands of customers process exabytes of data with Amazon Redshift daily

NTT DOCOMO WARNER Yelp Jack in the Box Pfizer

Moved >10 PB of BROS. Enabling a Improved ops by Provide scientists
data from on- Performance, scale, data-driven moving off of with near real-time
premises to cloud cost-efficiency organization with on-premises DW analysis
concurrency scaling

© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
 Amazon Redshift
Analyze all your data

Amazon
Aurora Analyze all your data
Deepest integration with your data lake

Amazon Amazon
EMR DynamoDB

Amazon
Athena

Amazon
S3
Performance at any scale
Up to 3x better price performance than other cloud DW
Amazon
Amazon
Elasticsearch
SageMaker
Service

Lower your costs

Amazon At least 50% less expensive than other cloud DW
Redshift

© 2021, Amazon Web Services, Inc. or its Affiliates.

Analyze all your data
With a lake house approach to analytics

BI and analytics apps

Connect apps to analyze and visualize your data

SQL
Federated Query
Amazon
AWS Glue Elastic Views Redshift ML

Operational
Amazon Redshift ML & analytics
databases Spectrum query S3 Data lake export services
Query live data Analyze open
standards-based
data formats

Amazon S3 data lake

Keep up to exabytes of data in S3

© 2021, Amazon Web Services, Inc. or its Affiliates.

 Redshift Architecture Overview

Anil Chalasani
Gainsight, VP Product Operations

Using Redshift’s DC2 node, we

generate reports 35 percent faster.
This enables our customers to
spend more time curating and
visualizing their data in Gainsight
to take advantage of opportunities
to drive customer success

© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Redshift cluster architecture
SQL Clients / BI Tools
• Leader node
• SQL endpoint JDBC/ODBC
• Stores metadata
• Coordinates parallel SQL processing & Leader
ML optimizations node
• Leader node is no-charge for clusters
with 2+ nodes Compute Compute Compute
• Compute nodes node node node
• Local, columnar storage
• Executes queries in parallel Load … … … … …

Spectrum
Redshift
• Load, unload, backup, restore from S3 Unload
Query ...
Backup 1 2 3 4 N
• Amazon Redshift Spectrum nodes Restore
Execute queries directly against data lake

• Massively parallel, shared nothing Redshift Managed Storage

architecture Amazon S3
Exabyte-scale object storage

© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Evolving architecture (2017–2020)
Incremental features released in the last few years
Redshift Spectrum for Concurrency Scaling for RA3 with independent Data sharing across clusters
data lake analytics bursty workloads compute and storage scaling
Producer Consumers

Spectrum AQUA - Query acceleration with computational cache

AQUA node AQUA node AQUA node AQUA node

Amazon Redshift Managed Storage

Amazon S3

© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Data Sharing
A SECURE AND EASY WAY TO SHARE DATA ACROSS AMAZON REDSHIFT CLUST ERS

“Data sharing feature

seamlessly allows multiple
Amazon Redshift clusters to
query data located in our RA3
clusters and their managed
storage. This eliminates our
concerns with delays in making
data available for our teams,
reduces the amount of data
duplication and associated
backfill headache. We now can
concentrate even more of our
time making use of our data in
Amazon Redshift and enable
better collaboration instead of
data orchestration.”
• Instant, granular, high-performance data access • Workloads accessing shared data are isolated from Steven Moy, Yelp
without data copies / movement each other
• Live and consistently updating views of data across • Use cases: Cross-group collaboration and sharing,
all consumers workload isolation and chargeability, data as a
• Secure and governed collaboration within and service
across organizations and with external parties • Sharing to other AWS analytic services – coming
soon
© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Redshift ML
EASILY CREATE AND TRAIN ML MODELS USING SQL QUERIES WITH AMAZON SAGEMAKER

Use case: Product recommendations, fraud

prevention, reduce customer churn

Create, train, and apply ML models using SQL

Automatic selection of ML algorithms or

select your algorithm with XGBoost

Automatic pre-processing, creation,

training, deployment of your model
CREATE MODEL demo_ml.customer_churn
Deploy inference models locally in Amazon FROM (SELECT c.age, c.zip, c.monthly_spend,
Redshift; run an inference as invoking a user- c.monthly_cases, c.active FROM
defined function as part of SQL statements customer_info_table c)
TARGET c.active;

© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Redshift Federated Query
UNIFIED ANALYTICS ACROSS DATABASES, DATA WAREHOUSE, AND DATA LAK E

Use case: Integrate operational data with DW

and data lake for real-time analytics
Amazon Redshift
Analytics on operational data without JDBC/ODBC
data movement and ETL delays
Query and join data from one or more
Amazon RDS and Aurora PostgreSQL
databases

Flexible and easy way to ingest data

avoiding complex ETL pipelines

Intelligent distribution of computation to remote

sources to optimize performance
Amazon Aurora Amazon S3 Amazon RDS
Amazon RDS and Aurora MySQL support PostgreSQL, data lake PostgreSQL,
MySQL MySQL

© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Native semi-structured data support
id name phones
SUPER
INTEGER SUPER

New data type: SUPER [{"type":"work",

{"given":"Jane", "num":"9255550100"},
1
"family":"Doe"} {"type":"cell",
Easy, efficient, and powerful JSON processing "num": 6505550101} ]
{"given":„Richard", [{"type":"work",
2
Fast row-oriented data ingestion "family":„Roe"}, "num": 5105550102}]

Fast column-oriented analytics with

materialized views over SUPER/JSON
SELECT name.given AS firstname, ph.num
Access to schema-less nested data with FROM customers c, c.phones ph
WHERE ph.type = ‘cell’;
easy-to-use SQL extensions powered
by the PartiQL query language firstname | num
----------+---------------
"Jane" | 6505550101

© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Redshift Spectrum Overview Amazon
Aurora

Redshift Spectrum is a feature of Redshift that allows Amazon

EMR
Amazon
DynamoDB

SQL queries on external data stored in Amazon S3 Amazon

Athena

Amazon
Benefits S3

• Enables the Lake House pattern to query exabytes of data Amazon

Elasticsearch
Amazon
SageMaker
Service Amazon
in an S3 data lake Redshift
Spectrum

• Data is queried in-place, no loading of data

Amazon
Redshift
• Keep warm data local and cold/historical data in data lake
• Write query results from Redshift direct to S3 external
tables
• Powered by a separate fleet of powerful Spectrum nodes
• Create materialized views on S3 data

© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Steps to define and create External Schema & Tables

1. Define an external schema in Amazon Redshift using the

Examples from the Redshift documentation
AWS Glue data catalog or your own Apache Hive
Metastore
CREATE EXTERNAL SCHEMA <schema_name>

2. Register external tables using Athena, your Hive

Metastore client, or from Amazon Redshift CREATE
EXTERNAL TABLE syntax
CREATE EXTERNAL TABLE <table_name>
[PARTITIONED BY <column_name, data_type, …>]
STORED AS file_format
LOCATION s3_location
[TABLE PROPERTIES property_name=property_value, …];

3. Query external tables using

<schema_name>.<table_name>

© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Data Lake query services: How to choose?
Relational
databases

Non-
Big data
relational
processing
databases

Data Lake

Log Machine
analytics learning

Data
warehousing

• Data warehouse, highly-relational, • Interactive ad-hoc queries • Process large volume of data
complex joins • Serverless • Use big data tools like Apache
• Lake house architecture • No data warehouse, not 24x7 Hadoop, Spark, Presto, Hive
• Sub-second latency • Log analysis • Run Jupyter-based EMR
• Joins between data warehouse notebooks
• Offload S3 workload from
data & an S3 data lake Datawarehouse

Amazon EMR
Amazon Redshift Amazon Athena
© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Lambda UDFs
INTEGRATE WITH EXTERNAL SERVICES USING AWS LAMBDA

SELECT my_lambda_fn(t.value)
FROM my_table t Invoke AWS Lambda programs as UDFs in Amazon Redshift
WHERE customer = “c1” SQL queries

<return results>

1
Simple integration with external services
5
• Tokenization with third-party vendors like Protegrity
• More languages runtimes (C++, Java, etc.)
• Access Amazon DynamoDB, Amazon SageMaker, etc.

Concurrent and batch processing

2 2 2 C, Java,
my_lambda_fn(value) Node.js etc.

Cost controls and error controls

3
4 <return results>

© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Native partner integration
INTEGRATE WITH SELECT PARTNERS FROM THE AMAZON REDSHIFT CONSOLE

Ingest business data and gain insights

in minutes

Deep links into partner platforms streamline

account setup

Centralized dashboard for all partner

integration statuses

Reach out to [email protected] to integrate your product into the Amazon Redshift console today!

© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Reference Architectures

© 2021, Amazon Web Services, Inc. or its Affiliates.

Reference Architecture
Data Lake first approach - Collect raw data in S3, curate and load to target

Note: Lake Formation leverages the Glue Data Catalog

© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Reference Architecture
Data warehouse first approach - Load raw data to Redshift and publish refined data on S3 Data Lake

Note: Lake Formation leverages the Glue Data Catalog

© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
 Healthcare Analytics use case
Analytics as a Service

Store & Share inpatient bed occupancy by state

North Carolina
Subscriber1

Virginia
Healthcare Analytics provider Subscriber2

© 2020, Amazon Web Services, Inc. or its Affiliates.

Reference Architecture
Subscriber Redshift clusters access provider Redshift cluster

North Carolina
Subscriber1

Healthcare Analytics provider

Virginia
Subscriber2

© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
 Amazon Redshift
Performance at any scale

Amazon
Aurora Analyze all your data
Deepest integration with your data lake

Amazon Amazon
EMR DynamoDB

Amazon
Athena

Amazon
S3
Performance at any scale
Up to 3x better price performance than other cloud DW
Amazon
Amazon
Elasticsearch
SageMaker
Service

Lower your costs

Amazon At least 50% less expensive than other cloud DW
Redshift

© 2021, Amazon Web Services, Inc. or its Affiliates.

 Get the best price performance and scale
Amazon Redshift has up to 3X better price performance than other cloud data warehouses

Customers like Duolingo, Yelp, and Codecademy are

doubling performance and saving costs with RA3

Amazon Redshift has up to 3x better

price performance than other cloud
data warehouses (see blog)

Scales data volume from 1TB to 1PB+ with predictable

cost and performance

Boost queries up to 10x with AQUA for Amazon Redshift

© 2021, Amazon Web Services, Inc. or its Affiliates.

 Redshift Scalability

Martin Brambley
Sirocco Systems, Director
Data Users
We saw an immediate 30 percent
improvement in end-to-end ETL
loading using the new DC2 node
from Redshift. This is fantastic
news for our clients as data
volumes and demand for analytics
Separation of continue to grow rapidly
Workloads
Storage & Compute

© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
RA3 nodes with managed storage
SCALE COMPUTE AND STORAGE INDEPENDENTLY

Leader node

RA3
Large high- High-bandwidth compute
Managed
storage speed cache networking nodes

Size of data warehouse only based on

steady state compute needs

Scale and pay independently for

compute and storage Amazon Redshift
managed storage
Automatic, no changes to any workflows,
no need to manage storage

© 2020, Amazon Web Services, Inc. or its Affiliates.

AQUA (Advanced Query Accelerator)

Compute
Compute Compute Compute
Compute
New distributed & hardware-accelerated
Compute Compute
Compute Compute
Compute
Clusters
Clusters
Redshift Compute
Clusters Compute
Clusters
Clusters
Redshift processing layer
Clusters Clusters
Redshift
Clusters Clusters
Clusters
Cluster Clusters Clusters
Cluster
Cluster

With AQUA, Amazon Redshift is multiple times

faster than any other cloud data warehouse, no
extra cost

AQUA AQUA AQUA AQUA

node
AQUA Nodes with custom AWS-designed analytics
node node node
processors to make operations (compression,
encryption, filtering, and aggregations) faster than
traditional CPUs

Available with RA3 and no code changes required

Amazon Redshift Managed Storage

© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Redshift automates performance tuning
ML-BASED OPTIMIZATIONS TO GET STARTED EASILY AND GET THE FASTEST PERFORMANCE QUICKLY

Auto property for tables – Dist, Sort

NEW NEW
Continuously scans workload patterns and
automates physical design optimization

Optimizes for peak performance

as data and workloads scale Automatic Automatic Automatic
vacuum delete distribution keys sort keys
Leverages machine learning to adapt
to shifting workloads
V2 NEW

Can be enabled or disabled per table

svv_alter_table_recommendations
logs the recommended changes Automatic MV auto-refresh
Auto workload
manager table sort and rewrite
svl_auto_worker_action logs audit
trail of changes
© 2021, Amazon Web Services, Inc. or its Affiliates.
Concurrency scaling
Compute elasticity and scalability to handle unpredictable user demand

Size cluster on Steady State compute

needs

Scale-out to multiple Redshift clusters

from a single endpoint in seconds

Support virtually unlimited concurrent

users while maintaining SLAs
Backup Caching Layer

Per-second billing for additional

clusters used

Free 1hr per day (free for 97% of

clusters)
© 2021, Amazon Web Services, Inc. or its Affiliates.
 Redshift cluster resizing Resizing a cluster is easily achieved with a few
clicks on the Redshift console, and there are
two resizing approaches to choose from

• Elastic Resize:
• In-place: Add or remove nodes to/from existing cluster
• Scale-Out: Performance scales proportionally
• Time: Completes within few minutes. Limited disruption to
sessions and queries
• Slice count: remains the same as original cluster

• Classic Resize:
• New cluster: new cluster is provisioned and data copied
• Time: Proportional to data volume in original cluster
• Slice count: changes based on the new cluster

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Materialized views
Compute once, query many times

• Speed-up queries by orders of magnitude

loc_sales
• Joins, filters, aggregations, and projections loc total_sales
SF 12.00
NY 10.00
• Simplify and accelerate ETL/BI pipelines
• Incremental refresh “What were
• Auto refresh the total sales
by loc?”
• Easier and faster migration to Redshift
store_info sales
• Auto-query rewrite store owner loc item store cust price
• Supports MV on Redshift local, Spectrum and Federated s1 Joe SF i1 s1 c1 12.0
s2 Ann NY i2 s2 c1 3.0
queries s3 Lisa SF i3 s2 c2 7.0

© 2021, Amazon Web Services, Inc. or its Affiliates.

 Amazon Redshift
Lower your costs

Amazon
Aurora Analyze all your data
Deepest integration with your data lake

Amazon Amazon
EMR DynamoDB

Amazon
Athena

Amazon
S3
Performance at any scale
Up to 3x better price performance than other cloud DW
Amazon
Amazon
Elasticsearch
SageMaker
Service

Lower your costs

Amazon At least 50% less expensive than other cloud DW
Redshift

© 2021, Amazon Web Services, Inc. or its Affiliates.

Built-in security and compliance
SECURITY AND COMPLIANCE FEATURES WITH NO EXTRA COSTS WITH AMAZON REDSHIFT

Helps achieve
Authentication Access control Audit Encryption compliance

IAM integration SOC

AWS KMS integration
AWS CloudTrail
integration
ID federation PCI
Column-level
privileges for Encrypted data in
Amazon Redshift motion, data at rest
Azure AD, AD, Okta, and data lake Enable Audit logging FedRAMP
Ping Federate to S3
Connection, user, Tokenization
user activity with Lambda UDFs
Multifactor
HIPAA and others
authentication and third-party tools

Additional Documentation: Security Overview

© 2021, Amazon Web Services, Inc. or its Affiliates.

Enhanced Auto WLM – Adaptive Concurrency
ML based memory prediction algorithm maximizes system resource
usage and optimizes throughput with adaptive concurrency and
feedback mechanisms
Based on the query
traffic and resource
utilization, Amazon
Redshift automatically
determines the number
of concurrent queries to
optimize query
throughput

Amazon Redshift’s Auto WLM

allows you to create user or
query group queues based on
business function, priority, and
SLA

© 2020, Amazon Web Services, Inc. or its Affiliates.

Resilience: Cross-AZ cluster recovery
RELOCATE YOUR CLUSTER IN ANOTHER AZ IN RESPONSE TO FAILURE

AZ–1 AZ–2
Amazon Redshift has a service SLA of 99.9%

Recovery with zero data loss (RP = Zero)

No need to restore from a snapshot

On-demand failover

Cluster is created in another AZ on-demand,

so cost of a standby replica cluster is avoided Redshift Managed Storage

Supported on the RA3 instance family

© 2021, Amazon Web Services, Inc. or its Affiliates.

 Amazon Redshift Data API
Simplifies data access from web services based applications

aws redshift-data execute-statement

Simplifies data access from languages --database [DATABASE]
--query [QUERY]
such as Python, Go, Java, Node.js and --secret-arn [CREDENTIALS_ARN]
other languages

Query, Load and Unload data from

CLI/SDK

Do not have to worry about

configuring drivers, connection pools

Leverages IAM credentials or Secrets

manager

Improved notebook experience

© 2020, Amazon Web Services, Inc. or its Affiliates.

Cost optimization: pause/resume cluster
Easily start and stop a cluster to save cost for intermittent workloads

Pause a development or QA cluster during non-business hours

Reserved Instances can be shared between clusters if not running at the same time
Cluster used for ingest/transform/unload to the Data Lake can be Paused after the load process.
Pause and Resume cluster operations with 1-click in the console or via API
Pause and Resume using a user-defined schedule

© 2021, Amazon Web Services, Inc. or its Affiliates.

Cost controls
Control your spend by creating usage limits for concurrency scaling and Amazon
redshift spectrum

Create daily, weekly, and monthly limits

Up to 4 limits per feature

Log to system table, Alert, Disable feature

Automatically generate Amazon CloudWatch alarm

© 2021, Amazon Web Services, Inc. or its Affiliates.

Next steps

Already using Getting started

Redshift? with Redshift?
Ask your account team for a free Ask your account team for a
Redshift optimization session 10-minute demo
Learn more about what’s new Learn more about starting a
proof of concept
https://aws.amazon.com/redshift/whats-new/ https://docs.aws.amazon.com/redshift/
proof-of-concept-playbook.html

Or request help from AWS

https://pages.awscloud.com/redshift-proof-of-concept-request
© 2021, Amazon Web Services, Inc. or its Affiliates.
 Thank You!
Gartner Magic Quadrant
2021
AWS named a
Leader in the
Cloud Database
Management
Systems Magic
Quadrant with the
highest score in
Ability to Execute
among the 16
vendors evaluated

© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved.