AWS SSA Webinar 20 - Getting Started with Data Warehouses on AWS
1 like158 views
In this session, we will take you through setting up an Amazon Redshift cluster and at the ways you can populate it with data. We will start by using AWS DMS to replicate the data as-is as well as doing some ETL on it. This will be followed by AWS Glue where you can do more advanced ETL operations. Lastly, we will look at how you can use Amazon Kinesis Firehose to stream event directly to the Redshift cluster.
Ad
More Related Content
Similar to AWS SSA Webinar 20 - Getting Started with Data Warehouses on AWS (12)
More from Cobus Bernard (20)
Ad
Recently uploaded (19)
Ad
AWS SSA Webinar 20 - Getting Started with Data Warehouses on AWS
- 1. Cobus Bernard Sr Developer Advocate Amazon Web Services Getting Started with Data Warehouses on AWS @cobusbernard cobusbernard cobusbernard
- 2. Agenda Amazon Redshift Columnar Data Compression Query execution with Redshift Spectrum Demo Q&A
- 3. Load Unload Backup Restore Amazon Redshift architecture Massively parallel, shared nothing columnar architecture Leader node SQL endpoint Stores metadata Coordinates parallel SQL processing Compute nodes Local, columnar storage Executes queries in parallel Load, unload, backup, restore Amazon Redshift Spectrum nodes Execute queries directly against Amazon S3 SQL clients/BI tools JDBC/ODBC Compute node Compute node Compute node Leader node Amazon S3 ... 1 2 3 4 N Amazon Redshift Spectrum Load Query
- 4. Terminology and concepts: Node types Amazon Redshift analytics – RA3 Amazon Redshift managed storage – Solid state disks + Amazon S3 Dense compute – DC2 Solid state disks Dense storage – DS2 Magnetic disks Instance type Disk type Size Memory CPUs Slices RA3 4xlarge RMS Scales to 16 TB 96 GB 12 4 RA3 16xlarge RMS Scales to 64 TB 384 GB 48 16 DC2 large SSD 160 GB 16 GB 2 2 DC2 8xlarge SSD 2.56 TB 244 GB 32 16 DS2 xlarge Magnetic 2 TB 32 GB 4 2 DS2 8xlarge Magnetic 16 TB 244 GB 36 16
- 5. Terminology and concepts: Columnar Amazon Redshift uses a columnar architecture for storing data on disk Goal: reduce I/O for analytics queries Physically store data on disk by column rather than row Only read the column data that is required
- 6. Row-based storage • Need to read everything • Unnecessary I/O aid loc dt CREATE TABLE deep_dive ( aid INT --audience_id ,loc CHAR(3) --location ,dt DATE --date ); aid loc dt 1 SFO 2017-10-20 2 JFK 2017-10-20 3 SFO 2017-04-01 4 JFK 2017-05-14 SELECT min(dt) FROM deep_dive; Columnar architecture: Example
- 7. Column-based storage • Only scan blocks for relevant column aid loc dt CREATE TABLE deep_dive ( aid INT --audience_id ,loc CHAR(3) --location ,dt DATE --date ); aid loc dt 1 SFO 2017-10-20 2 JFK 2017-10-20 3 SFO 2017-04-01 4 JFK 2017-05-14 SELECT min(dt) FROM deep_dive; Columnar architecture: Example
- 8. Compression example Add 1 of 13 different encodings to each column aid loc dt CREATE TABLE deep_dive ( aid INT --audience_id ,loc CHAR(3) --location ,dt DATE --date ); aid loc dt 1 SFO 2017-10-20 2 JFK 2017-10-20 3 SFO 2017-04-01 4 JFK 2017-05-14
- 9. Compression example • More efficient compression is due to storing the same data type in the columnar architecture • Columns grow and shrink independently • Reduces storage requirements • Reduces I/O aid loc dt CREATE TABLE deep_dive ( aid INT ENCODE AZ64 ,loc CHAR(3) ENCODE BYTEDICT ,dt DATE ENCODE RUNLENGTH ); aid loc dt 1 SFO 2017-10-20 2 JFK 2017-10-20 3 SFO 2017-04-01 4 JFK 2017-05-14
- 10. CREATE TABLE deep_dive ( aid INT --audience_id ,loc CHAR(3) --location ,dt DATE --date ) SORTKEY (dt, loc); Example: Sort key Add a sort key to one or more columns to physically sort the data on disk deep_dive aid loc dt 1 SFO 2017-10-20 2 JFK 2017-10-20 3 SFO 2017-04-01 4 JFK 2017-05-14 deep_dive (sorted) aid loc dt 3 SFO 2017-04-01 4 JFK 2017-05-14 2 JFK 2017-10-20 1 SFO 2017-10-20 deep_dive (sorted) aid loc dt 3 SFO 2017-04-01 4 JFK 2017-05-14 2 JFK 2017-10-20 deep_dive (sorted) aid loc dt 3 SFO 2017-04-01 4 JFK 2017-05-14 deep_dive (sorted) aid loc dt 3 SFO 2017-04-01
- 11. SELECT count(*) FROM deep_dive WHERE dt = '06-09-2017'; MIN: 01-JUNE-2017 MAX: 06-JUNE-2017 MIN: 07-JUNE-2017 MAX: 12-JUNE-2017 MIN: 13-JUNE-2017 MAX: 21-JUNE-2017 MIN: 21-JUNE-2017 MAX: 30-JUNE-2017 Sorted by date MIN: 01-JUNE-2017 MAX: 20-JUNE-2017 MIN: 08-JUNE-2017 MAX: 30-JUNE-2017 MIN: 12-JUNE-2017 MAX: 20-JUNE-2017 MIN: 02-JUNE-2017 MAX: 25-JUNE-2017 Unsorted table Example: Zone maps and sorting
- 12. Data distribution Distribution style is a table property that dictates how that table’s data is distributed throughout the cluster KEY: Value is hashed, same value goes to same location (slice) ALL: Full table data goes to the first slice of every node EVEN: Round robin AUTO: Combines EVEN and ALL Goals Distribute data evenly for parallel processing Minimize data movement during query processing KEY Node 1 Slice 1 Slice 2 Node 2 Slice 3 Slice 4 keyA keyB keyC keyD ALL Node 1 Slice 1 Slice 2 Node 2 Slice 3 Slice 4 EVEN Node 1 Slice 1 Slice 2 Node 2 Slice 3 Slice 4
- 13. Node 1 Data distribution example Slice 0 Slice 1 Node 2 Slice 2 Slice 3 Table: deep_dive User columns System columns aid loc dt ins del row CREATE TABLE deep_dive ( aid INT --audience_id ,loc CHAR(3) --location ,dt DATE --date ) (EVEN|KEY|ALL|AUTO);
- 14. Node 1 Slice 0 Slice 1 Node 2 Slice 2 Slice 3 Data distribution, EVEN example INSERT INTO deep_dive VALUES (1, 'SFO', '2016-09-01'), (2, 'JFK', '2016-09-14'), (3, 'SFO', '2017-04-01'), (4, 'JFK', '2017-05-14'); Table: deep_dive User Columns System Columns aid loc dt ins del row Table: deep_dive User Columns System Columns aid loc dt ins del row Table: deep_dive User Columns System Columns aid loc dt ins del row Table: deep_dive User Columns System Columns aid loc dt ins del row Rows: 0 Rows: 0 Rows: 0 Rows: 0Rows: 1 Rows: 1 Rows: 1 Rows: 1 CREATE TABLE deep_dive ( aid INT --audience_id ,loc CHAR(3) --location ,dt DATE --date ) DISTSTYLE EVEN;
- 15. Node 1 Slice 0 Slice 1 Node 2 Slice 2 Slice 3 Data distribution, KEY example #1 INSERT INTO deep_dive VALUES (1, 'SFO', '2016-09-01'), (2, 'JFK', '2016-09-14'), (3, 'SFO', '2017-04-01'), (4, 'JFK', '2017-05-14'); Table: deep_dive User Columns System Columns aid loc dt ins del row Rows: 2 Rows: 0 Rows: 0Rows: 0Rows: 1 Table: deep_dive User Columns System Columns aid loc dt ins del row Rows: 2Rows: 0Rows: 1 CREATE TABLE deep_dive ( aid INT --audience_id ,loc CHAR(3) --location ,dt DATE --date ) DISTSTYLE KEY DISTKEY (loc);
- 16. Node 1 Slice 0 Slice 1 Node 2 Slice 2 Slice 3 Data distribution, KEY example #2 INSERT INTO deep_dive VALUES (1, 'SFO', '2016-09-01'), (2, 'JFK', '2016-09-14'), (3, 'SFO', '2017-04-01'), (4, 'JFK', '2017-05-14'); Table: deep_dive User Columns System Columns aid loc dt ins del row Table: deep_dive User Columns System Columns aid loc dt ins del row Table: deep_dive User Columns System Columns aid loc dt ins del row Table: deep_dive User Columns System Columns aid loc dt ins del row Rows: 0 Rows: 0 Rows: 0 Rows: 0Rows: 1 Rows: 1 Rows: 1 Rows: 1 CREATE TABLE deep_dive ( aid INT --audience_id ,loc CHAR(3) --location ,dt DATE --date ) DISTSTYLE KEY DISTKEY (aid);
- 17. Node 1 Slice 0 Slice 1 Node 2 Slice 2 Slice 3 Data distribution, ALL example INSERT INTO deep_dive VALUES (1, 'SFO', '2016-09-01'), (2, 'JFK', '2016-09-14'), (3, 'SFO', '2017-04-01'), (4, 'JFK', '2017-05-14'); Rows: 0 Rows: 0 Table: deep_dive User Columns System Columns aid loc dt ins del row Rows: 0Rows: 1Rows: 2Rows: 4Rows: 3 Table: deep_dive User Columns System Columns aid loc dt ins del row Rows: 0Rows: 1Rows: 2Rows: 4Rows: 3 CREATE TABLE deep_dive ( aid INT --audience_id ,loc CHAR(3) --location ,dt DATE --date ) DISTSTYLE ALL;
- 18. ... 1 2 3 4 N Amazon Redshift Spectrum Load Query Amazon Redshift architecture Massively parallel, shared nothing columnar architecture Leader node SQL endpoint Stores metadata Coordinates parallel SQL processing Compute nodes Local, columnar storage Executes queries in parallel Load, unload, backup, restore Amazon Redshift Spectrum nodes Execute queries directly against Amazon S3 SQL clients/BI tools JDBC/ODBC Compute node Compute node Compute node Leader node Amazon S3
- 19. Query SELECT COUNT(*) FROM S3.EXT_TABLE GROUP BY… Life Of A Query Amazon Redshift JDBC/ODBC ... 1 2 3 4 N Amazon S3 Exabyte-scale object storage Data Catalog Apache Hive Compatible Metastore 1
- 20. Life Of A Query Amazon Redshift JDBC/ODBC ... 1 2 3 4 N Amazon S3 Exabyte-scale object storage Data Catalog Apache Hive Compatible Metastore Query is optimised and compiled at the leader node. Determine what gets run locally and what goes to Amazon Redshift Spectrum 2
- 21. Life Of A Query Amazon Redshift JDBC/ODBC ... 1 2 3 4 N Amazon S3 Exabyte-scale object storage Data Catalog Apache Hive Compatible Metastore Query plan is sent to all compute nodes3
- 22. Life Of A Query Amazon Redshift JDBC/ODBC ... 1 2 3 4 N Amazon S3 Exabyte-scale object storage Data Catalog Apache Hive Compatible Metastore Compute nodes obtain partition info from Data Catalog; dynamically prune partitions4
- 23. Life Of A Query Amazon Redshift JDBC/ODBC ... 1 2 3 4 N Amazon S3 Exabyte-scale object storage Data Catalog Apache Hive Compatible Metastore Each compute node issues multiple requests to the Amazon Redshift Spectrum layer 5
- 24. Life Of A Query Amazon Redshift JDBC/ODBC ... 1 2 3 4 N Amazon S3 Exabyte-scale object storage Data Catalog Apache Hive Compatible Metastore Amazon Redshift Spectrum nodes scan your S3 data 6
- 25. Life Of A Query Amazon Redshift JDBC/ODBC ... 1 2 3 4 N Amazon S3 Exabyte-scale object storage Data Catalog Apache Hive Compatible Metastore 7 Amazon Redshift Spectrum projects, filters, and aggregates
- 26. Life Of A Query Amazon Redshift JDBC/ODBC ... 1 2 3 4 N Amazon S3 Exabyte-scale object storage Data Catalog Apache Hive Compatible Metastore Final aggregations and joins with local Amazon Redshift tables done in-cluster 8
- 27. Life Of A Query Amazon Redshift JDBC/ODBC ... 1 2 3 4 N Amazon S3 Exabyte-scale object storage Data Catalog Apache Hive Compatible Metastore Result is sent back to client9
- 28. Data warehouse (business data) Data lake (event data) Amazon Redshift Amazon Redshift enables you to have a lake house approach Customers moving to data lake architectures
- 29. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
- 30. Sources Targets Relational NoSQL Analytics Data Warehouse* Amazon Aurora Amazon Aurora SAP ASE Amazon DynamoDB Amazon DocumentDB (with MongoDB compatibility) Db2 LUW SQL Azure SAP ASE MongoDB Cassandra Amazon Redshift Amazon S3 AWS SnowballAmazon S3 Amazon Elasticsearch Service Amazon Kinesis Data Streams Oracle SQL Server Netezza Greenplum Teradata Vertica * Supported via SCT data extractors Supported DMS source and targets
- 32. Thank you! © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Cobus Bernard Sr Developer Advocate Amazon Web Services @cobusbernard cobusbernard cobusbernard