Data & Analytics ReInvent Recap [AWS Basel Meetup - Jan 2023]

© All rights reserved 2022 2
Basel AWS User Group
re:Invent re:Cap
Data & Analytics

© All rights reserved 2022
Cloudreach Service Lines
Cloudreach
Services
Advisory
Software Platform
Data & Analytics
Platform Development
Application Modernisation
Security Transformation
Migration
Management & Resale

Our Cloud Partnerships
UK&I Consulting Partner
of the Year 2019
EMEA GSI Partner of the Year
2022
3x Golden Jackets!
900+ Certiﬁcations
15 Competency
Specializations
Data Analytics - Services Specialisation
GCP Security Partner
of the Year 2021
12 Competency
Specializations
Gold Data Analytics Competency
Gold Data Platform Competency
Microsoft UK Partner
of the Year 2020
12 Gold Certiﬁcations
15 Competency
Specializations

Marketing
Achieved!

The Big Data & Analytics
Announcements

Zero-ETL Integration - Aurora with RedShift
Eliminate the need to build and maintain complex data pipelines to perform
extract, transform, and load (ETL) operations
Auto-Copy from S3
Load files from S3 as it arrives
Dynamic data masking
Protect sensitive information stored in your data warehouse and ensure that
only the relevant data is accessible by users based on their roles
Centralised access control with AWS Lake Formation
Simplify governance of data shared from Amazon Redshift and centrally manage
granular access across all data-sharing consumers
Integration with Apache Spark
Build and run Spark applications on Amazon Redshift and Redshift Serverless,
opening up the data warehouse for a broader set of AWS analytics and machine
learning solutions
Streaming Ingestion
Natively ingest hundreds of megabytes of data per second from Amazon Kinesis
Data Streams and Amazon MSK
Redshift
Glue 4.0
New and Updated Engines, More Data Formats, and More
This version of Glue includes Python 3.10 and Apache Spark 3.3.0, plus native
support for the Cloud Shuffle Service Plugin for Spark. It also includes Pandas
support and more.
Glue Data Quality
Analyse your tables and recommend a set of rules automatically based on what it
finds
Glue for Ray
Process large datasets with Python and popular Python libraries, serverless
option for data integration with Ray, a popular new open-source compute
framework that helps you scale Python workloads
Glue

Built-in Data Preparation, Real-Time Collaboration, and
Notebook Automation
Improve data quality in minutes with the built-in data preparation capability, edit
the same notebooks with your teams in real time, and automatically convert
notebook code to production-ready jobs.
ML Governance Tools for Amazon SageMaker
Simplify Access Control and Enhance Transparency Over Your ML Projects
Build, Train, and Deploy ML Models Using Geospatial Data
This collection of features offers pre-trained deep neural network (DNN) models
and geospatial operators that make it easy to access and prepare large
geospatial datasets.
Support for Shadow Testing
Compare Inference Performance Between ML Model Variants
Deploying a model in shadow mode lets you conduct a more holistic test by
routing a copy of the live inference requests for a production model to the new
(shadow) model.
Real-Time and Batch Inference support in Data Wrangler
Reuse the data transformation ﬂow which you created in SageMaker Data
Wrangler as a step in Amazon SageMaker inference pipelines.
SageMaker
Automated Data Preparation
Automated data preparation utilises machine learning to infer semantic
information about data and adds it to datasets as metadata about the columns
(ﬁelds), making it faster for you to prepare data in order to support natural
language questions.
Paginated Reports
Create and Share Operational Reports at Scale.
This feature allows customers to create and share highly formatted,
personalized reports containing business-critical data to hundreds of thousands
of end-users — without any infrastructure setup or maintenance, up-front
licensing, or long-term commitments.
New API Capabilities to Accelerate Your BI Transformation
New QuickSight API capabilities allow programmatic creation and management
of dashboards, analysis, and templates.
QuickSight

Amazon Security Lake
A Purpose-Built Customer-Owned Data Lake Service. This new service
automatically centralises your organisation’s security data from cloud and
on-premises sources into a purpose-built data lake stored in your account.
Step Functions Distributed Map
A Serverless Solution for Large-Scale Parallel Data Processing. The new
distributed map state can launch up to ten thousand parallel workﬂows to
process data.
Amazon Athena for Apache Spark
Using Spark without having to manually provision the infrastructure. Run
Apache Spark workloads, use Jupyter Notebook as the interface to perform data
processing on Athena, and programmatically interact with Spark applications
using Athena APIs.
AWS Database Migration Service: Fully Managed Schema
Conversion
Streamlines database migrations by making schema assessment and conversion
available inside AWS DMS. You can now plan, assess, convert and migrate under
one central DMS service.
Other New Stuff
Amazon DataZone
Unlock data across organisational boundaries with built-in governance. Discover
and share data at scale across organisational boundaries with governance and
access controls.
Amazon Omics
Store, query, and analyze genomic, transcriptomic, and other omics data and
then generate insights from that data to improve health and advance scientiﬁc
discoveries.
AWS Clean Rooms
Securely match, analyse, and collaborate on combined datasets across
organisations without revealing the underlying data.

So What is DataZone?

Provides the technology infrastructure for pursuing
a data mesh architecture within an organisation
Key components:
Business Data Catalog
Make data assets discoverable and searchable with business
metadata, manageable by Data Stewards via metadata forms
Data Projects
Groups users and data assets within a business context where they
can collaborate and manage their output
Governance & Access Control
Includes self-service data access requests, data producer review
processes, and access control to manage where data assets are
used
Data Portal
A dedicated web portal (outside of the AWS console) for working
with DataZones
Domains
Groupings of the above which enable DataZone to map it’s
governance and access control structure to your organisational
structure
Basically; Data Mesh Tech Layer-as-a-Service Coming Soon

So What is Omics?

Basically; Genomic Research Infra-as-a-Service
Provides specialised storage and processing workﬂows for
omics data
Built for Petabyte scale storage and processing
Designed to be used in GDPR and HIPAA regulated
environments
Automatically manages underlying infrastructure for
analytics workﬂows on omics data
Enables utilisation of omics data via Athena, EMR,
QuickSight, and more
GA

So What is Clean Rooms?

Provides the plumbing for multiple
organisations to collaborate on their data,
without actually copying/transferring it
Can share any data in your Glue Data
Catalog / S3 Data Lake
Key terms:
Analysis Rules
Basically; Zero Transfer X-Org Analytics Preview
Define what types of SQL queries other collaborators can use with a data asset you associate with a clean room
E.g. limiting which SQL aggregations can be used
Output Restrictions
Define limits on the results other collaborators can receive from a data asset you associate with a clean room
E.g. setting a minimum result value for a COUNT DISTINCT query
Can restrict collaborators access to only rows that match an inner join on their own associated data assets
Can run queries on data whilst it remains encrypted - i.e. encryption in-use!
Optionally logs all queries performed against your data to CloudWatch Logs in your AWS account
Important T&C contractual limit:
You may not use AWS Clean Rooms or any information obtained from your use of AWS Clean Rooms to identify a person or associate such information with an identifiable
person, unless otherwise permitted by the applicable third party contributor of the data.

My Favourite Session:
AWS IoT ExpressLink Workshop

What is IoT ExpressLink?
Physical hardware for integration into IoT devices, built by AWS partners
Provides AWS connectivity without the need to write your own connectivity
code
Implements AWS security best practices for IoT connectivity
Effectively:
● Ofﬂoad the undifferentiated heavy lifting of connectivity and
cryptography in IoT devices
● Simplify and accelerate AWS-integrated IoT device and service
development
● Rapid mass onboarding of IoT devices, e.g. via bluetooth
Conﬁgured for WiFi or 5G connections using serial AT commands e.g. (in
Python):
send_command(uart, f"AT+CONF SSID={wifi_ssid}")
send_command(uart, f"AT+CONF Passphrase={wifi_key}")

We’ll use a freebie from the re:Invent
workshop to look at:
● IoT ExpressLink in use
● AWS IoT Core device shadows
● Wrapping device shadows in a
serverless API with AWS Lambda
● Building an AWS Amplify app on top
of that API
The demo badge contains:
● IoT ExpressLink chip (red circle)
● Raspberry Pi Pico-esque board
● A bunch of widgets!
Hardware details and image source here
Demo Time!

IoT Core Device Shadow

Basic Serverless Device Shadow API

Device Shadow Web App

Questions?
Perhaps Answers

Thank you
Cloudreach Tech Blog: https://www.cloudreach.com/en/technical-blog
These Slides: https://bit.ly/3QFl7Xr
Demo Repos: https://bit.ly/3GLt3BH
https://bit.ly/3IRfXWg
Slides

Data & Analytics ReInvent Recap [AWS Basel Meetup - Jan 2023]

More Related Content

What's hot (20)

Similar to Data & Analytics ReInvent Recap [AWS Basel Meetup - Jan 2023] (20)

More from Chris Bingham (7)

Recently uploaded (20)

Data & Analytics ReInvent Recap [AWS Basel Meetup - Jan 2023]