A T L A N T A | M A Y 1 8 – 1 9 , 2 0 2 2

ISV201

Event streaming with modern data

pipelines in a SaaS architecture

Matthew Duren Ray Dickenson

Director of Reliability Engineering ISV Solutions Architect
KnowBe4 AWS

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Agenda

About us

The problem

AWS Well-Architected

The old versus the new

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
About KnowBe4

• Founded 2010 in Tampa, FL

• Industry-leading security awareness training and
phishing simulation service
• Fortune’s Best Places to Work in Technology
• Ranked a best place to work for women,
millennials, and in technology nationwide by
Fortune Magazine three years in a row
• Ranked #1 or #2 in the Tampa Bay Top
Workplaces Survey for the last six years

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The problem

• Our internal warehouse needs data

• Relational databases have lots of stakeholders
• Our customers need better reporting

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Unicorn – as envisioned

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Unicorn – as implemented

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Well-Architected Framework

• Stop guessing our capacity needs • Drive our architecture

using data
• Test systems at production scale
• Improve with the help of
Game Days
• Automate to make architectural
experimentation easier

• Allow for evolutionary

architectures

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Well-Architected Framework

• Stop guessing our capacity needs • Drive our architecture

using data

• Allow for evolutionary

architectures

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The old versus the new

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Evolution of the architecture

The old The new

• Nightly, long-running Amazon • No waiting around
ECS-based job (AWS Lambda)
• Clone the entire database • Work with new data only (CDC)
• Anonymize the entire dataset • Anonymize the data once
• Export to Amazon S3 (Parquet) • Product to Amazon S3
• Slow • Fast!

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Colonial – the pipeline

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Database activity streams

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Database activity streams

• Free with Amazon Aurora

• No full load
• Not proper CDC
• No “middleman” features
▪ Select schemas/tables
▪ Stop/pause
▪ Data store targeting

https://aws.amazon.com/blogs/database/filter-amazon-aurora-database-activity-stream-data-for-segregation-and-monitoring

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Write-ahead logs (WAL)

Amazon
Kinesis Data
Streams

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Write-ahead logs (WAL)

• Postgres only
• Delivery to
Amazon CloudWatch

https://aws.amazon.com/blogs/database/stream-changes-from-amazon-rds-for-postgresql-using-amazon-kinesis-data-streams-and-aws-lambda

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Database Migration Service (AWS DMS)
Data producers

Amazon RDS Amazon Redshift

MySQL instance

Amazon DynamoDB
RDS Oracle
instance

Amazon QuickSight

AWS DMS Amazon Kinesis Amazon EMR Amazon Kinesis

RDS Microsoft SQL Data Streams Data Streams
Kinesis target Streaming layer
Server instance
Amazon CloudSearch

Amazon DynamoDB
Amazon Kinesis Amazon Kinesis
Data Firehose Data Firehose Amazon Athena

RDS PostgreSQL
instance
Kinesis-enabled app
Persistent layer

Amazon S3 bucket Amazon S3 bucket

DB2 LUW Amazon S3

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Database activity streams

• Full load capabilities

• Control over the export task
(stop/start/pause)
• True CDC solution
• Many options for
schema selection
• Native anonymization
(bonus points!)

https://aws.amazon.com/blogs/database/use-the-aws-database-migration-service-to-stream-change-data-to-amazon-kinesis-data-streams

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS DMS

Amazon S3 bucket Amazon SNS

Amazon S3 bucket Amazon SNS

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How it’s going so far

• A few minor issues with DDL pausing the DMS (e.g., adding columns)
• Limited anonymization capabilities
▪ Will lead to AWS Glue some day
• “If you build it, they will come” has been successful
▪ User sync
▪ New reporting capabilities
▪ Internal search tool
• Applied to 3 Amazon RDS databases and growing
• Overall, very successful!

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Learn in-demand AWS Cloud skills

AWS Skill Builder AWS Certifications

Access 500+ free digital courses Earn an industry-recognized
and Learning Plans credential

Explore resources with a variety Receive Foundational,

of skill levels and 16+ languages Associate, Professional,
to meet your learning needs and Specialty certifications

Deepen your skills with digital Join the AWS Certified community
learning on demand and get exclusive benefits

Access new
Train now exam guides

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Thank you!

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.