All Databases Are Equal, But Some Databases Are More Equal than Others: How to Choose the Right Database for your Data Needs

S U M M I TS U M M I T
Z uri ch

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
All databases are equal, but some databases are more equal than others:
How to choose the right database for your data
needs
Javier Ramirez
@supercoco9
Technical Evangelist
Amazon Web Services
D A T 0 1
Markus Winterholer
Chief Product Owner
Siemens Smart Infrastructure

“The giant, monolithic "bookstore" application and giant database that we used to
power Amazon.com limited our speed and agility. Whenever we wanted to add a new
feature or product for our customers, like video streaming, we had to edit and rewrite vast
amounts of code on an application that we'd designed specifically for our first product—
the bookstore. This was a long, unwieldy process requiring complicated coordination, and it
limited our ability to innovate fast and at scale.
(...) Modern applications are built with decoupled data stores in which there is a one-to-one
mapping of database and microservice, rather than a single database.
(...) By decoupling data along with microservices, you free yourself to choose the database
that best fits your need.
Werner Vogels, CTO - Amazon.com
https://www.allthingsdistributed.com/2019/08/modern-applications-at-aws.html

Traditional Application Requirements
Users: 10–100k
Data volume: GB–TB
Locality: HQ
Performance: Seconds
Request Rate: Tens of thousands
Access: Internal servers, PCs
Scale: Up
Economics: Pay up front
Developer Access: Days/weeks/months
HR Payroll …
CRM ERP

1970 1980 1990 2000 2010
Oracle DB2
SQL Server
MySQL
PostgreSQL
DynamoDB
Redis
MongoDB
Elasticsearch
Neptune
Cassandra Amazon Redshift
Aurora

Modern Application Requirements
Users: 1M+
Data volume: TB–PB–EB
Locality: Global
Performance: Milliseconds–microseconds
Request Rate: Millions
Access: Mobile, IoT, devices
Scale: Up-out-in
Economics: Pay as you go
Developer Access: Instant API access
Relational Key-value Document
In-memory Graph

Common data categories and use cases
Relational
Referential
integrity, ACID
transactions,
schema-
on-write
Lift and shift, ERP,
CRM, finance
Key-value
High
throughput, low-
latency reads
and writes,
endless scale
Real-time bidding,
shopping cart,
social, product
catalog, customer
preferences
Document
Store
documents and
quickly access
querying on
any attribute
Content
management,
personalization,
mobile
In-memory
Query by key
with
microsecond
latency
Leaderboards,
real-time analytics,
caching
Graph
Quickly and
easily create
and navigate
relationships
Fraud detection,
social networking,
recommendation
engine
Time-series
Collect, store,
and process
data sequenced
by time
IoT applications,
event tracking
Ledger
Complete,
immutable, and
verifiable history
of all changes to
application data
Systems
of record, supply
chain, health care,
registrations,
financial
Complex query
support via SQL
Simple queries and
filters
Simple queries with
filters, aggregates
and projections
Simple queries
and filters
Easily express
queries in terms of
relations
Specialized queries
for interpolations,
smoothing, and
approximations
SQL-like with
support for nested
data and history
queries
Search
Indexing and
searching
semistructured
data
Search engines,
operational and
monitoring
insights
Full-text search
with filters and
aggregates

Common data categories and use cases
Relational Key-value Document In-memory Graph Time-series Ledger Search
Amazon Aurora
Amazon RDS
Amazon
DynamoDB
Amazon
DocumentDB
Amazon
ElastiCache
for Redis &
Memcached
Amazon
Neptune
Amazon
Timestream
Amazon
Quantum
Ledger
Database
Amazon
ElasticSearch

400,000+ Customers using AWS Databases
and Analytics Services

USER
PERCEPTION
ENGINEERING REALITY
Minimalist UI
Global Scale
Data Science
Machine Learning
Bi-Directional Ranking

196countries
52languages
1Bswipes daily
20Bmatches
17Bevent counts daily
GLOBAL SCALE
Amazon DynamoDB
ElastiCache

Purpose-Built Databases Example: Airbnb
Airbnb uses different databases based
on the purpose
User search history: DynamoDB
• Massive data volume
• Need quick lookups for personalized search
Session state: ElastiCache
• In-memory store for sub-millisecond site rendering
Transactional data: RDS
• Referential integrity
• Primary transactional database

Amazon RDS
Managed relational database service with a choice of popular database engines
Easy to
administer
No need to provision
infrastructure, install, and
maintain DB software
Highly scalable
Scale DB compute and
storage with a few clicks;
minimal downtime for
your application
Available &
durable
Automatic Multi-AZ
data replication;
automated backup,
snapshots, and failover
Fast & secure
SSD storage and
guaranteed provisioned
I/O; data encryption at
rest and in transit

Amazon Aurora
MySQL and PostgreSQL compatible relational database built for the cloud
Performance and availability of commercial-grade databases at 1/10th the
cost
Performance
& scalability
5x throughput of standard
MySQL and 3x of standard
PostgreSQL; scale-out up to
15 read replicas
Availability
& durability
Fault-tolerant, self-healing
storage; six copies of data
across three AZs; continuous
backup to S3
Highly secure
Network isolation,
encryption at
rest/transit
Fully managed
Managed by RDS: no
hardware provisioning,
software patching, setup,
configuration, or backups

Local
Storage
SQL
Transactions
Caching
Logging
Compute
Traditional Database Architecture
Monolithic stack in a Single box
Large blast radius

Aurora: Scale-out, Distributed architecture
Push Log applicator to Storage
Master Replica Replica Replica
Master
Shared storage volume
Replica Replica
SQL
Transactions
Caching
SQL
Transactions
Caching
SQL
Transactions
Caching
AZ1 AZ2 AZ3
ü Write performance
ü Read scale out
ü AZ + 1 failure tolerance
• 4/6 Write Quorum & Local tracking

Amazon Aurora—High Performance
Scale out to millions of reads per second
Availability
Zone 1
Scale-out read performance
Availability
Zone 2
Availability
Zone 3
Application
Read Replica
1
Read Replica
2
Master Node
Shared distributed storage volume
Up to 15 read replicas across three AZs
Auto-scale new read replicas
Seamless recovery from read replica failures

1B+
rows
4K+
writes/second
500K+
rows/second
<1ms
insert
latency
<50ms
select
latency

Continuous backup
• Take periodic snapshot of each segment in parallel; stream the redo logs to Amazon S3
• Backup happens continuously without performance or availability impact
Segment snapshot Log records
Recovery point
Segment 1
Segment 2
Segment 3
Time

Database backtrack
Backtrack brings the database to a point in time without requiring restore from backups
• Backtracking from an unintentional DML or DDL operation
• Backtrack is not destructive. You can backtrack multiple times to find the right point in time
t0 t1 t2
t0 t1
t2
t3 t4
t3
t4
Rewind to t1
Rewind to t3
Invisible Invisible

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Global physical replication
• Primary region • Secondary region
1
ASYNC 4/6 QUORUM
Continuous
backup
AZ 1
Primary
Instance
Amazon
S3
AZ 2
Replica
Instance
AZ 3
Replica
Instance
Replication
Server
Replication Fleet
Storage Fleet
11
4
AZ 1
Replica
Instance
AZ 2 AZ 3
ASYNC 4/6 QUORUM
Continuous
backup
Amazon
S3
Replica
Instance
Replica
Instance
Replication
Agent
Replication Fleet
Storage Fleet
3
3
2
① Primary instance sends log records in parallel to storage nodes, replica
instances and replication server
② Replication server streams log records to Replication Agent in secondary
region
③ Replication agent sends log records in parallel to storage nodes, and replica
instances
④ Replication server pulls log records from storage nodes to catch up after
outages
High throughput: Up to 150K writes/sec – negligible
performance impact
Low replica lag: < 1 sec cross-region replica lag under heavy
load
Fast recovery: < 1 min to accept full read-write workloads after
region failure

Aurora Multi-Master
First relational database to scale out reads and writes across multiple data centers
Availability
Zone 1
Scale-out both reads and writes
Availability
Zone 2
Availability
Zone 3
Application
Read/Write
Master 3
Read/Write
Master 1
Shared distributed storage volume
Zero application downtime from ANY
instance failure
Zero application downtime from ANY
AZ failure
Faster write performance and higher scale
Multi-region multi-master coming in the
future

Aurora Serverless
On-demand, auto-scaling database for applications with variable workloads
Warm
Capacity Pool
Application
Database Endpoint
Scalable Database Capacity
(Compute + Memory)
Shared Distributed Storage
Starts up on demand, shuts down
when not in use
Automatically scales with no instances
to manage
Pay per second for the database
capacity you use

Aurora: Fastest Growing Service in AWS History

Amazon DynamoDB
We needed to adapt to power Amazon.com
Needed to power
Amazon.com
Required massive
scalability and reliability
DynamoDB designed
to meet this need

Amazon DynamoDB
Fully-managed nonrelational database for any scale
High performance
Fast, consistent performance
Virtually unlimited throughput
Virtually unlimited storage
Secure
Encryption at rest and transit
Fine-grained access control
PCI, HIPAA, FIPS140-2 eligible
Fully managed
Maintenance-free
Serverless
Auto scaling
Backup and restore
Global tables
Reliability
Data replicated across
multiple Azs and regionally
available APIs

DynamoDB Schema
Table
Items
Attributes
Partition
key
Sort
key
Mandatory
Key-value access pattern
Determines data distribution
Optional
Model 1:N relationships
Enables rich query capabilities
All items for key
==, <, >, >=, <=
“begins with”
“between”
“contains”
“in”
sorted results
counts
top/bottom N values

DynamoDB Global Tables
Build high-performance,
globally distributed
applications
Low latency reads
and writes to locally
available tables
Multi-region redundancy
and resiliency
Easy to set up and
no application
rewrites required

Amazon DynamoDB
200,000+ customers

https://aws.amazon.com/blogs/aws/amazon-prime-day-2019-powered-by-aws/
Amazon Prime Day powered by AWS

Graph Use Cases
Social news feed Recommendations Retail fraud detection
Friends
Use
Play
Like
Check in
Like
Connect
Read
Credit
card
Product
Email
address
Credit
card
Known
fraud
Uses
Paid
with
Uses
Paid
with
Paid with
Purchased
Approve
purchase?
Sport
Product
Purchased
Purchased
People
who also
follow sports
purchased…
Purchased
Knows
Knows
Do you
know…
Follows
Follows
Follows

Challenges Building Apps with
Highly-Connected Data
Existing graph databasesRelational databases
Too expensiveDifficult to
maintain high
availability
Difficult
to scale
Limited
support for
open standards
Inefficient
graph
processing
Unnatural for
querying graph
Rigid schema
inflexible for
changing
graphs

Amazon Neptune
Fully managed graph database
Fast & scalable
Store billions of
relationships; query with
millisecond latency
Reliable
Six replicas of your data
across three AZs with full
backup and restore
Easy
Build powerful queries
easily with Gremlin
and SPARQL
Open
Supports Apache
TinkerPop and W3C RDF
graph models
Gremlin
SPARQL

Amazon Neptune high-level architecture
Bulk load from S3
Database Mgmt.

https://github.com/aws-samples/amazon-neptune-
samples/tree/master/gremlin/collaborative-filtering

Time-series data
What is time-series
data?
A sequence of data points
recorded over a time interval

Time-series data
What is time-series
data?

Time-series data
What is special about a
time-series database?
Time is the
single primary axis
of the data model
t

Existing time-series databasesRelational databases
Difficult to
maintain high
availability
Difficult to scale Limited data
lifecycle
management
Inefficient
time-series data
processing
Unnatural for
time-series
data
Rigid schema
inflexible for fast
moving time-series
data
Building with time-series data is challenging

Amazon Timestream (sign up for the preview)
Fast, scalable, fully managed time-series database
1,000x faster and 1/10th the
cost of relational databases
Collect data at the rate of
millions of inserts per
second (10M/second)
Trillions of
daily events
Adaptive query processing
engine maintains steady,
predictable performance
Time-series analytics
Built-in functions for
interpolation, smoothing,
and approximation
Serverless
Automated setup,
configuration, server
provisioning, software patching

Common customer use cases
Ledgers with centralized control
Healthcare
Verify and track hospital
equipment inventory
Manufacturers
Track distribution of a
recalled product
HR & Payroll
Track changes to an
individual’s profile
Government
Track vehicle title
history

Challenges with building ledgers
Adds unnecessary
complexity
BlockchainRDBMS - audit tables
Difficult to
maintain
Hard to use
and slow
Hard to build
Custom audit functionality using
triggers or stored procedures
Impossible to verify
No way to verify changes made
to data by sys admins

Amazon Quantum Ledger Database (QLDB)
Fully managed ledger database
Track and verify history of all changes made to your application’s data
Immutable
Maintains a sequenced record of
all changes to your data, which
cannot be deleted or modified;
you have the ability to query and
analyze the full history
Cryptographically
verifiable
Uses cryptography to
generate a secure output
file of your data’s history
Easy to use
Easy to use, letting you
use familiar database
capabilities like SQL APIs
for querying the data
Highly scalable
Executes 2–3X as many
transactions than ledgers
in common blockchain
frameworks

How it works
J
history.cars
H
INSERT cars
ID:1
Manufacturer: Tesla
Model: Model S
Year: 2012
VIN: 123456789
Owner: Traci Russell
Metadata: {
Date:07/16/2012
}
current.cars
C
H (T1) UPDATE cars
ID:1
Owner: Ronnie Nash
Metadata: {
Date:08/03/2013
}
H (T2)
ID Manufacturer Model Year VIN Owner
1 Tesla Model S 2012 123456789 Elmer Hubbard
FROM cars WHERE VIN = '123456789' UPDATE owner = 'Elmer Hubbard'
UPDATE cars
ID:1
Owner: Elmer Hubbard
Metadata: {
Date: 09/02/2016
}
H (T3)
ID Version Start Manufacturer Model Year VIN Owner
1 1 07/16/2012 Tesla Model S 2012 123456789 Traci Russell
1 2 08/03/2013 Tesla Model S 2012 123456789 Ronnie Nash
1 3 09/02/2016 Tesla Model S 2012 123456789 Elmer Hubbard

Three types of database migrations
Database Migration
Options
Database Freedom
(i.e. re-platforming)
Lift and Shift
(run on EC2)
Managed Service
(with same legacy databases)
• Database runs exactly as on-
premises
• Quickest way to onboard
workloads
• Vendor lock-in
• Still running Oracle, SQL
Server, etc.
• Auto provisioning, backups,
recovery, patching
• Reduce undifferentiated heavy
lifting
• RDS Commercial Engines (Oracle,
SQL Server)
• AWS Cloud native managed database
services
• Reduced licensing costs
• Migrate to Amazon Aurora, Amazon
RDS open-source databases, Amazon
Redshift, Amazon DynamoDB, etc.
• Purpose built

AWS Database Migration Service
Migrating
Databases
to AWS
90,000+
Databases migrated
Migrate between on-premises and AWS
Migrate between databases
Data replication for zero-downtime migration
Automated schema conversion

90,000+ Databases Migrated with DMS

Customers gaining value from database migrations
Verizon is migrating over 1,000 business-critical applications and database backend systems to AWS, several of
which also include the migration of production databases to Amazon Aurora.
By migrating from Microsoft SQL Server to Amazon Aurora, Ryanair can run one of the largest email
campaigns in Europe with higher performance at a fraction of the cost, sending out 22 million
emails daily to customers.
Trimble migrated their Oracle databases to Amazon RDS and project they will pay about 1/4th
of what they paid when managing their private infrastructure.
Intuit migrated from Microsoft SQL Server to Amazon Redshift to reduce data-processing timelines
and get insights to decision makers faster and more frequently.
By December 2018, Amazon.com had migrated 88% of their Oracle DBs (and 97% of critical system DBs) to
Amazon Aurora and Amazon DynamoDB. They also migrated their 50 PB Oracle Data Warehouse to AWS
(Amazon S3, Amazon Redshift, and Amazon EMR).
Samsung Electronics migrated their Cassandra clusters to Amazon DynamoDB for their Samsung
Cloud workload with 70% cost savings.
Equinox Fitness migrated its Teradata on-premises data warehouse to Amazon Redshift. They went
from static reports to a modern data lake that delivers dynamic reports.
Migrated their Market Data system from SQL Server to Aurora MySQL using AWS Database Migration
Service (AWS DMS) to replicate data nightly. Reduces their processing times from 8 hours to 3 hours.

https://github.com/aws-samples/aws-bookstore-demo-app

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I TS U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

All Databases Are Equal, But Some Databases Are More Equal than Others: How to Choose the Right Database for your Data Needs

Recommended

More Related Content

More from javier ramirez (20)

Recently uploaded (20)

All Databases Are Equal, But Some Databases Are More Equal than Others: How to Choose the Right Database for your Data Needs