No Hassle NoSQL - Amazon DynamoDB & Amazon DocumentDB | AWS Summit Tel Aviv 2019
0 likes375 views
The document provides an overview of Amazon's NoSQL databases, particularly DynamoDB and DocumentDB, highlighting their features, benefits, and use cases. Key topics include managing table capacity, security features, transaction support, and planned resource usage with adaptive capacity. It also includes a customer success story from Riskified, detailing the transition from MongoDB to DynamoDB for improved performance and scalability.
![© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Document databases
• Data is stored in JSON-like
documents
• Documents map naturally to
how humans model data
• Flexible schema and indexing
• Expressive query language
built for documents (ad hoc
queries and aggregations)
JSON documents are first-class objects
of the database
{
id: 1,
name: "sue",
age: 26,
email: "sue@example.com",
promotions: ["new user", "5%", "dog lover"],
memberDate: 2018-2-22,
shoppingCart: [
{product:"abc", quantity:2, cost:19.99},
{product:"edf", quantity:3, cost: 2.99}
]
}](https://image.slidesharecdn.com/dat304-no-hassle-nosql-tel-avi-bb5516fb-4e80-46e4-ae52-ac492c1fd3b0-649674977-190321091052/85/No-Hassle-NoSQL-Amazon-DynamoDB-Amazon-DocumentDB-AWS-Summit-Tel-Aviv-2019-54-320.jpg)
No Hassle NoSQL - Amazon DynamoDB & Amazon DocumentDB | AWS Summit Tel Aviv 2019
- 2. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. No Hassle, NoSQL DynamoDB, DocumentDB Leonid Koren Solutions Architect AWS D A T 3 0 4 Ziv Shenhav Technical Account Manager AWS Inbar Yogev Data Engineering Lead Riskified Drorit Baron Software Developer Riskified
- 3. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda • NoSQL Databases • DynamoDB • Managing Table Capacity (under the hood) • Features for Developers • Customer Story • DocumentDB
- 4. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Optimized for storage Optimized for compute Normalized/relational Denormalized/hierarchical Ad hoc queries Instantiated views Scale vertically Scale horizontally Good for OLAP, Limited Scale OLTP Built for OLTP at scale SQL NoSQL Why NoSQL?
- 5. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Operating Self-Managed NoSQL Is Challenging It’s too much work It costs too much
- 6. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon DynamoDB Fast and flexible key value database service for any scale Comprehensive security Encrypts all data by default and fully integrates with AWS Identity and Access Management for robust security Performance at scale Consistent, single-digit millisecond response times at any scale; build applications with virtually unlimited throughput Global database for global users and apps Build global applications with fast access to local data by easily replicating tables across multiple AWS Regions Serverless No server provisioning, software patching, or upgrades; scales up or down automatically; continuously backs up your data
- 7. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Table, Item, Attribute, Index Table Items Attributes Partition Key Sort Key Customer ID (Partition Key) Order Date (Sort Key) Store Order Sum Country Local Secondary Index Example Table Customer ID (Partition Key) Order Sum (Sort Key) Order Date Country (Partition Key) Order Sum Store Order DateGlobal Secondary Index
- 8. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. 00 55 A954 FFAA00 FF Partition Keys Partition Key uniquely identifies an item Partition Key is used for building an unordered hash index Allows table to be partitioned for scale Id = 1 Name = Jim Hash (1) = 7B Id = 2 Name = Andy Dept = Eng Hash (2) = 48 Id = 3 Name = Kim Dept = Ops Hash (3) = CD Key Space
- 9. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Partitions are replicated across three AZs Id = 2 Name = Andy Dept = Engg Id = 3 Name = Kim Dept = Ops Id = 1 Name = Jim Id = 2 Name = Andy Dept = Engg Id = 3 Name = Kim Dept = Ops Id = 1 Name = Jim Id = 2 Name = Andy Dept = Engg Id = 3 Name = Kim Dept = Ops Id = 1 Name = Jim Replica 1 Replica 2 Replica 3
- 10. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
- 11. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Provisioning Table Capacity
- 12. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Provisioning Table Capacity 1 RCU = Items up to 4KB 1 WCU = Items up to 1KB
- 13. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Provisioning Trade-off
- 14. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Provisioning Example
- 15. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. 0x12A8 236294 { name:“Sara”, city:”Tampa”, …} 0x3391 445104 { name:“James”, city:”Miami”, …} 0x6134 333363 { name:“Betty”, city:”Madison”, …} 0x9531 145783 { name:“Bob”, city:”London”, …} 0xB082 643145 { name:“Val”, city:”Seattle”, …} 0xEA8A 723342 { name:“Jeff”, city:”Toledo”, …} 0xF355 523422 { name:“Alex”, city:”London”, …} Provisioning
- 16. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Token Bucket Algorithm
- 17. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Burst Capacity
- 18. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Unbalanced Load 0x12A8 236294 { name:“Sara”, city:”Tampa”, …} 0x3391 445104 { name:“James”, city:”Miami”, …} 0x6134 333363 { name:“Betty”, city:”Madison”, …} 0x9531 145783 { name:“Bob”, city:”London”, …} 0xB082 643145 { name:“Val”, city:”Seattle”, …} 0xEA8A 723342 { name:“Jeff”, city:”Toledo”, …} 0xF355 523422 { name:“Alex”, city:”London”, …}
- 19. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Unbalanced Load 0x12A8 236294 { name:“Sara”, city:”Tampa”, …} 0x3391 445104 { name:“James”, city:”Miami”, …} 0x6134 333363 { name:“Betty”, city:”Madison”, …} 0x9531 145783 { name:“Bob”, city:”London”, …} 0xB082 643145 { name:“Val”, city:”Seattle”, …} 0xEA8A 723342 { name:“Jeff”, city:”Toledo”, …} 0xF355 523422 { name:“Alex”, city:”London”, …} 50 throttles per second
- 20. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Adaptive Capacity
- 21. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Adaptive Capacity
- 22. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Adaptive Capacity Active 0x12A8 236294 { name:“Sara”, city:”Tampa”, …} 0x3391 445104 { name:“James”, city:”Miami”, …} 0x6134 333363 { name:“Betty”, city:”Madison”, …} 0x9531 145783 { name:“Bob”, city:”London”, …} 0xB082 643145 { name:“Val”, city:”Seattle”, …} 0xEA8A 723342 { name:“Jeff”, city:”Toledo”, …} 0xF355 523422 { name:“Alex”, city:”London”, …}
- 23. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Total Demand Change 0x12A8 236294 { name:“Sara”, city:”Tampa”, …} 0x3391 445104 { name:“James”, city:”Miami”, …} 0x6134 333363 { name:“Betty”, city:”Madison”, …} 0x9531 145783 { name:“Bob”, city:”London”, …} 0xB082 643145 { name:“Val”, city:”Seattle”, …} 0xEA8A 723342 { name:“Jeff”, city:”Toledo”, …} 0xF355 523422 { name:“Alex”, city:”London”, …} à à à
- 24. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Auto Scaling
- 25. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Actual Auto-scaled Table
- 26. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Auto Scaling – Creating A New Table
- 27. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Provisioning Recap
- 28. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. DynamoDB On-Demand Features • No capacity planning, provisioning, or reservations - simply make API calls • Pay only for the reads and writes you perform Key benefits • Eliminates tradeoffs of over- or under- provisioning • Instantly accommodates your workload as traffic ramps up or down
- 29. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. DynamoDB On-Demand
- 30. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. DynamoDB On-Demand
- 31. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. DynamoDB: Capacity managed for you Govern max consumption
- 32. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
- 33. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. DynamoDB transactions Simplify your code by executing multiple, all-or-nothing actions within and across tables with a single API call Single API Call
- 34. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Global Tables Build high-performance, globally distributed applications. Low latency reads and writes to locally available tables. Multi-region redundancy and resiliency. Easy to set up, and no application re-writes required.
- 35. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. DynamoDB Streams • DynamoDB Streams captures a time-ordered sequence of item- level modifications (stored up to 24 hours) in a DynamoDB table • Process multiple modifications at a time with Lambda Amazon DynamoDB DynamoDB Stream AWS Lambda function
- 36. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Time-To-Live (TTL) TTL job CustomerActiveOrder OrderId: 1 CustomerId: 1 Removes data that is no longer relevant MyTTL: 1492641900
- 37. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Point in time restore for short-term retention and data corruption protection Backup hundreds of TB instantly with NO performance impact On-demand backups for long- term data archival and compliance Backup and Restore
- 38. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Riskified DynamoDB talk for AWS summit Inbar Yogev Data Engineering Lead Riskified Drorit Baron Software Developer Riskified
- 39. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Meet Riskified We’re an AI platform helping merchants optimize the online shopping journey Login Checkout Post CheckoutAuthorization Fraud Review Over 300 employees 200% CAGR (2014-2018) $64 million in funding
- 40. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Riskified’s solution is effective across all online industries Marketplaces High-End Fashion Electronics GiftcardsTravelHome Sneakers Big Box Retail Automobile Jewelry/ Precious Metals
- 41. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. ClickStream Analysis Behavioral analysis Device fingerprinting Proxy detection Research Real-time analysis 120K events / second 200 reads / second
- 42. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Old Architecture Replica set Sharded cluster 15 Shards (30 nodes total) MongoDB MongoDB Research Real-time analysis
- 43. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. New Architecture Kinesis Data Streams DynamoDB Redshift Research Real-time analysis
- 44. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Data Model The Mongo Era The DynamoDB Era Nested documents (JSON) Partition keys Sort keys Indexes Lookup tables Manually purge TTL
- 45. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. OPS The Mongo Era The DynamoDB Era Complex installation Zero installation Limited metrics Usable metrics Complex DR Easy DR
- 46. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Transaction Performance 99% 95% AVG
- 47. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Scaling The Mongo Era The DynamoDB Era Weeks Minutes
- 48. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Costs Prediction 30K $ 10K $
- 49. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Join Our Team Team members across our offices in Tel Aviv and NYC, and growing! 300 Talented team members in R&D. 45
- 50. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Join Our Team Come be part of the eCommerce revolution and join us for the ride! To learn more and apply: www.riskified.com/careers Our stack includes RDS, Redshift, Elasticsearch, DynamoDB, Redis, Amazon Kinesis, Kafka, and Spark. We're looking for Big Data Engineers, Backend, and Fullstack Developers to join us.
- 51. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Thank you for your time! For additional information, visit: www.riskified.com
- 52. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
- 53. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS databases services Purpose-built for all your app needs NeptuneRDS Aurora CommercialCommunity Timestream QLDBElastiCache Relational Key-value Document In-memory Graph Time series Ledger DynamoDB DocumentDB NEW! Redis Memcached
- 54. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Document databases • Data is stored in JSON-like documents • Documents map naturally to how humans model data • Flexible schema and indexing • Expressive query language built for documents (ad hoc queries and aggregations) JSON documents are first-class objects of the database { id: 1, name: "sue", age: 26, email: "[email protected]", promotions: ["new user", "5%", "dog lover"], memberDate: 2018-2-22, shoppingCart: [ {product:"abc", quantity:2, cost:19.99}, {product:"edf", quantity:3, cost: 2.99} ] }
- 55. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon DocumentDB Fast, scalable, highly available MongoDB-compatible database Highly Available Automatic Multi-AZ data replication; automated backup, snapshots, failover; Performance at scale 2X the throughput of currently available MongoDB managed services; scale read capacity to millions of requests per second by adding up to 15 low latency read replicas across three AZs in minutes, regardless of data size MongoDB-compatible Implements the Apache 2.0 open source MongoDB 3.6 API by emulating the responses that a MongoDB client expects from a MongoDB server Durable Fault-tolerant, self-healing storage; six copies of data across three Availability Zones; continuous backup to Amazon S3
- 56. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. reads AWS Region Availability Zone 1 Availability Zone 2 Availability Zone 3 Instance (primary) Cluster volume Instance (replica) Instance (replica) Data copies Data copies Data copies writes writes writes reads reads Amazon DocumentDB Architecture Separate compute and storage
- 57. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Distributed Storage Volume Distributed Storage Volume Distributed Storage Volume Scenario 1: Dev/test with a single instance Scenario 2: Read scaling in minutes Scenario 3: Scale-up and scale-out for analytics Flexible Durability and replication are handled by the distributed storage volume
- 58. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Fully-managed Fast, reliable, and fully-managed MongoDB-compatible database service Automatic patching Quick start Monitoring Integrated Up-to-date with the latest patches Provision production- ready clusters in minutes Over 20 key operational metrics for your clusters at no extra charge Deeply integrated with AWS services such as CloudFormation, CloudTrail, CloudWatch, DMS, IAM, VPC, and more.
- 59. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Backup Automatic backups 35 days of PITR Archive snapshots Automatic, incremental, and continuous backups Backups do not affect database performance Point-in-time recovery (PITR) for up to 35 days Keep snapshots for as long as you need No performance impact
- 60. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Migrate on-premises MongoDB databases to DocumentDB Migrate EC2 MongoDB databases to DocumentDB Data replication for near-zero downtime migration Migrate from replica sets and sharded clusters Migration Migrate to DocumentDB with the Amazon Database Migration Service (DMS) Free for 6 months if you are moving to DocumentDB
- 61. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Which database to use? • Key-value • Hyper Scale • Consistent low latency at any scale • Multi-master with global replication • Item size up to 400KB DynamoDB DocumentDB • Documents • Efficiently query nested structures • Rich suite of query operators • Aggregations • Document size up to 16MB • Database size up to 64TB • Simple migration for existing MongoDB users
- 62. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
- 63. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Thank you! © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Leonid Koren Solutions Architect - AWS http://bit.ly/2SGYFNY Ziv Shenhav Technical Account Manager - AWS