SlideShare a Scribd company logo
Dynamo: Amazon’s Highly Available Key-Value Store
Farley Lai
University of Iowa
poyuan-lai@uiowa.edu

February 21, 2014

Farley Lai (UIOWA)

Amazon Dynamo (Big Data)

February 21, 2014

1 / 14
Motivation

MapReduce processes big data in a parallel and distributed fashion.
Daynamo forms the foundation of big data, namely, the storage.

Shopping Cart
Clients tend to insert and update items frequenty but review the cart to
check out only at the end. Is it fun for the sytem to always ask you to
retry later in minutes whenever there is an item inserted/updated in the
shopping cart?

Farley Lai (UIOWA)

Amazon Dynamo (Big Data)

February 21, 2014

2 / 14
SOA of Amazon’s Platform

Farley Lai (UIOWA)

Amazon Dynamo (Big Data)

February 21, 2014

3 / 14
Roles
Service Provider: Amazon
Service: Dynamo, the storage service
Customer: application/service vendors
Client: applications/services
User: human and/or bots

Service Level Agreements (SLA)
SLA are contracts signed by service providers and customers, specifying
the quality of service guaranteed for a client access distribution.
Example: service guaranteeing that it will provide a response within
300ms for 99.9% of its requests for a peak client load of 500 requests per
second.
Farley Lai (UIOWA)

Amazon Dynamo (Big Data)

February 21, 2014

4 / 14
What is Dynamo?

A distributed key-value storage service built on a ring topology with
high availability for writes
eventual consistency

Farley Lai (UIOWA)

Amazon Dynamo (Big Data)

February 21, 2014

5 / 14
Requirements and Assumptions

Requirements
Simple read/write to data items identified by unique keys
ACID: automicity, consistency, isolation and durability
SLA: latency constraints on the 99.9th percentile of the
distribution
Assumptions
Trusted environment and machines without security concerns

Farley Lai (UIOWA)

Amazon Dynamo (Big Data)

February 21, 2014

6 / 14
Problems, Techniques and Advantages

Problems
Partitioning
High write availability
Temporary failures

Permanent failures
Membership

Farley Lai (UIOWA)

Techniques

Advantages

Consistent Hashing
Vector clocks with
conlict resolution
Sloppy
Quorum,
hinted handoff

Incremental Scalability
Version size is decoupled
from update rates
High availability and durability guarantee despite
some unavailable replicas
Fast replica synchronization
decentralized registry for
storing membership and
liveness info

Merkle trees
Gossip protocol

Amazon Dynamo (Big Data)

February 21, 2014

7 / 14
Partitioning

Consistent hashing
1

key space

2

tokens assignment

3

replication

4

load distribution

5

node availability

6

node capacity

Farley Lai (UIOWA)

Amazon Dynamo (Big Data)

February 21, 2014

8 / 14
Data Versioning

Operations
1

read()⇒get()

2

write()⇒put()

3

conflict resolution

4

vector clock

Farley Lai (UIOWA)

Amazon Dynamo (Big Data)

February 21, 2014

9 / 14
Sloppy Quorum

1

R(2) + W (2) > N(3)

2

latency

Farley Lai (UIOWA)

Amazon Dynamo (Big Data)

February 21, 2014

10 / 14
Replica Synchronization

Figure : Merkle hash tree1

Farley Lai (UIOWA)

Figure : Merkle hash tree2

Amazon Dynamo (Big Data)

February 21, 2014

11 / 14
Evaluation: latency

Farley Lai (UIOWA)

Amazon Dynamo (Big Data)

February 21, 2014

12 / 14
Evaluation: load balance

Farley Lai (UIOWA)

Amazon Dynamo (Big Data)

February 21, 2014

13 / 14
Evaluation: write buffer

Farley Lai (UIOWA)

Amazon Dynamo (Big Data)

February 21, 2014

14 / 14
Ad

More Related Content

Viewers also liked (9)

Google - Bigtable
Google - BigtableGoogle - Bigtable
Google - Bigtable
영원 서
 
Introduction to Knockout Js
Introduction to Knockout JsIntroduction to Knockout Js
Introduction to Knockout Js
Knoldus Inc.
 
Big table
Big tableBig table
Big table
Adhinarayanan Ramanathan
 
Introduction to Apache Cassandra
Introduction to Apache Cassandra Introduction to Apache Cassandra
Introduction to Apache Cassandra
Knoldus Inc.
 
Couch db
Couch dbCouch db
Couch db
Rashmi Agale
 
CouchDB – A Database for the Web
CouchDB – A Database for the WebCouchDB – A Database for the Web
CouchDB – A Database for the Web
Karel Minarik
 
Real World CouchDB
Real World CouchDBReal World CouchDB
Real World CouchDB
John Wood
 
GOOGLE BIGTABLE
GOOGLE BIGTABLEGOOGLE BIGTABLE
GOOGLE BIGTABLE
Tomcy Thankachan
 
The Google Chubby lock service for loosely-coupled distributed systems
The Google Chubby lock service for loosely-coupled distributed systemsThe Google Chubby lock service for loosely-coupled distributed systems
The Google Chubby lock service for loosely-coupled distributed systems
Romain Jacotin
 
Google - Bigtable
Google - BigtableGoogle - Bigtable
Google - Bigtable
영원 서
 
Introduction to Knockout Js
Introduction to Knockout JsIntroduction to Knockout Js
Introduction to Knockout Js
Knoldus Inc.
 
Introduction to Apache Cassandra
Introduction to Apache Cassandra Introduction to Apache Cassandra
Introduction to Apache Cassandra
Knoldus Inc.
 
CouchDB – A Database for the Web
CouchDB – A Database for the WebCouchDB – A Database for the Web
CouchDB – A Database for the Web
Karel Minarik
 
Real World CouchDB
Real World CouchDBReal World CouchDB
Real World CouchDB
John Wood
 
The Google Chubby lock service for loosely-coupled distributed systems
The Google Chubby lock service for loosely-coupled distributed systemsThe Google Chubby lock service for loosely-coupled distributed systems
The Google Chubby lock service for loosely-coupled distributed systems
Romain Jacotin
 

Similar to Amazon Dynamo (20)

Dynamo Amazon’s Highly Available Key-value Store Giuseppe D.docx
Dynamo Amazon’s Highly Available Key-value Store Giuseppe D.docxDynamo Amazon’s Highly Available Key-value Store Giuseppe D.docx
Dynamo Amazon’s Highly Available Key-value Store Giuseppe D.docx
jacksnathalie
 
Oracle Coherence: in-memory datagrid
Oracle Coherence: in-memory datagridOracle Coherence: in-memory datagrid
Oracle Coherence: in-memory datagrid
Emiliano Pecis
 
amazon-dynamo-sosp2007
amazon-dynamo-sosp2007amazon-dynamo-sosp2007
amazon-dynamo-sosp2007
Thomas Hughes
 
Amazon dynamo-sosp2007
Amazon dynamo-sosp2007Amazon dynamo-sosp2007
Amazon dynamo-sosp2007
huangjunsk
 
Storage strategy and tsm roadmap
Storage strategy and tsm roadmapStorage strategy and tsm roadmap
Storage strategy and tsm roadmap
IBM Danmark
 
Graph Data: a New Data Management Frontier
Graph Data: a New Data Management FrontierGraph Data: a New Data Management Frontier
Graph Data: a New Data Management Frontier
Demai Ni
 
Critical Preflight Checks for Your EPM Applications
Critical Preflight Checks for Your EPM ApplicationsCritical Preflight Checks for Your EPM Applications
Critical Preflight Checks for Your EPM Applications
Datavail
 
Amazon dynamo-sosp2007
Amazon dynamo-sosp2007Amazon dynamo-sosp2007
Amazon dynamo-sosp2007
Julio Pari
 
Dynamo amazon’s highly available key value store
Dynamo amazon’s highly available key value storeDynamo amazon’s highly available key value store
Dynamo amazon’s highly available key value store
João Gabriel Lima
 
Amazon的分布式Key Value存储系统(Dynamo)的论文
Amazon的分布式Key Value存储系统(Dynamo)的论文Amazon的分布式Key Value存储系统(Dynamo)的论文
Amazon的分布式Key Value存储系统(Dynamo)的论文
yiditushe
 
Cloud Crowd GigaSpaces Presentation
Cloud Crowd GigaSpaces PresentationCloud Crowd GigaSpaces Presentation
Cloud Crowd GigaSpaces Presentation
jimliddle
 
Cloud computing aenc - final
Cloud computing   aenc - finalCloud computing   aenc - final
Cloud computing aenc - final
Association Executives of North Carolina
 
Optimized Couchbase Data Management
Optimized Couchbase Data ManagementOptimized Couchbase Data Management
Optimized Couchbase Data Management
Imanis Data
 
CloudCrowd gigaSpaces Presentation
CloudCrowd gigaSpaces PresentationCloudCrowd gigaSpaces Presentation
CloudCrowd gigaSpaces Presentation
Nati Shalom
 
Computational Patterns of the Cloud - QCon NYC 2014
Computational Patterns of the Cloud - QCon NYC 2014Computational Patterns of the Cloud - QCon NYC 2014
Computational Patterns of the Cloud - QCon NYC 2014
Ines Sombra
 
Improve your Tech Quotient
Improve your Tech QuotientImprove your Tech Quotient
Improve your Tech Quotient
Tarence DSouza
 
Cloud Computing Best Practices
Cloud Computing Best PracticesCloud Computing Best Practices
Cloud Computing Best Practices
BluePiIT
 
ScalabilityAvailability
ScalabilityAvailabilityScalabilityAvailability
ScalabilityAvailability
webuploader
 
Virtualization Licensing Lottery
Virtualization Licensing LotteryVirtualization Licensing Lottery
Virtualization Licensing Lottery
Freeform Dynamics
 
Top challenges in cloud computing
Top challenges in cloud computingTop challenges in cloud computing
Top challenges in cloud computing
TISEE
 
Dynamo Amazon’s Highly Available Key-value Store Giuseppe D.docx
Dynamo Amazon’s Highly Available Key-value Store Giuseppe D.docxDynamo Amazon’s Highly Available Key-value Store Giuseppe D.docx
Dynamo Amazon’s Highly Available Key-value Store Giuseppe D.docx
jacksnathalie
 
Oracle Coherence: in-memory datagrid
Oracle Coherence: in-memory datagridOracle Coherence: in-memory datagrid
Oracle Coherence: in-memory datagrid
Emiliano Pecis
 
amazon-dynamo-sosp2007
amazon-dynamo-sosp2007amazon-dynamo-sosp2007
amazon-dynamo-sosp2007
Thomas Hughes
 
Amazon dynamo-sosp2007
Amazon dynamo-sosp2007Amazon dynamo-sosp2007
Amazon dynamo-sosp2007
huangjunsk
 
Storage strategy and tsm roadmap
Storage strategy and tsm roadmapStorage strategy and tsm roadmap
Storage strategy and tsm roadmap
IBM Danmark
 
Graph Data: a New Data Management Frontier
Graph Data: a New Data Management FrontierGraph Data: a New Data Management Frontier
Graph Data: a New Data Management Frontier
Demai Ni
 
Critical Preflight Checks for Your EPM Applications
Critical Preflight Checks for Your EPM ApplicationsCritical Preflight Checks for Your EPM Applications
Critical Preflight Checks for Your EPM Applications
Datavail
 
Amazon dynamo-sosp2007
Amazon dynamo-sosp2007Amazon dynamo-sosp2007
Amazon dynamo-sosp2007
Julio Pari
 
Dynamo amazon’s highly available key value store
Dynamo amazon’s highly available key value storeDynamo amazon’s highly available key value store
Dynamo amazon’s highly available key value store
João Gabriel Lima
 
Amazon的分布式Key Value存储系统(Dynamo)的论文
Amazon的分布式Key Value存储系统(Dynamo)的论文Amazon的分布式Key Value存储系统(Dynamo)的论文
Amazon的分布式Key Value存储系统(Dynamo)的论文
yiditushe
 
Cloud Crowd GigaSpaces Presentation
Cloud Crowd GigaSpaces PresentationCloud Crowd GigaSpaces Presentation
Cloud Crowd GigaSpaces Presentation
jimliddle
 
Optimized Couchbase Data Management
Optimized Couchbase Data ManagementOptimized Couchbase Data Management
Optimized Couchbase Data Management
Imanis Data
 
CloudCrowd gigaSpaces Presentation
CloudCrowd gigaSpaces PresentationCloudCrowd gigaSpaces Presentation
CloudCrowd gigaSpaces Presentation
Nati Shalom
 
Computational Patterns of the Cloud - QCon NYC 2014
Computational Patterns of the Cloud - QCon NYC 2014Computational Patterns of the Cloud - QCon NYC 2014
Computational Patterns of the Cloud - QCon NYC 2014
Ines Sombra
 
Improve your Tech Quotient
Improve your Tech QuotientImprove your Tech Quotient
Improve your Tech Quotient
Tarence DSouza
 
Cloud Computing Best Practices
Cloud Computing Best PracticesCloud Computing Best Practices
Cloud Computing Best Practices
BluePiIT
 
ScalabilityAvailability
ScalabilityAvailabilityScalabilityAvailability
ScalabilityAvailability
webuploader
 
Virtualization Licensing Lottery
Virtualization Licensing LotteryVirtualization Licensing Lottery
Virtualization Licensing Lottery
Freeform Dynamics
 
Top challenges in cloud computing
Top challenges in cloud computingTop challenges in cloud computing
Top challenges in cloud computing
TISEE
 
Ad

Recently uploaded (20)

tecnologias de las primeras civilizaciones.pdf
tecnologias de las primeras civilizaciones.pdftecnologias de las primeras civilizaciones.pdf
tecnologias de las primeras civilizaciones.pdf
fjgm517
 
Quantum Computing Quick Research Guide by Arthur Morgan
Quantum Computing Quick Research Guide by Arthur MorganQuantum Computing Quick Research Guide by Arthur Morgan
Quantum Computing Quick Research Guide by Arthur Morgan
Arthur Morgan
 
UiPath Community Berlin: Orchestrator API, Swagger, and Test Manager API
UiPath Community Berlin: Orchestrator API, Swagger, and Test Manager APIUiPath Community Berlin: Orchestrator API, Swagger, and Test Manager API
UiPath Community Berlin: Orchestrator API, Swagger, and Test Manager API
UiPathCommunity
 
Noah Loul Shares 5 Steps to Implement AI Agents for Maximum Business Efficien...
Noah Loul Shares 5 Steps to Implement AI Agents for Maximum Business Efficien...Noah Loul Shares 5 Steps to Implement AI Agents for Maximum Business Efficien...
Noah Loul Shares 5 Steps to Implement AI Agents for Maximum Business Efficien...
Noah Loul
 
Technology Trends in 2025: AI and Big Data Analytics
Technology Trends in 2025: AI and Big Data AnalyticsTechnology Trends in 2025: AI and Big Data Analytics
Technology Trends in 2025: AI and Big Data Analytics
InData Labs
 
Massive Power Outage Hits Spain, Portugal, and France: Causes, Impact, and On...
Massive Power Outage Hits Spain, Portugal, and France: Causes, Impact, and On...Massive Power Outage Hits Spain, Portugal, and France: Causes, Impact, and On...
Massive Power Outage Hits Spain, Portugal, and France: Causes, Impact, and On...
Aqusag Technologies
 
Increasing Retail Store Efficiency How can Planograms Save Time and Money.pptx
Increasing Retail Store Efficiency How can Planograms Save Time and Money.pptxIncreasing Retail Store Efficiency How can Planograms Save Time and Money.pptx
Increasing Retail Store Efficiency How can Planograms Save Time and Money.pptx
Anoop Ashok
 
#StandardsGoals for 2025: Standards & certification roundup - Tech Forum 2025
#StandardsGoals for 2025: Standards & certification roundup - Tech Forum 2025#StandardsGoals for 2025: Standards & certification roundup - Tech Forum 2025
#StandardsGoals for 2025: Standards & certification roundup - Tech Forum 2025
BookNet Canada
 
The Evolution of Meme Coins A New Era for Digital Currency ppt.pdf
The Evolution of Meme Coins A New Era for Digital Currency ppt.pdfThe Evolution of Meme Coins A New Era for Digital Currency ppt.pdf
The Evolution of Meme Coins A New Era for Digital Currency ppt.pdf
Abi john
 
Manifest Pre-Seed Update | A Humanoid OEM Deeptech In France
Manifest Pre-Seed Update | A Humanoid OEM Deeptech In FranceManifest Pre-Seed Update | A Humanoid OEM Deeptech In France
Manifest Pre-Seed Update | A Humanoid OEM Deeptech In France
chb3
 
AI and Data Privacy in 2025: Global Trends
AI and Data Privacy in 2025: Global TrendsAI and Data Privacy in 2025: Global Trends
AI and Data Privacy in 2025: Global Trends
InData Labs
 
Big Data Analytics Quick Research Guide by Arthur Morgan
Big Data Analytics Quick Research Guide by Arthur MorganBig Data Analytics Quick Research Guide by Arthur Morgan
Big Data Analytics Quick Research Guide by Arthur Morgan
Arthur Morgan
 
Drupalcamp Finland – Measuring Front-end Energy Consumption
Drupalcamp Finland – Measuring Front-end Energy ConsumptionDrupalcamp Finland – Measuring Front-end Energy Consumption
Drupalcamp Finland – Measuring Front-end Energy Consumption
Exove
 
Build Your Own Copilot & Agents For Devs
Build Your Own Copilot & Agents For DevsBuild Your Own Copilot & Agents For Devs
Build Your Own Copilot & Agents For Devs
Brian McKeiver
 
Semantic Cultivators : The Critical Future Role to Enable AI
Semantic Cultivators : The Critical Future Role to Enable AISemantic Cultivators : The Critical Future Role to Enable AI
Semantic Cultivators : The Critical Future Role to Enable AI
artmondano
 
SAP Modernization: Maximizing the Value of Your SAP S/4HANA Migration.pdf
SAP Modernization: Maximizing the Value of Your SAP S/4HANA Migration.pdfSAP Modernization: Maximizing the Value of Your SAP S/4HANA Migration.pdf
SAP Modernization: Maximizing the Value of Your SAP S/4HANA Migration.pdf
Precisely
 
HCL Nomad Web – Best Practices and Managing Multiuser Environments
HCL Nomad Web – Best Practices and Managing Multiuser EnvironmentsHCL Nomad Web – Best Practices and Managing Multiuser Environments
HCL Nomad Web – Best Practices and Managing Multiuser Environments
panagenda
 
Cyber Awareness overview for 2025 month of security
Cyber Awareness overview for 2025 month of securityCyber Awareness overview for 2025 month of security
Cyber Awareness overview for 2025 month of security
riccardosl1
 
AI Changes Everything – Talk at Cardiff Metropolitan University, 29th April 2...
AI Changes Everything – Talk at Cardiff Metropolitan University, 29th April 2...AI Changes Everything – Talk at Cardiff Metropolitan University, 29th April 2...
AI Changes Everything – Talk at Cardiff Metropolitan University, 29th April 2...
Alan Dix
 
Special Meetup Edition - TDX Bengaluru Meetup #52.pptx
Special Meetup Edition - TDX Bengaluru Meetup #52.pptxSpecial Meetup Edition - TDX Bengaluru Meetup #52.pptx
Special Meetup Edition - TDX Bengaluru Meetup #52.pptx
shyamraj55
 
tecnologias de las primeras civilizaciones.pdf
tecnologias de las primeras civilizaciones.pdftecnologias de las primeras civilizaciones.pdf
tecnologias de las primeras civilizaciones.pdf
fjgm517
 
Quantum Computing Quick Research Guide by Arthur Morgan
Quantum Computing Quick Research Guide by Arthur MorganQuantum Computing Quick Research Guide by Arthur Morgan
Quantum Computing Quick Research Guide by Arthur Morgan
Arthur Morgan
 
UiPath Community Berlin: Orchestrator API, Swagger, and Test Manager API
UiPath Community Berlin: Orchestrator API, Swagger, and Test Manager APIUiPath Community Berlin: Orchestrator API, Swagger, and Test Manager API
UiPath Community Berlin: Orchestrator API, Swagger, and Test Manager API
UiPathCommunity
 
Noah Loul Shares 5 Steps to Implement AI Agents for Maximum Business Efficien...
Noah Loul Shares 5 Steps to Implement AI Agents for Maximum Business Efficien...Noah Loul Shares 5 Steps to Implement AI Agents for Maximum Business Efficien...
Noah Loul Shares 5 Steps to Implement AI Agents for Maximum Business Efficien...
Noah Loul
 
Technology Trends in 2025: AI and Big Data Analytics
Technology Trends in 2025: AI and Big Data AnalyticsTechnology Trends in 2025: AI and Big Data Analytics
Technology Trends in 2025: AI and Big Data Analytics
InData Labs
 
Massive Power Outage Hits Spain, Portugal, and France: Causes, Impact, and On...
Massive Power Outage Hits Spain, Portugal, and France: Causes, Impact, and On...Massive Power Outage Hits Spain, Portugal, and France: Causes, Impact, and On...
Massive Power Outage Hits Spain, Portugal, and France: Causes, Impact, and On...
Aqusag Technologies
 
Increasing Retail Store Efficiency How can Planograms Save Time and Money.pptx
Increasing Retail Store Efficiency How can Planograms Save Time and Money.pptxIncreasing Retail Store Efficiency How can Planograms Save Time and Money.pptx
Increasing Retail Store Efficiency How can Planograms Save Time and Money.pptx
Anoop Ashok
 
#StandardsGoals for 2025: Standards & certification roundup - Tech Forum 2025
#StandardsGoals for 2025: Standards & certification roundup - Tech Forum 2025#StandardsGoals for 2025: Standards & certification roundup - Tech Forum 2025
#StandardsGoals for 2025: Standards & certification roundup - Tech Forum 2025
BookNet Canada
 
The Evolution of Meme Coins A New Era for Digital Currency ppt.pdf
The Evolution of Meme Coins A New Era for Digital Currency ppt.pdfThe Evolution of Meme Coins A New Era for Digital Currency ppt.pdf
The Evolution of Meme Coins A New Era for Digital Currency ppt.pdf
Abi john
 
Manifest Pre-Seed Update | A Humanoid OEM Deeptech In France
Manifest Pre-Seed Update | A Humanoid OEM Deeptech In FranceManifest Pre-Seed Update | A Humanoid OEM Deeptech In France
Manifest Pre-Seed Update | A Humanoid OEM Deeptech In France
chb3
 
AI and Data Privacy in 2025: Global Trends
AI and Data Privacy in 2025: Global TrendsAI and Data Privacy in 2025: Global Trends
AI and Data Privacy in 2025: Global Trends
InData Labs
 
Big Data Analytics Quick Research Guide by Arthur Morgan
Big Data Analytics Quick Research Guide by Arthur MorganBig Data Analytics Quick Research Guide by Arthur Morgan
Big Data Analytics Quick Research Guide by Arthur Morgan
Arthur Morgan
 
Drupalcamp Finland – Measuring Front-end Energy Consumption
Drupalcamp Finland – Measuring Front-end Energy ConsumptionDrupalcamp Finland – Measuring Front-end Energy Consumption
Drupalcamp Finland – Measuring Front-end Energy Consumption
Exove
 
Build Your Own Copilot & Agents For Devs
Build Your Own Copilot & Agents For DevsBuild Your Own Copilot & Agents For Devs
Build Your Own Copilot & Agents For Devs
Brian McKeiver
 
Semantic Cultivators : The Critical Future Role to Enable AI
Semantic Cultivators : The Critical Future Role to Enable AISemantic Cultivators : The Critical Future Role to Enable AI
Semantic Cultivators : The Critical Future Role to Enable AI
artmondano
 
SAP Modernization: Maximizing the Value of Your SAP S/4HANA Migration.pdf
SAP Modernization: Maximizing the Value of Your SAP S/4HANA Migration.pdfSAP Modernization: Maximizing the Value of Your SAP S/4HANA Migration.pdf
SAP Modernization: Maximizing the Value of Your SAP S/4HANA Migration.pdf
Precisely
 
HCL Nomad Web – Best Practices and Managing Multiuser Environments
HCL Nomad Web – Best Practices and Managing Multiuser EnvironmentsHCL Nomad Web – Best Practices and Managing Multiuser Environments
HCL Nomad Web – Best Practices and Managing Multiuser Environments
panagenda
 
Cyber Awareness overview for 2025 month of security
Cyber Awareness overview for 2025 month of securityCyber Awareness overview for 2025 month of security
Cyber Awareness overview for 2025 month of security
riccardosl1
 
AI Changes Everything – Talk at Cardiff Metropolitan University, 29th April 2...
AI Changes Everything – Talk at Cardiff Metropolitan University, 29th April 2...AI Changes Everything – Talk at Cardiff Metropolitan University, 29th April 2...
AI Changes Everything – Talk at Cardiff Metropolitan University, 29th April 2...
Alan Dix
 
Special Meetup Edition - TDX Bengaluru Meetup #52.pptx
Special Meetup Edition - TDX Bengaluru Meetup #52.pptxSpecial Meetup Edition - TDX Bengaluru Meetup #52.pptx
Special Meetup Edition - TDX Bengaluru Meetup #52.pptx
shyamraj55
 
Ad

Amazon Dynamo

  • 1. Dynamo: Amazon’s Highly Available Key-Value Store Farley Lai University of Iowa [email protected] February 21, 2014 Farley Lai (UIOWA) Amazon Dynamo (Big Data) February 21, 2014 1 / 14
  • 2. Motivation MapReduce processes big data in a parallel and distributed fashion. Daynamo forms the foundation of big data, namely, the storage. Shopping Cart Clients tend to insert and update items frequenty but review the cart to check out only at the end. Is it fun for the sytem to always ask you to retry later in minutes whenever there is an item inserted/updated in the shopping cart? Farley Lai (UIOWA) Amazon Dynamo (Big Data) February 21, 2014 2 / 14
  • 3. SOA of Amazon’s Platform Farley Lai (UIOWA) Amazon Dynamo (Big Data) February 21, 2014 3 / 14
  • 4. Roles Service Provider: Amazon Service: Dynamo, the storage service Customer: application/service vendors Client: applications/services User: human and/or bots Service Level Agreements (SLA) SLA are contracts signed by service providers and customers, specifying the quality of service guaranteed for a client access distribution. Example: service guaranteeing that it will provide a response within 300ms for 99.9% of its requests for a peak client load of 500 requests per second. Farley Lai (UIOWA) Amazon Dynamo (Big Data) February 21, 2014 4 / 14
  • 5. What is Dynamo? A distributed key-value storage service built on a ring topology with high availability for writes eventual consistency Farley Lai (UIOWA) Amazon Dynamo (Big Data) February 21, 2014 5 / 14
  • 6. Requirements and Assumptions Requirements Simple read/write to data items identified by unique keys ACID: automicity, consistency, isolation and durability SLA: latency constraints on the 99.9th percentile of the distribution Assumptions Trusted environment and machines without security concerns Farley Lai (UIOWA) Amazon Dynamo (Big Data) February 21, 2014 6 / 14
  • 7. Problems, Techniques and Advantages Problems Partitioning High write availability Temporary failures Permanent failures Membership Farley Lai (UIOWA) Techniques Advantages Consistent Hashing Vector clocks with conlict resolution Sloppy Quorum, hinted handoff Incremental Scalability Version size is decoupled from update rates High availability and durability guarantee despite some unavailable replicas Fast replica synchronization decentralized registry for storing membership and liveness info Merkle trees Gossip protocol Amazon Dynamo (Big Data) February 21, 2014 7 / 14
  • 8. Partitioning Consistent hashing 1 key space 2 tokens assignment 3 replication 4 load distribution 5 node availability 6 node capacity Farley Lai (UIOWA) Amazon Dynamo (Big Data) February 21, 2014 8 / 14
  • 9. Data Versioning Operations 1 read()⇒get() 2 write()⇒put() 3 conflict resolution 4 vector clock Farley Lai (UIOWA) Amazon Dynamo (Big Data) February 21, 2014 9 / 14
  • 10. Sloppy Quorum 1 R(2) + W (2) > N(3) 2 latency Farley Lai (UIOWA) Amazon Dynamo (Big Data) February 21, 2014 10 / 14
  • 11. Replica Synchronization Figure : Merkle hash tree1 Farley Lai (UIOWA) Figure : Merkle hash tree2 Amazon Dynamo (Big Data) February 21, 2014 11 / 14
  • 12. Evaluation: latency Farley Lai (UIOWA) Amazon Dynamo (Big Data) February 21, 2014 12 / 14
  • 13. Evaluation: load balance Farley Lai (UIOWA) Amazon Dynamo (Big Data) February 21, 2014 13 / 14
  • 14. Evaluation: write buffer Farley Lai (UIOWA) Amazon Dynamo (Big Data) February 21, 2014 14 / 14