UNIT II (1).pptx
Download as PPTX, PDF•0 likes•619 views
This document discusses various data distribution models and consistency approaches in NoSQL databases. It covers replication, which copies data across nodes, and sharding, which distributes different data to different nodes. Replication can be master-slave, with one master node handling writes, or peer-to-peer, allowing writes to any node. Sharding and replication can be combined. The document also discusses consistency models and handling write-write conflicts through pessimistic locking or optimistic conditional updates.
UNIT II (1).pptx
- 1. UNIT II NOSQL DATA MANAGEMENT
- 2. DISTRIBUTION MODELS • Depending on your distribution model, you can get a data store that will give you the ability to handle larger quantities of data, the ability to process a greater read or write traffic, or more availability in the face of network slowdowns or breakages. • Broadly, there are two paths to data distribution: 1. Replication and 2. sharding. • Replication takes the same data and copies it over multiple nodes. • Sharding puts different data on different nodes
- 3. Replication comes into two forms: • Master-slave Replication • Peer-to-peer Replication
- 4. SINGLE SERVER • The first and the simplest distribution option is the one we would most often recommend—no distribution at all. • Run the database on a single machine that handles all the reads and writes to the data store. • We prefer this option because it eliminates all the complexities that the other options introduce; it’s easy for operations people to manage and easy for application developers to reason about. • Although a lot of NoSQL databases are designed around the idea of running on a cluster, it can make sense to use NoSQL with a single-server distribution model if the data model of the NoSQL store is more suited to the application.
- 5. • Graph databases are the obvious category here—these work best in a single-server configuration. • If your data usage is mostly about processing aggregates, then a single-server document or key-value store may well be worthwhile because it’s easier on application developers.
- 6. SHARDING • Often, a busy data store is busy because different people are accessing different parts of the dataset. • In these circumstances we can support horizontal scalability by putting different parts of the data onto different servers—a technique that’s called sharding. • In the ideal case, we have different users all talking to different server nodes. • Each user only has to talk to one server, so gets rapid responses from that server. • The load is balanced out nicely between servers—for example, if we have ten servers, each one only has to handle 10% of the load.
- 7. • In order to get close to it we have to ensure that data that’s accessed together is clumped together on the same node and that these clumps are arranged on the nodes to provide the best data access. • The whole point of aggregates is that we design them to combine data that’s commonly accessed together—so aggregates leap out as an obvious unit of distribution. • If you know that most accesses of certain aggregates are based on a physical location, you can place the data close to where it’s being accessed. If you have orders for someone who lives in Boston, you can place that data in your eastern US data center. • Another factor is trying to keep the load even. This means that you should try to arrange aggregates so they are evenly distributed across the nodes which all get equal amounts of the load.
- 8. • Furthermore, rebalancing the sharding means changing the application code and migrating the data. • Many NoSQL databases offer auto-sharding, where the database takes on the responsibility of allocating data to shards and ensuring that data access goes to the right shard. • This can make it much easier to use sharding in an application. • Sharding is particularly valuable for performance because it can improve both read and write performance.
- 9. MASTER-SLAVE REPLICATION • With master-slave distribution, you replicate data across multiple nodes. One node is designated as the master, or primary. • This master is the authoritative source for the data and is usually responsible for processing any updates to that data. • The other nodes are slaves, or secondaries. A replication process synchronizes the slaves with the master
- 11. • Master-slave replication is most helpful for scaling when you have a read-intensive dataset. • You can scale horizontally to handle more read requests by adding more slave nodes and ensuring that all read requests are routed to the slaves. • Consequently it isn’t such a good scheme for datasets with heavy write traffic, although offloading the read traffic will help a bit with handling the write load. • A second advantage of master-slave replication is read resilience: Should the master fail, the slaves can still handle read requests. Again, this is useful if most of your data access is reads. • The failure of the master does eliminate the ability to handle writes until either the master is restored or a new master is appointed. • However, having slaves as replicates of the master does speed up recovery after a failure of the master since a slave can be appointed a new master very quickly.
- 12. • Masters can be appointed manually or automatically. • Manual appointing typically means that when you configure your cluster, you configure one node as the master. • With automatic appointment, you create a cluster of nodes and they elect one of themselves to be the master. • Apart from simpler configuration, automatic appointment means that the cluster can automatically appoint a new master when a master fails, reducing downtime.
- 13. PEER-TO-PEER REPLICATION • Master-slave replication helps with read scalability but doesn’t help with scalability of writes. • It provides resilience against failure of a slave, but not of a master. Essentially, the master is still a bottleneck and a single point of failure. • Peer-to-peer replication attacks these problems by not having a master. • All the replicas have equal weight, they can all accept writes, and the loss of any of them doesn’t prevent access to the data store.
- 15. • With a peer-to-peer replication cluster, you can ride over node failures without losing access to data. • Furthermore, you can easily add nodes to improve your performance. • The biggest complication is, again, consistency. When you can write to two different places, you run the risk that two people will attempt to update the same record at the same time—a write-write conflict.
- 16. COMBINING SHARDING AND REPLICATION • Replication and sharding are strategies that can be combined. If we use both master-slave replication and sharding his means that we have multiple masters, but each data item only has a single master. • Depending on your configuration, you may choose a node to be a master for some data and slaves for others, or you may dedicate nodes for master or slave duties. • Using peer-to-peer replication and sharding is a common strategy for column-family databases. • In a scenario like this you might have tens or hundreds of nodes in a cluster with data sharded over them.
- 17. • A good starting point for peer-to-peer replication is to have a replication factor of 3, so each shard is present on three nodes. • Should a node fail, then the shards on that node will be built on the other nodes (see Figure ).
- 19. KEY POINTS There are two styles of distributing data: • Sharding distributes different data across multiple servers, so each server acts as the single source for a subset of data. • Replication copies data across multiple servers, so each bit of data can be found in multiple places. A system may use either or both techniques. • Replication comes in two forms: • Master-slave replication makes one node the authoritative copy that handles writes while slaves synchronize with the master and may handle reads. • Peer-to-peer replication allows writes to any node; the nodes coordinate to synchronize their copies of the data. • Master-slave replication reduces the chance of update conflicts but peer-to-peer replication avoids loading all writes onto a single point of failure.
- 20. CONSISTENCY • One of the biggest changes from a centralized relational database to a cluster-oriented NoSQL database is in how you think about consistency. Update Consistency • We’ll begin by considering updating a telephone number. Coincidentally, Martin and Pramod are looking at the company website and notice that the phone number is out of date. • Implausibly, they both have update access, so they both go in at the same time to update the number. • To make the example interesting, we’ll assume they update it slightly differently, because each uses a slightly different format. • This issue is called a write-write conflict: two people updating the same data item at the same time.
- 21. • When the writes reach the server, the server will serialize them—decide to apply one, then the other. • Let’s assume it uses alphabetical order and picks Martin’s update first, then Pramod’s. Without any concurrency control, Martin’s update would be applied and immediately overwritten by Pramod’s. • In this case Martin’s is a lost update. Here the lost update is not a big problem, but often it is. • We see this as a failure of consistency because Pramod’s update was based on the state before Martin’s update, yet was applied after it.
- 22. • Approaches for maintaining consistency in the face of concurrency are often described as pessimistic or optimistic. • A pessimistic approach works by preventing conflicts from occurring; • An optimistic approach lets conflicts occur, but detects them and takes action to sort them out. • For update conflicts, the most common pessimistic approach is to have write locks, so that in order to change a value you need to acquire a lock, and the system ensures that only one client can get a lock at a time. • So Martin and Pramod would both attempt to acquire the write lock, but only Martin (the first one) would succeed. • Pramod would then see the result of Martin’s write before deciding whether to make his own update.
- 23. • A common optimistic approach is a conditional update where any client that does an update tests the value just before updating it to see if it’s changed since his last read. • In this case, Martin’s update would succeed but Pramod’s would fail. The error would let Pramod know that he should look at the value again and decide whether to attempt a further update. • There is another optimistic way to handle a write-write conflict—save both updates and record that they are in conflict
- 24. READ CONSISTENCY • Let’s imagine we have an order with line items and a shipping charge. The shipping charge is calculated based on the line items in the order. • If we add a line item, we thus also need to recalculate and update the shipping charge. • In a relational database, the shipping charge and line items will be in separate tables. • The danger of inconsistency is that Martin adds a line item to his order, Pramod then reads the line items and shipping charge, and then Martin updates the shipping charge. • This is an inconsistent read or read-write conflict: Pramod has done a read in the middle of Martin’s write. • We refer to this type of consistency as logical consistency: ensuring that different data items make sense together.
- 26. • To avoid a logically inconsistent read-write conflict, relational databases support the notion of transactions. • Providing Martin wraps his two writes in a transaction, the system guarantees that Pramod will either read both data items before the update or both after the update. • In NOSQL databases tend to support ACID transactions just the same as relational databases. • Secondly, aggregate-oriented databases do support atomic updates, but only within a single aggregate. • This means that you will have logical consistency within an aggregate but not between aggregates. • So in the example, you could avoid running into that inconsistency if the order, the delivery charge, and the line items are all part of a single order aggregate.
- 27. • The length of time an inconsistency is present is called the inconsistency window. • A NoSQL system may have a quite short inconsistency window: As one data point, Amazon’s documentation says that the inconsistency window for its SimpleDB service is usually less than a second.
- 28. Replication Consistency • Let’s imagine there’s one last hotel room for a desirable event. The hotel reservation system runs on many nodes. Martin and Cindy are a couple considering this room, but they are discussing this on the phone because Martin is in London and Cindy is in Boston. • Meanwhile Pramod, who is in Mumbai, goes and books that last room. That updates the replicated room availability, but the update gets to Boston quicker than it gets to London. • When Martin and Cindy fire up their browsers to see if the room is available, Cindy sees it booked and Martin sees it free. • This is another inconsistent read—but it’s a breach of a different form of consistency we call replication consistency: ensuring that the same data item has the same value when read from different replicas
- 30. • You may post a message using one node, then refresh your browser, but the refresh goes to a different node which hasn’t received your post yet—and it looks like your post was lost. • In situations like this, you can tolerate reasonably long inconsistency windows, but you need read your- writes consistency which means that, once you’ve made an update, you’re guaranteed to continue seeing that update. • One way to get this in an otherwise eventually consistent system is to provide session consistency: Within a user’s session there is read- your-writes consistency. • This does mean that the user may lose that consistency should their session end for some reason or should the user access the same system simultaneously from different computers, but these cases are relatively rare.
- 31. • There are a couple of techniques to provide session consistency. • A common way, and often the easiest way, is to have a sticky session: a session that’s tied to one node (this is also called session affinity). A sticky session allows you to ensure that as long as you keep read-your-writes consistency on a node, you’ll get it for sessions too. The downside is that sticky sessions reduce the ability of the load balancer to do its job. • Another approach for session consistency is to use version stamps (“Version Stamps,” p. 61) and ensure every interaction with the data store includes the latest version stamp seen by a session. • The server node must then ensure that it has the updates that include that version stamp before responding to a request.
- 32. Relaxing Consistency • Consistency is a Good Thing—but, sadly, sometimes we have to sacrifice it. It is always possible to design a system to avoid inconsistencies, but often impossible to do so without making unbearable • sacrifices in other characteristics of the system. As a result, we often have to tradeoff consistency for something else. • While some architects see this as a disaster, we see it as part of the inevitable tradeoffs involved in system design. • Furthermore, different domains have different tolerances for inconsistency, and we need to take this tolerance into account as we make our decisions.
- 33. RELAXING DURABILITY • Example of relaxing durability is capturing telemetric data from physical devices. • It may be that you’d rather capture data at a faster rate, at the cost of missing the last updates should the server go down. • A simple case of this may happen if you have a master-slave distribution model where the slaves appoin t a new master automatically should the existing master fail. • If a master does fail, any writes not passed onto the replicas will effectively become lost. Should the master come back online, those updates will conflict with updates that have happened since. • We think of this as a durability problem because you think your update has succeeded since the master acknowledged it, but a master node failure caused it to be lost.
- 34. QUORUMS • Imagine some data replicated over three nodes. You don’t need all nodes to acknowledge a write to ensure strong consistency; all you need is two of them—a majority. • If you have conflicting writes, only one can get a majority. This is referred to as a write quorum and expressed in a slightly pretentious inequality of W > N/2, meaning the number of nodes participating in the write (W) must be more than the half the number of nodes involved in replication (N). • The number of replicas is often called the replication factor.
- 35. • Let’s consider a replication factor of 3. If all writes need two nodes to confirm (W = 2) then we need to contact at least two nodes to be sure we’ll get the latest data. • If, however, writes are only confirmed by a single node (W = 1) we need to talk to all three nodes to be sure we have the latest updates.
- 36. Key Points • Write-write conflicts occur when two clients try to write the same data at the same time. • Read write conflicts occur when one client reads inconsistent data in the middle of another client’s write. • Pessimistic approaches lock data records to prevent conflicts. Optimistic approaches detect conflicts and fix them. • Distributed systems see read-write conflicts due to some nodes having received updates while other nodes have not. Eventual consistency means that at some point the system will become consistent once all the writes have propagated to all the nodes. • Clients usually want read-your-writes consistency, which means a client can write and then immediately read the new value. This can be difficult if the read and the write happen on different nodes. • To get good consistency, you need to involve many nodes in data operations, but this increases latency. So you often have to trade off consistency versus latency.
- 37. • The CAP theorem states that if you get a network partition, you have to trade off availability of data versus consistency. • Durability can also be traded off against latency, particularly if you want to survive failures with replicated data. • You do not need to contact all replicants to preserve strong consistency with replication; you just need a large enough quorum.