Anton Moldovan "Building an efficient replication system for thousands of terminals"
0 likes172 views
For one of our projects, we needed to improve the current content delivery system for terminals. In this talk, I will share our experience in building an efficient data replication system for thousands of terminals. We will touch on architecture decisions and tradeoffs, technologies that we used, and a bit of load testing. Spoiler: We didn't use Kafka.
![Basically, every message has
a monotonically increasing number
(aka Kafka’s offset)
type UpdateBulkMessage = {
StreamerId: string
Updates: UpdateMessage[]
BulkNumber: uint
}](https://image.slidesharecdn.com/fwdays-building-an-efficient-replication-system-201103130646/85/Anton-Moldovan-Building-an-efficient-replication-system-for-thousands-of-terminals-75-320.jpg)
![RAFT
(2014)
[<Property>]
let ``handleRequest should handle heartbeat correctly`` (nodeTerm: int, requestTerm: int) =
let requestNodeId = TestInfra.createNodeId(port = 50)
let request = HeartBeat({ NodeId = requestNodeId; Term = requestTerm |> UMX.tag })
let node = Follower <| TestInfra.createFollower(port = 2, currentTerm = UMX.tag nodeTerm)
match LeaderElection.handleRequest(node, request) with
| LeaderAccepted (response, newState) ->
test <@ nodeTerm <= requestTerm @>
test <@ newState.LeaderId.Value = requestNodeId @>
test <@ (int)newState.State.CurrentTerm = requestTerm @>
test <@ response.VoteGranted = true @>
| LeaderRejected response ->
test <@ nodeTerm > requestTerm @>
test <@ (int)response.CurrentTerm = nodeTerm @>
test <@ response.VoteGranted = false @>](https://image.slidesharecdn.com/fwdays-building-an-efficient-replication-system-201103130646/85/Anton-Moldovan-Building-an-efficient-replication-system-for-thousands-of-terminals-81-320.jpg)
![RAFT
(2014)
[<Fact>]
let ``Leader may be only one among all nodes`` () =
let nodes = TestInfra.startNodes(nodeCount = 3, ipMinPort = 10_010, udpPort = 9001)
try
Task.Delay(3_000).Wait() // wait on leader election
let leader = nodes |> Seq.find(fun x -> x.IsLeader)
let leaderCount = nodes |> Seq.filter(fun x -> x.IsLeader) |> Seq.length
let sameLeaderForAllMembers =
nodes
|> Seq.filter(fun x -> not x.IsLeader)
|> Seq.map(fun x -> x.GetCurrentLeader())
|> Seq.forall(fun x -> leader.GetCurrentLeader() = x)
test <@ leaderCount = 1 @>
test <@ sameLeaderForAllMembers = true @>
finally
nodes |> Seq.iter(fun x -> x.Stop())](https://image.slidesharecdn.com/fwdays-building-an-efficient-replication-system-201103130646/85/Anton-Moldovan-Building-an-efficient-replication-system-for-thousands-of-terminals-82-320.jpg)
![[오픈소스컨설팅] Linux Network Troubleshooting](https://cdn.slidesharecdn.com/ss_thumbnails/opensourceconsultingnetworktroubleshootingyjlee20210412-210413014206-thumbnail.jpg?width=560&fit=bounds)
![[OpenInfra Days Korea 2018] Day 2 - E6 - OpenInfra monitoring with Prometheus](https://cdn.slidesharecdn.com/ss_thumbnails/e61520monitoringopeninfrawithprometheusv1-180704062709-thumbnail.jpg?width=560&fit=bounds)
![[233] level 2 network programming using packet ngin rtos](https://cdn.slidesharecdn.com/ss_thumbnails/233level2networkprogrammingusingpacketnginrtos-150915022357-lva1-app6891-thumbnail.jpg?width=560&fit=bounds)
![[245] presto 내부구조 파헤치기](https://cdn.slidesharecdn.com/ss_thumbnails/245presto-150915054242-lva1-app6891-thumbnail.jpg?width=560&fit=bounds)
![[오픈소스컨설팅] EFK Stack 소개와 설치 방법](https://cdn.slidesharecdn.com/ss_thumbnails/elasticstack-210712042246-thumbnail.jpg?width=560&fit=bounds)
![[오픈소스컨설팅] 쿠버네티스와 쿠버네티스 on 오픈스택 비교 및 구축 방법](https://cdn.slidesharecdn.com/ss_thumbnails/osck8svsk8sonopenstackkhoj-210310051504-thumbnail.jpg?width=560&fit=bounds)
Ad
More Related Content
What's hot (20)
Similar to Anton Moldovan "Building an efficient replication system for thousands of terminals" (20)
Ad
More from Fwdays (20)
![[QUICK TALK] "Why Some Teams Grow Better Under Pressure", Oleksandr Marchenko...](https://cdn.slidesharecdn.com/ss_thumbnails/softskillsfwdays25-250317080651-1a2cccb1-thumbnail.jpg?width=560&fit=bounds)
![[QUICK TALK] "How to study to acquire a skill, not a certificate?", Uliana Du...](https://cdn.slidesharecdn.com/ss_thumbnails/ulianadutchakhowtostudytoacquireaskillnotacertificate-250317080232-421f309c-thumbnail.jpg?width=560&fit=bounds)
![[QUICK TALK] "Coaching 101: How to Identify and Develop Your Leadership Quali...](https://cdn.slidesharecdn.com/ss_thumbnails/softskillsfwdays25ilonalesko-250317075721-cb56ed6c-thumbnail.jpg?width=560&fit=bounds)
Ad
Recently uploaded (20)
Anton Moldovan "Building an efficient replication system for thousands of terminals"
- 1. BUILDING AN EFFICIENT REPLICATION SYSTEM by @antyadev FOR THOUSANDS OF TERMINALS
- 2. About me Software Architect at DraftKings: - F# - Distributed Systems - Databases OSS: - Author of NBomber (https://nbomber.com) Community: - Organizer of Kyiv F# user group - Organizer of F# Ukraine conference Twitter: @antyadev Github: @antyadev
- 3. 01. Architecture overview 02. Load testing Agenda
- 4. PRELUDE
- 6. Backend The Global Problem GET data to display Terminal Bet Shop
- 7. The Global Problem GET initial data GET real-time updates Terminal Backend Bet Shop
- 8. The Global Problem Bet Shop Bet Shop Bet Shop Backend GET GET GET
- 9. The Global Problem for > 10K terminals GET Backend GET GET GET GET GET GET GET
- 11. 2008
- 12. Web Scale
- 13. Web Scale (2008) 1. Scalability 2. High availability 3. Fault Tolerance
- 14. I extend them a bit
- 15. Predictable Web Scale 1. Predictability (aka determinism) 2. Cost-effectiveness 3. Scalability (independent resource scale) 4. High availability 5. Fault Tolerance (fast recovery, redundancy)
- 16. Our hardware limitations - Slow terminal machines (1-2 CPU, x86, 600MB RAM, Windows/Linux/Android OS) - Some customers have a slow internet (5 Mbps) - Some customers have limited internet bandwidth
- 17. Data - At pick we send 2MB per min of compressed data - At pick we can get about 1-2K updates per sec
- 19. Solution 1.0
- 20. PERIODIC PULL (web scale 2008)
- 21. PERIODIC PULL (web scale 2008) for > 10K terminals GET GET GET GET GET Backend
- 22. - hard to achieve near real-time - very inefficient communication (extra traffic, extra connections) - no offline - bad UX experience (slow) - noticeable desynchronization between terminals displays, TV PERIODIC PULL
- 23. Solution 2.0
- 24. PUSH + LOCAL STORAGE (predictable web scale)
- 25. Terminal Local db GET initial data (snapshot) SUBSCRIBE on updates (change stream) PUSH + LOCAL STORAGE Backend
- 26. But we have problems with this solution - SYNC data - Network spikes
- 27. The Technical Problem SYNC data
- 30. for > 10K db replicas REPLICATION Backend SYNC data
- 31. The Technical Problem Network spikes
- 32. Network spikes Usually, all bet shops open at the same time, which means that all terminals start to pull an initial snapshot simultaneously GET initial data (snapshot)Backend
- 33. let’s imagine that 1 terminal snapshot = 100MB Local db Network spikes Usually, all bet shops open at the same time, which means that all terminals start to pull an initial snapshot simultaneously
- 34. let’s imagine that 1 terminal snapshot = 100MB 1K terminals = 1GB Local db Local db Local db Network spikes Usually, all bet shops open at the same time, which means that all terminals start to pull an initial snapshot simultaneously
- 35. let’s imagine that 1 terminal snapshot = 100MB 1K terminals = 1GB 10K terminals = 10GB for > 10K terminals Network spikes Usually, all bet shops open at the same time, which means that all terminals start to pull an initial snapshot simultaneously
- 37. Naive Solution KAFKA REPLICATION for > 10K db replicasBackend
- 38. Why not? KAFKA
- 39. - Kafka is designed for internal network communication (not external): very tricky to expose to a public network very tricky to apply any rate limit or concurrent clients limit - Kafka client is not lightweight: it eats 1 thread and quite expensive to run on a slow machine (1-2 CPU, 600MB RAM, Windows/Linux/Android OS) - We need a cache: it’s not designed as elastic cache layer to handle spikes: all messages persisted no fast point lookup to fetch missed updates Why not?
- 40. And it’s only blockers! There are other minor drawbacks: elastic/independent scaling, messaging optimizations, topic routing flexibility, P2P channel support, etc. Why not?
- 42. What then?
- 43. Let’s back to the problem
- 44. for > 10K db replicas REPLICATION SYNC data Network spikes
- 45. for > 10K db replicas REPLICATION LOG + PUSH SYNC data Network spikes
- 46. for > 10K db replicas REPLICATION Network spikes SYNC data Cache LOG + PUSH IMPORTANT: with the ability to resume the downloading process from the last read message
- 47. Okay, and remember 1. Predictability (aka determinism) 2. Cost-effective 3. Scalability (independent resource scale) 4. High availability 5. Fault Tolerance (fast recovery, redundancy)
- 56. Message Broker streamer > 10K terminals Cache An independently scalable component that represents only persistent connections (sockets) Computations are separated from sockets and scales independently An independently scalable component that represents only data
- 58. Cache
- 59. streams
- 61. producer consumer groups conceptually very similar to Kafka but: - all data in memory (very fast access, range lookup) - fixed size stream (autoclean by FIFO) - you can read data from several topics in one RTT
- 62. How do we use it?
- 63. change log (1 msg = batch of updates) - fixed size stream - autoclean by FIFO snapshot (splitted by chunks) - autoclean by TTL
- 64. change log (1 msg = batch of updates) - fixed size stream - autoclean by FIFO snapshot (splitted by chunks) - autoclean by TTL snapshot 1 snapshot 2 snapshot 3 time change log
- 66. Message Broker
- 68. EMQ X Broker - massively scalable, highly extensible distributed MQTT message broker written in Erlang/OTP (masterless, low latency, high concurrency) - open sourced - TCP/SSL, WebSockets (SSL), MQTT v3.1/5.0 - one instance (16 vCPUs, 64 GB RAM) can handle 1 million connections (we didn’t check it)
- 69. MQTT Protocol - very lightweight PUB/SUB protocol designed for IoT (binary and header requires 2 bytes) - supports hierarchical flexible topic structure (+, #) - supports standard levels of message delivery guarantee - good support on all popular platforms/runtimes
- 70. FULL picture
- 72. Now, a bit of detail...
- 73. How do we guarantee consistency?
- 74. Flow Control
- 75. Basically, every message has a monotonically increasing number (aka Kafka’s offset) type UpdateBulkMessage = { StreamerId: string Updates: UpdateMessage[] BulkNumber: uint }
- 76. What can we do regards network traffic within one betshop?
- 77. Leader Election
- 78. RAFT (2014)
- 79. RAFT (2014) Leader election (we implemented only this) Log replication Safety
- 80. RAFT (2014) F#: 230 LOC property & integration tests: ~200 LOC
- 81. RAFT (2014) [<Property>] let ``handleRequest should handle heartbeat correctly`` (nodeTerm: int, requestTerm: int) = let requestNodeId = TestInfra.createNodeId(port = 50) let request = HeartBeat({ NodeId = requestNodeId; Term = requestTerm |> UMX.tag }) let node = Follower <| TestInfra.createFollower(port = 2, currentTerm = UMX.tag nodeTerm) match LeaderElection.handleRequest(node, request) with | LeaderAccepted (response, newState) -> test <@ nodeTerm <= requestTerm @> test <@ newState.LeaderId.Value = requestNodeId @> test <@ (int)newState.State.CurrentTerm = requestTerm @> test <@ response.VoteGranted = true @> | LeaderRejected response -> test <@ nodeTerm > requestTerm @> test <@ (int)response.CurrentTerm = nodeTerm @> test <@ response.VoteGranted = false @>
- 82. RAFT (2014) [<Fact>] let ``Leader may be only one among all nodes`` () = let nodes = TestInfra.startNodes(nodeCount = 3, ipMinPort = 10_010, udpPort = 9001) try Task.Delay(3_000).Wait() // wait on leader election let leader = nodes |> Seq.find(fun x -> x.IsLeader) let leaderCount = nodes |> Seq.filter(fun x -> x.IsLeader) |> Seq.length let sameLeaderForAllMembers = nodes |> Seq.filter(fun x -> not x.IsLeader) |> Seq.map(fun x -> x.GetCurrentLeader()) |> Seq.forall(fun x -> leader.GetCurrentLeader() = x) test <@ leaderCount = 1 @> test <@ sameLeaderForAllMembers = true @> finally nodes |> Seq.iter(fun x -> x.Stop())
- 84. It’s our internal in-process in-memory database NoSQLite
- 85. - In-memory KV-store with ability to persist on disk and configure memory footprint - ACID (per single node), limited SQL, secondary indexes (in-memory) - Cluster: - zero configuration (service discovery via UDP broadcast) - auto failover - async replication NoSQLite
- 87. Why!?
- 89. do you know the throughput (RPS) and latency of your server/API?
- 91. How many instances you need to have to handle 1K concurrent users?
- 92. How you identify performance degradation?
- 93. How you prove that your architecture is bulletproofed?
- 94. What about POC?
- 95. Ok, what do we use for load testing?
- 96. NBomber - your drugs for load testing https://nbomber.com/ https://nbomber.com/docs/ https://github.com/pragmaticflow
- 97. Why we build NBomber and what you can do with it? 01. The main reason behind NBomber is to provide a lightweight framework for writing load tests which you can use to test literally any system and simulate any production workload. We wanted to provide only a few abstractions so that we could describe any type of load and still have a simple, intuitive API. 02. Another goal is to provide building blocks to validate your POC (proof of concept) projects by applying any complex load distribution. 03. With NBomber you can test any PULL or PUSH system (HTTP, WebSockets, GraphQl, gRPC, SQL Databse, MongoDb, Redis etc). 04. With NBomber you can convert some of your integration tests to load tests easily.
- 98. NBomber as a modern framework provides: - Zero dependencies on protocol (HTTP/WebSockets/AMQP/SQL) - Zero dependencies on semantic model (Pull/Push) - Very flexible configuration and dead simple API (F#/C#/JSON) - Cluster support - Realtime Reporting - Plugins support - CI/CD integration - Data feed support
- 99. Show me a code!
- 100. Hello World var step = Step.Create("step", async context => { // you can define and execute any logic here, // for example: send http request, SQL query etc // NBomber will measure how much time it takes to execute your logic await Task.Delay(TimeSpan.FromSeconds(1)); return Response.Ok(); }); // second, we add our step to the scenario var scenario = ScenarioBuilder.CreateScenario("hello_world", step); NBomberRunner .RegisterScenarios(scenario) .Run();
- 101. Hello World var step = Step.Create("step", async context => { // you can define and execute any logic here, // for example: send http request, SQL query etc // NBomber will measure how much time it takes to execute your logic await Task.Delay(TimeSpan.FromSeconds(1)); return Response.Ok(); }); // second, we add our step to the scenario var scenario = ScenarioBuilder.CreateScenario("hello_world", step); NBomberRunner .RegisterScenarios(scenario) .Run();
- 102. var httpClient = new HttpClient(); var step = Step.Create("step", async context => { var response = await httpClient.GetAsync("https://nbomber.com", context.CancellationToken); return response.IsSuccessStatusCode ? Response.Ok() : Response.Fail(); }); var scenario = ScenarioBuilder.CreateScenario("hello_world", step); NBomberRunner .RegisterScenarios(scenario) .Run(); HTTP load test (raw)
- 103. var httpClient = new HttpClient(); var step = Step.Create("step", async context => { var response = await httpClient.GetAsync("https://nbomber.com", context.CancellationToken); return response.IsSuccessStatusCode ? Response.Ok() : Response.Fail(); }); var scenario = ScenarioBuilder.CreateScenario("hello_world", step); NBomberRunner .RegisterScenarios(scenario) .Run(); HTTP load test (raw)
- 104. var step = HttpStep.Create("fetch_html_page", fun context -> Http.CreateRequest("GET", "https://nbomber.com") .WithHeader("GET", "GET") ); var scenario = ScenarioBuilder.CreateScenario("hello_world", step); NBomberRunner .RegisterScenarios(scenario) .Run(); HTTP load test with HTTP plugin (production-ready)
- 106. Redis v6.0.3 1 master and 2 slaves with async replication, without sharding, without persistence (ONLY in-memory) test: 1 writer is publishing to the stream 32KB per 500ms and many readers are reading random items 20K readers (open system) RPS 50% 75% 95% StdDev data transfer 20K 303 390 476 121 32 KB 30K readers (open system) RPS 50% 75% 95% StdDev data transfer 30K 414 520 627 158 32 KB 10K readers (open system) RPS 50% 75% 95% StdDev data transfer 10K 120 160 216 66 32 KB
- 110. EMQX v4.0.7 3 services without persistence (k8 limits: 1CPU, memory: 512MB) test: 1 writer is publishing to the topic 32KB per 500ms and many subscribers are consuming messages and verifying order of messages 10K subscribers (closed system) RPS 50% 75% 95% StdDev data transfer 20K 177 260 400 42 32 KB 5K subscribers (closed system) RPS 50% 75% 95% StdDev data transfer 10K 121 190 240 22 32 KB
- 111. What else?
- 112. NBomber plugins & extensions Http plugin - a simple DSL for working with HTTP. Ping plugin - a plugin that executes ping logic to gather networking metrics. InfluxDb sink - a data sink for InfluxDb. Prometheus sink - a data sink for Prometheus. Db Benchmark - a plugin that helps you to simulate popular database workloads (OLTP, OLAP). K8 Metrics - a plugin that allows you to fetch and render metrics from K8 and apply assertions. K8 Chaos Monkey - a plugin that helps you to inject faults into your target system and render stats.
- 113. Documentation, cluster JSON Cluster schema with agents and groups