Kafka Mirror Tester: Go and Kubernetes Powered Test Suite for Kafka Replication. By Ran Tavory
0 likes337 views
Inspired by the Jepsen series of database test suites I created kafka-mirror-tester, a cross-Atlantic automated test suite for Kafka mirroring using Golang and Kubernetes. There, I said k8s, need I say more? Join me to learn how k8s solves database automation tasks and Go drives those tests.
![[20200720]cloud native develoment - Nelson Lin](https://cdn.slidesharecdn.com/ss_thumbnails/20200720cloudnativedeveloment-200824031228-thumbnail.jpg?width=560&fit=bounds)
Ad
More Related Content
What's hot (20)
Similar to Kafka Mirror Tester: Go and Kubernetes Powered Test Suite for Kafka Replication. By Ran Tavory (20)
Ad
More from Cloud Native Day Tel Aviv (20)
Ad
Recently uploaded (20)
Kafka Mirror Tester: Go and Kubernetes Powered Test Suite for Kafka Replication. By Ran Tavory
- 2. OR: Testing Kafka replication over the Atlantic using Golang, Kubernetes and friends
- 3. Hello! I am Ran Tavory I work on multi-region at appsflyer You can find me @rantav (twitter/gmail/facebook) 3
- 4. What is AppsFlyer? Data collection and analytics system 4
- 5. The Goal: Test Kafka Replication ⊙ Replicate 100MB per second ⊙ Validate correctness ⊙ Measure latency ⊙ Run test cases (failure scenarios) ○ Broker crash, cluster resize, packet loss... 5
- 6. Does this remind anyone anything? ⊙ Run test cases (failure scenarios) ○ Broker crash ○ cluster resize ○ packet loss ○ etc... 6
- 7. “ Jepsen: Call me maybe // Kyle Kingsbury 7
- 8. Kafka ? Queueing System Stream Processing Brokers Topics Partitions 8
- 9. HOW ? High level design 9
- 11. With a home-grown producer and consumer written in Golang 11
- 12. WHAT DO WE WANT? 12 DOCUMENTATION HOW DO WE DO IT? With Code
- 13. WHAT DO WE WANT? 13 REPRODUCIBILITY HOW DO WE DO IT? With Code
- 14. WHAT DO WE WANT? 14 MONITORING HOW DO WE DO IT? With Code
- 15. KUBERNETES 15
- 16. A brief introduction to Kubernetes 16 Do we need an intro to Kubernetes?
- 17. CLUSTER MANAGER Kubernetes is a 17
- 18. NODES The host running a kubelet and a set of Pods 18 It manages
- 19. PODS Pods consist of 1 or more containers 19 Nodes contain
- 20. DEPLOYMENTS A set of stateless pods 20 Pods run
- 21. STATEFUL SET A set of stateful pods 21 Pods also run
- 22. KUBECTL The k8s CLI 22 You control k8s using
- 24. What does that do? ⊙ Provision VMs in AWS ⊙ In two regions. Ireland and Virginia. ⊙ Setup VPCs, Subnets, Routing Tables ⊙ Create Security Groups ⊙ Setup Load Balancers ⊙ Install Kubernetes (etcd, masters, nodes) ⊙ Setup Monitoring (Prometheus & Grafana and install dashboard) ⊙ Install Weave Scope ⊙ Install Kafkas (and test them) ⊙ Install uReplicator (and test it) ⊙ Install test programs (written by my in Go) ⊙ And more… (ASGs, DHCP etc) 24
- 25. $ kops create cluster --zones us-east-1a,us-east-1b,us-east-1c --node-count 40 --node-size i3.large --master-size m4.large --master-zones us-east-1a --networking calico --cloud aws 25
- 27. What are we building? us-east-1 40 nodes k8s cluster 27 eu-west-1 48 nodes k8s cluster
- 28. What are we building? us-east-1 40 nodes k8s cluster 30 brokers kafka cluster 28 eu-west-1 48 nodes k8s cluster 30 brokers kafka cluster
- 29. What are we building? us-east-1 40 nodes k8s cluster 30 brokers kafka cluster 29 eu-west-1 48 nodes k8s cluster 30 brokers kafka cluster 8 workers uReplicator
- 30. What are we building? us-east-1 40 nodes k8s cluster 30 brokers kafka cluster 10 producers 30 eu-west-1 48 nodes k8s cluster 30 brokers kafka cluster 8 workers uReplicator 4 consumers
- 31. 31 End Result
- 33. 33 Weave Scope
- 34. 34 Weave Scope
- 36. TEST IT. Let’s put all this to test now 36
- 37. $ kubectl --context us-east-1.k8s.local -n kafka-source delete pod kafka-source-2 37 KILL A BROKER
- 38. 38 KILL A BROKER
- 39. $ kubectl --context us-east-1.k8s.local -n kafka-source scale statefulset kafka-source --replicas 29 39 RESIZE KAFKA CLUSTER
- 41. $ kubectl --context eu-west-1.k8s.local -n ureplicator scale deployment ureplicator-worker --replicas 9 41 ADD uREPLICATOR WORKER
- 43. $ make k8s-redeploy-tests 43 ADD NEW TOPIC
- 44. 44 ADD NEW TOPIC
- 45. $ make k8s-kafka-shell-source $ bin/kafka-topics.sh --zookeeper zookeeper:2181 --alter --topic topic5 --partitions 300 45 ADD PARTITIONS
- 46. $ kubectl --context eu-west-1.k8s.local -n ureplicator port-forward ureplicator-controller-76ff85b889-l9mzl 9000 $ curl -X DELETE http://localhost:9000/topics/topic5 $ curl -X POST -d '{"topic":"topic5", "numPartitions":"300"}' http://localhost:9000/topics 46 ADD PARTITIONS (cont)
- 47. 47 PACKET LOSS
- 48. 48 PACKET LOSS
- 49. This is so much FUN!!! But how do we actually do it? 49
- 50. CODE DEEP DIVE Golang Producer/Consumer (just a few bites) 50
- 54. “ Question: How can multiple consumers validate message arrival order? 54 Redis? DynamoDB? No! There’s a trick!
- 56. 56 Producer Sequence numbers messageKey = seq % partitions perKeySeq = seq / partitions
- 57. CODE DEEP DIVE Kubernetes, head first (just a few bites) 57
- 64. 64
- 65. Thanks! Any questions? You can find me at @rantav & [email protected] 65 This presentation: https://speakerdeck.com/rantav/infrastructure-testing-using-kubernetes- and-go The project: https://github.com/AppsFlyer/kafka-mirror-tester