Ad
More Related Content
Similar to Scaling Open Source Big Data Cloud Applications is Easy/Hard (20)
More from Paul Brebner (20)
Ad
Recently uploaded (20)
Ad
Scaling Open Source Big Data Cloud Applications is Easy/Hard
- 1. Scaling Open Source Big Data Cloud Applications is Easy/Hard Paul Brebner Instaclustr—Technology Evangelist ©Instaclustr Pty Limited, 2022 HotCloudPerf - 9 April 2022
- 2. Who am I? • 1999-2007 CSIRO/UCL • Enterprise Java evaluation, SPECJAppServer200X benchmarks • OGSA (Grid) evaluation (UCL) • 2007-2017 NICTA/Startup CTO • R&D and consulting - performance modelling large scale distributed systems, for government and large enterprises • 2017-present Instaclustr (soon NetApp) • Technology Evangelist for Instaclustr • 100+ Blogs, Talks
- 3. Performance modelling from APM data (2007-2017, NICTA/Startup) APM data (Dynatrace’s PurePath®) Performance Model Simulation Tool + Model Visualisation and Graphs Transformation Execution Automated pipeline, worked with Dynatrace PurePath® data Distributed traces with breakdown data (resource types and time) Revisit with OpenTelemetry?
- 4. Cloud Platform for Big Data Open Source Technologies New! And Workflows Instaclustr Managed Platform ©Instaclustr Pty Limited, 2021
- 5. Scaling is Easy! Cassandra and Kafka Homogeneous distributed clusters à horizontally scalable www.cassandra.apache.org/_/cassandra-basics.html
- 6. But actually lots of moving parts (source: http://trumpetb.net/loco/rodsf.html)
- 7. Complications – DCs, Racks, Nodes, Partitions, Replication Factor, Time (for auto-scaling) Rows have a partition key and are stored in different partitions
- 8. Example 1 – Cassandra Auto-Scaling ©Instaclustr Pty Limited, 2021
- 9. Two Ways of Resizing Clusters 1 - Horizontal Scaling • Add nodes, no interruption • But scale up only (not down) • Takes time, puts extra load on cluster as data streams to extra nodes 2 - Vertical Scaling • Replace nodes with bigger (or smaller) node types (more/less cores) • Scale up and down • Takes time, temporary reduction in capacity • Choice of how many nodes are replaced concurrently – by “node” (1 node at a time) or by “rack” (all nodes in a rack) , or in-between
- 10. Cluster resizing time – by node vs. by rack – by rack is faster but …? Cluster = 6 nodes, 3 racks, 2 nodes per rack By node (concurrency 1) By rack (concurrency 2)
- 11. Resizing by node – capacity reduced by 1/6 total nodes each resize operation (simplified model)
- 12. Resizing by rack – capacity reduced by 2/6 nodes each resize operation
- 13. Comparison – resize by rack faster but has bigger capacity hit during resize
- 14. Observations • In both cases • The eventual capacity is double the original • The cluster capacity is reduced during resizing • By rack is faster, but has worst capacity reduction during resizing • By node is slower, but has less capacity reduction during resizing • If the capacity during resize is exceeded latencies will increase • Made worse by Cassandra load balancing which assumes equal sized nodes • By node, more nodes in the Cluster reduces the impact of reduced cluster capacity during resizing (some clusters have 100s of nodes) • But majority of our clusters have <= 6 nodes
- 15. Auto-scaling model - increasing load à linear regression over 1 hour extrapolated to future We predict the cluster will reach 100% capacity around the 280 minute mark (220 minutes in the future) Extrapolated Measured
- 16. Resize by Rack vs. Node - initiated in time to prevent overloading during resize operation Resize by rack must be initiated sooner c.f. resize by node, even thought it’s faster to resize, as it has less capacity during resize (67% c.f. 83% of initial capacity) By Rack By Node
- 17. Auto-scaling POC – worked! Monitoring API Linear Regression + Rules Provisioning API Rules generalized to allow for • scaling up and down • resizing by any number of nodes concurrently, up to rack size
- 18. Example 2– Kafka Topics “Kongo” Logistics IoT Application ©Instaclustr Pty Limited, 2021 ©Instaclustr Pty Limited 2019, 2021, 2022
- 19. Kafka is a topic based pub-sub messaging system: - Producers send messages to topics. - Consumers subscribe to topics of interest, e.g. parties. - When they poll they only receive messages sent to those topics. Producer Consumer Consumer Consumer Consumer Topic “Parties” Topic “Work” Consumers subscribed to Topic “Parties” Consumers poll to receive messages from “Parties” Consumers not subscribed to “Work” messages ©Instaclustr Pty Limited 2019, 2021, 2022
- 20. Design choices: Many vs. One Topic? 1 - Many (100s) of topics 100s of locations (Warehouses, Trucks) Each location has a topic and multiple consumers
- 21. 2 - One topic One topic for all locations
- 22. Single topic wins! But why? Many topics, 7200 Single topic, 1120000 0 200000 400000 600000 800000 1000000 1200000 Many topics Single topic TPS
- 23. Example 3 – Anomaly Detection ©Instaclustr Pty Limited, 2021 JoAnn Morgan Apollo 11 Mission Control
- 24. Multiple technologies: Kafka, Cassandra, Kubernetes
- 25. Massively Scalable Anomaly Detection – Tuning knobs (Orange h/w, yellow s/w) Scaling is (too) Easy! Initially just increased h/w resources
- 26. But scalability not great 0 1 2 3 4 5 6 7 8 0 100 200 300 400 500 600 700 Billions checks/day Total Cores Total Cores vs. Billions of checks/day (pre-tuning)
- 27. Tuning required! Scalability Post-tuning 0 2 4 6 8 10 12 14 16 18 20 0 100 200 300 400 500 600 700 Billions checks/day Total Cores Total Cores vs. Billions of checks/day (pre-tuning) Billions of checks/day (pre-tuning) Billions of checks/day (post-tuning)
- 28. Tuning – Optimize s/w resources (red arrows) 1 2 3 1. Minimize Kafka Consumers (thread pool 1) 2. Minimize Cassandra Connections 3. Maximize Cassandra client concurrency (thread pool 2)
- 29. Example 4 – What’s really going on - behind the Kafka partitions? ©Instaclustr Pty Limited, 2021 ©Instaclustr Pty Limited 2019, 2021, 2022
- 30. Kafka topic partitions enable consumer concurrency partitions >= consumers Partition n Topic “Parties” Partition 1 Producer Partition 2 Consumer Group Consumer Consumer Consumers share work within groups Consumer
- 31. Fan out requires many consumers and partitions Can be caused by: 1 Design – many topics and/or many consumers (Kongo Example) 2 Little’s Law (Anomaly Detection Example) Concurrency (Consumers) = Time x Throughput Slow consumers requires more of them to keep up target throughput – having 2 thread pools helped for Anomaly Detection example to reduce the consumer time and count
- 32. Kafka write architecture – partition replication
- 33. Benchmarking revealed that partitions and replication factor are the culprit 0 100000 200000 300000 400000 500000 600000 700000 800000 900000 1 10 100 1000 10000 TPS Partitions Kafka Partitions vs. Throughput Cluster: 3 nodes x 4 cores = 12 cores total Replication Factor 3 (TPS) Replication Factor 1 (TPS)
- 34. Implications? • Bigger Cluster (more nodes, bigger nodes) • Design to minimize topics and consumers • Optimize consumers for minimum time • Always benchmark with many partitions • Blame the Apache Zookeeper? • Responsible for Kafka control • From version 3.0 it’s being replaced by native KRaft protocol • May enable more partitions
- 35. Example 5 – Data Pipeline – multiple technologies Kafka Connect, Elasticsearch, PostgreSQL etc ©Instaclustr Pty Limited, 2021 (Source: Paul Brebner) See Blogs
- 36. ©Instaclustr Pty Limited, 2020 àApacheCon Open Source Performance Engineering Track à www.linkedin.com/pulse/call-papers-performance-engineering-track- apachecon-paul-brebner/ + Bullet Train à Speed and Scale
- 37. www.instaclustr.com [email protected] @instaclustr THANK YOU! For further Information see blogs www.instaclustr.com/paul-brebner/