Zero Down Time Move From Apache Kafka to Confluent With Justin Dempsey | Current 2022
0 likes951 views
The document outlines the transition from an open-source Apache Kafka setup to Confluent's managed Kafka solution to address scalability and operational challenges at SAS. It discusses objectives such as lowering total cost of ownership and minimizing operational complexity, and explores various migration strategies, including the use of Confluent Replicator and Cluster Linking. The migration was successfully completed by October 2021, achieving no data loss and meeting ISO certification requirements.
Zero Down Time Move From Apache Kafka to Confluent With Justin Dempsey | Current 2022
- 1. Copyright © SAS Institute Inc. All rights reserved. Copyright © SAS Institute Inc. All rights reserved. Zero Down Time Move from Apache Kafka to Confluent using the Confluent Operator and Cluster Linking
- 3. x x ABOUT US
- 4. x x x Customer Intelligence CI360 x CI360 merges digital & traditional data sources for a "360" view promoting a complete picture of your customer 360 View A/B testing and other features allow for users to evaluate the effectiveness of their marketing campaigns and creatives "What-if" Creative Content Analysis Cloud native software as a service solution utilizing modern architecture principals to ensure a secure and performant approach to managing your marketing campaigns SaaS Microservices Architecture
- 6. Monolithic VM backed Kafka - Hardware and OS Scaling options & Adaptability to Seasonal Demands Cross Cloud Future State Doubling YoY Kafka event Volume - Additional Regions EOS/EOL OS Concerns Inability to add capacity quickly CI360 KAFKA USAGE (2020) WEAKNESSES OPPORTUNITIES THREATS Processing Trillion+ Events (2020) w/o disruption Globally Distributed Presence STRENGTHS
- 7. x x _ What Problem are we trying to Solve?
- 8. "Scalability issues with self-supporting open source Kafka in-house were making it difficult to adjust to changing demand, and operational overhead and complexity were driving significant costs..." We had to evolve and adapt... - https://www.confluent.io/customers/sas/
- 9. address aging OS, hardware, and supportive software EOS/EOL Objective 1 lower Total Cost of Ownership (TCO) Objective 2 reduce complexity for adding capacity Objective 3 minimize SLA support risk(s) Objective 4 Objectives
- 10. 1. All migrations complete prior to Calendar Q4/2021 2. No data loss 3. No SLA/SLO violations during migration 4. ISO certification requirements for Kafka complete 5. Reduction in TCO Criteria For Success
- 11. 3 Options
- 12. Confluent Confluent provides services and technolog(ies) that are well suited for the CI360 SaaS Solution and that will help the product grow and evolve as CI360 expands into additional Data Centers and Cloud Providers
- 15. Kafka on K8S Proposed Architecture
- 16. In order to move from the original kafka cluster on VM to Kafka on Kubernetes, we needed to address three primary kafka usage patterns. Migration Problem Statement
- 17. 1 Raw Data Ingest 2 Transformational Topics 3 Agent / External
- 18. Option 1: MirrorMaker 1 w/ no Kafka Upgrade Flow Details PROS: o Setup a new cluster with Kafka 2.7 in Kubernetes o Make a list of all customer configs for all topics present o Start MM1 o Validate all topics and update the custom configs o Migrate consumers o Migrate producers o Stop MM1 o No upgrade to 2.7 for existing clusters CONS: o Topic configs are not copied o Offsets are not preserved from source to destination o Topic partitions are not preserved ~ No guarantee of order across partitions. o No UI/Portal integration - status monitoring brittle Producer Broker Consumer Producer Broker Consumer SAS Cloud Kafka 2.3 (Original) Kafka 2.7 (Future) MirrorMaker1
- 19. Option 2: MirrorMaker 2 w/ Upgrade (Minimum 2.4) Flow Details PROS: o Upgrade kafka in existing clusters to 2.5.x o Setup new cluster with minimum Kafka 2.7 in Kubernetes o Start MM2 o Migrate consumers o Migrate producers o Stop MM2 o Offset Translation o Consumer Group Checkpoints o Cross Cluster Replication metrics o High level driver and configuration file for defining global topology o Partition synchronization o Configuration synchronized CONS: o Prerequisite step of upgrading existing clusters to a minimum of 2.4.x o Uses Connect Converters o Not byte-level o No UI/Portal integration - status monitoring brittle MirrorMaker2 Producer Broker Consumer Producer Broker Consumer SAS Cloud Upgrade Original to Kafka 2.5.x Kafka 2.7 (Future)
- 20. Option 3: Confluent Replicator w/Upgrade (Minimum 2.4) Flow Details PROS: o Setup new cluster with minimum Kafka 2.7 in Kubernetes o Make a list of all configs for all topics present o Start Confluent Replicator o Migrate consumers o Migrate producers o Stop Confluent Replicator o Same as Replicator o Multi-region replication and hybrid cloud replication o Handles topic configuration and data o Integrates with Kafka Connect and Confluent Control Center CONS: o Source must be (minimum) confluent platform 5.4 or Kafka 2.4 and destination confluent 6.0 (minimum) o Must Upgrade Original to minimum of 2.4 o Uses Connect Converters o Not byte-level Producer Broker Consumer Producer Broker Consumer SAS Cloud Upgrade Original to Kafka 2.5.x Kafka 2.7 (Future) Confluent Replicator
- 21. Option 4: Confluent Cluster Linking w/Upgrade (Minimum 2.7) Flow Details PROS: o Setup new cluster with minimum Kafka 2.7 in Kubernetes o Create Cluster Link(s) o Create source (if relevant) and mirror topic(s) o Consume from the mirror topic(s) ("topic-to-link") o Complete replication/consumer offset(s) to zero o Migrate Consumers o Migrate Producers o Break the link(s) o Offset Translation o Consumer Group Checkpoints o Cross Cluster Replication metrics o High level driver and configuration file for defining global topology o Partition synchronization o Configuration synchronized o Monitoring the replication process via Control Center o Byte Level Replication o Connect/VM/Custom Tooling not needed CONS: o Prerequisite step of upgrading existing clusters to a minimum of 2.7 o Cluster Linking (as of April 2021 - Pre-General Availability) Producer Broker Consumer Producer Broker Consumer SAS Cloud Upgrade Original to Kafka 2.7.x Kafka 2.7 (Future) Cluster Linking
- 22. oUtilize Cluster Linking for Kafka Migration oUtilize Confluent Operator for Migration and Ongoing Systems Management Migration Key Technical Objectives
- 23. 1. Cluster Linking usability maturity a) Much orchestration to efficiently do parallel migrations/cutovers b) No UI/UX facet(s) c) Breaking the Mirror is a little "klunky" d) No cluster level topic mirror configuration a) Creating the links was a little tedious / an abstraction to create multiple links at one time or similar would have been nice (wasn’t overly mature in early 2021 - we had to to write a simple loop abstraction and loop through a dictionary of topic names) e) Not integrated with Control Center f) Active/Active configuration not practical (i.e., bidirectional – or “two-way” but same topic names) 2. Consumer Offset Mirroring is disconnected from the main mirroring mechanism for topics (a little brittle) 3. Fail-back options were very limited 4. Operator 2.0 (GA as of 5/11) – Working on PreGA a) Missing Components Challenges
- 24. 2 Allow all Consumer Groups to Drain x x x x x 3 4 5 6 Migration Steps Divert Ingest Topics to "Hold" 3 days prior to cutover Migrate Schemas No New Schema Updates Shutdown Consumer Facing APIs Setup Cluster Linking 1 Day for Replication Monitor Lag Break Cluster Linking & Migrate Ingest Topics - <30 minutes x 7 8 Shutdown Consumer and non-critical Producer Services Service Configuration overide for handful of Ingest Consumers x 9 10 Update Global Settings for Apps to use New Cluster Rolling Restart of all Services Bounce Config Service x 11 x x x Q42021 3 days prior to Migration Day of Migration Migration Complete x 1 Upgrade Original Cluster to 2.7
- 25. Review Objectives Address aging OS, hardware, and supportive software EOS/EOL Reduce complexity for adding capacity Lower Total Cost of Ownership (TCO) Minimize SLA support risk(s)
- 26. Calendar All migrations were completed by October 2021 addressing Hardware & Licensing Concerns going into Q4 Certification ISO Certification requirements were met going into October 2021 x TCO Projected Cost Reduction 69% SLA/SLO met No data Loss during the migration and customers were able to produce messages at all times supporting "minimal disruption"
- 27. Application Workload Requirements Direct Costs Indirect Costs Hidden Costs Total Cost Of Ownership o Support Staff o Downtime o Addl Network Mgmt o Infrastructure Costs o Software Purchase Costs o Administration Costs o Upgrades o Maintenance o Tech Support o Training
- 28. Consideration Details Infrastructure Costs Hardware, network, egress, & other factors Timeline Hard dates for completion – October 1, 2021 - Confluent Operator GA, EOL/EOS software dates, ISO certification Technologies Confluent Cloud, Confluent Platform, Apache Kafka, Strimzi Operator Support Services Confluent support tiers, existing SAS Cloud support teams ~ SME’s for Kubernetes and Kafka Licensing and Software Costs Confluent Cloud & Confluent Platform Pricing Structures SAS Procurement & Compliance Process(es) & Controls Onboarding a new cloud PaaS solution could take months to implement. Relationship with Confluent for on-premises licensing had already begun going into 2021. TCO Consideration Factors
- 29. 1 YEAR LATER
- 30. Copyright © SAS Institute Inc. All rights reserved. Copyright © SAS Institute Inc. All rights reserved. Justin Dempsey Senior Software Development Manager, SAS Cloud [email protected]