Why Cloud-Native Kafka Matters: 4 Reasons to Stop Managing it Yourself
0 likes538 views
The document discusses 4 reasons to use a cloud-native Kafka service like Confluent Cloud instead of managing Kafka yourself. It notes that managing Kafka requires significant investment of time and resources for tasks like architecture planning, cluster sizing, software upgrades, and more. A cloud-native service handles all operational overhead automatically so you can focus on your core business. Confluent Cloud specifically offers elastic scaling, infinite data retention, global access across clouds, and integrations to make it a complete data streaming platform.
![Core vs. Non-Core
“Whatever is the new thing that creates
differentiation, that’s core.” Meanwhile, context
is any activity that doesn’t differentiate the
company in the eyes of the customer. “If we do
[context] brilliantly, we don’t succeed. If we do
it badly, we get in trouble.”
— Geoffrey Moore, author and management expert](https://image.slidesharecdn.com/finalwhycloud-nativekafkamatters4reasonstostopmanagingityourself-210615194410/85/Why-Cloud-Native-Kafka-Matters-4-Reasons-to-Stop-Managing-it-Yourself-5-320.jpg)
![Elastic Scaling in Confluent Cloud
Confluent Cloud
Milliseconds Minutes
Basic, Standard [0-100Mbps]
*Even in public clouds provider quotas for VMs, disks, security groups can be encountered causing delays. Confluent has these limits raised already.
Dedicated [Mbps - Gbps]
Select CKU from drop down in
cluster management UI and
click Apply Changes
Other Kafka Services
Days - Weeks
Determine how much capacity is needed
Procure capacity*
Configure new brokers
a. Disks b. OS c. Network d. Kafka (application)
Identify partitions on specific brokers to
rebalance & topics they are part of
For each Topic: migrate partitions
a. Increase ISR +1 b. Wait for new replica to sync
c. Failover master d. Reduce ISR -1 e. Delete old replica
Automated 1 Click](https://image.slidesharecdn.com/finalwhycloud-nativekafkamatters4reasonstostopmanagingityourself-210615194410/85/Why-Cloud-Native-Kafka-Matters-4-Reasons-to-Stop-Managing-it-Yourself-11-320.jpg)
Ad
More Related Content
What's hot (20)
Similar to Why Cloud-Native Kafka Matters: 4 Reasons to Stop Managing it Yourself (20)
Ad
More from DATAVERSITY (20)
Ad
Recently uploaded (20)
Why Cloud-Native Kafka Matters: 4 Reasons to Stop Managing it Yourself
- 1. Why Cloud-Native Kafka Matters: 4 Reasons to Stop Managing It Yourself Greg Murphy Senior Product Marketing Manager, Confluent
- 2. Data-Driven Operations Real-time, Personalized, Customer Experiences New Business Models You know the value of data in motion
- 3. The Power of Kafka 80% Fortune 100 Companies Using Apache Kafka® (majority are Confluent customers) 3
- 4. I N V E S T M E N T & T I M E SCALE The toll and risk of managing Kafka grows over time ● Architecture planning ● Kafka cluster sizing ● Cluster provisioning ● Build & set up source/sink connectors ● Build monitoring and reporting tools ● Maintain software patches and upgrades ● Monitor and react to security and reliability risks ● Maintain, upgrade, and restart servers ● Monitor Kafka cluster loading and rebalancing ● Plan and execute cluster expansions ● Provide utilization visibility across the organization ● Maintain cluster performance through infra upgrades, migrations, etc ● Manage Zookeeper ● And so much more . . .
- 5. Core vs. Non-Core “Whatever is the new thing that creates differentiation, that’s core.” Meanwhile, context is any activity that doesn’t differentiate the company in the eyes of the customer. “If we do [context] brilliantly, we don’t succeed. If we do it badly, we get in trouble.” — Geoffrey Moore, author and management expert
- 6. The Move to Cloud-Native Data Systems 6 Storage Data Warehousing Databases Amazon S3 Blob storage Redshift Snowflake Google BigTable Self managed hardware and software Fully managed services Data Movement “By 2022, 75% of all databases will be deployed or migrated to a cloud platform.” - Gartner
- 7. The cloud-native difference with Confluent
- 8. Partially Managed (cloud-hosted) Manual operations with tools and/or support Ease and speed of use Low High Other managed services for Kafka Confluent is the only cloud-native Kafka solution Self-Managed (DIY) Development and maintenance without support Fully Managed (cloud-native) Automated product capabilities with no operational overhead Confluent Cloud
- 9. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. The 4 Benefits of a Cloud-Native Kafka Service Scale up instantly to meet any demand and scale back down to avoid over-provisioning infrastructure Elastic Build and launch faster with a complete data in motion platform that goes way beyond Kafka Complete Infinite Store unlimited data on Confluent to enhance your real-time apps and use cases with a broader set of data Global Create a consistent data fabric throughout your organization by linking clusters across your different environments
- 10. Elastic
- 11. Elastic Scaling in Confluent Cloud Confluent Cloud Milliseconds Minutes Basic, Standard [0-100Mbps] *Even in public clouds provider quotas for VMs, disks, security groups can be encountered causing delays. Confluent has these limits raised already. Dedicated [Mbps - Gbps] Select CKU from drop down in cluster management UI and click Apply Changes Other Kafka Services Days - Weeks Determine how much capacity is needed Procure capacity* Configure new brokers a. Disks b. OS c. Network d. Kafka (application) Identify partitions on specific brokers to rebalance & topics they are part of For each Topic: migrate partitions a. Increase ISR +1 b. Wait for new replica to sync c. Failover master d. Reduce ISR -1 e. Delete old replica Automated 1 Click
- 12. Elastic
- 13. Infinite
- 14. Infinite Retention Scale up storage without adding compute resources by removing storage limits on topics to enable infinite data retention in Kafka Cluster Broker 1 Broker 2 Broker 2 Topic 1 Topic 2 Topic 3 Topic 2 Topic 3 Topic 2 Topic 3 Topic 1 Topic 1 Available on AWS and Google Cloud
- 15. Infinite Retention Cluster Broker 1 Broker 2 Broker 2 Topic 1 Topic 2 Topic 3 Topic 2 Topic 3 Topic 2 Topic 3 Topic 1 Topic 1 Available on AWS and Google Cloud ● Reimagine event streaming apps that can use both real-time and historical events ● Faster and better informed decision making for your event-driven processes ● Meet regulatory compliance requirements for data retention
- 16. Infinite
- 17. “Confluent gives us the tools we need to drive innovation. Before Confluent Cloud, when we had broker outages that required rebuilds, it could take up to three days of developers’ time to resolve. Now, Confluent takes care of everything for us, so our developers can focus on building new features and applications.” — Jon Vines, Software Development Team Lead at AO.com AO.com maintains focus on value with Confluent
- 18. Global
- 19. 19 Deploy how and where you want Confluent AWS MARKETPLACE Azure MARKETPLACE Google Cloud MARKETPLACE Start your multi-cloud journey Simplify procurement with easy click-though through annual commitments and draw down against existing Enterprise Agreements in your cloud marketplace of choice 20 regions 16 regions 22 regions
- 20. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Global Access ● Create a consistent data layer across clouds: easily share data between fully independent clusters ● Simplify your architecture: bridge or migrate data without needing a separate Connect cluster Cluster Linking (preview) requires no additional infrastructure and fully preserves offsets Migrate clusters to Confluent Cloud Access real-time data globally and across public and private clouds with linked Kafka clusters that sync in real-time
- 21. Complete
- 22. Build and launch faster with a Complete data in motion platform Pre-built Integrations Data Governance Stream Processing Enterprise-Grade Security Instantly integrate clusters to the most popular data sources and sinks within the Kafka ecosystem Make data backwards compatible and future-proof with Schema Registry & Validation Launch real-time and continuous stream processing apps with a lightweight SQL syntax Ensure data confidentiality, control access management, and monitor user activity
- 23. Instantly Connect Popular Data Sources & Sinks Choose from a growing list of fully managed connectors in Cloud Confluent Sources Sinks Amazon Kinesis Azure Event Hubs Datagen GCP Pub/Sub Microsoft SQL Server CDC Microsoft SQL Server MongoDB Atlas MySQL CDC MySQL Oracle Database Postgres CDC Postgres Salesforce CDC Elasticsearch Service GCP BigQuery GCP Dataproc AWS Lambda Azure Functions GCP Cloud Storage Microsoft SQL Server MongoDB Atlas MySQL Postgres Snowflake Visit confluent.io/hub to see all available connectors Datadog Metrics
- 24. MongoDB Atlas Source & Sink Connectors Build and launch real-time apps faster with fully managed source & sink connectors. Now Generally Available! Deploy with just a few clicks and eliminate the efforts to build, test, & maintain custom integrations Work with data anywhere with setup available across all major clouds including AWS, Azure, and Google Cloud Drive analysis with real-time data by connecting Apache Kafka® and MongoDB
- 25. Manage Schema Registry Centrally create, edit and view topic schemas, and compare schema versions Configure Schema Validation Enable the feature at the topic level when creating or editing topics Simplify Enterprise Schema Management
- 26. ksqlDB provides everything you need to build a complete, end-to-end event streaming application entirely with SQL syntax DB APP APP DB PULL PUSH CONNECTORS STREAM PROCESSING STATE STORES ksqlDB 1 2 APP Simplify Your Stream Processing Architecture
- 27. Protect sensitive data in the cloud ● SOC 1, SOC 2, SOC 3 ● ISO 27001 ● PCI, HIPAA1 ready, GDPR ready Enterprise grade security and compliance Secure data against unwanted access ● Data-at-rest encryption ● Data-in-motion encryption ● Private networking connectivity Control access to cluster & resources ● SAML / SSO ● RBAC & Audit Logs ● Access Control Lists (ACLs) 1 Available on Dedicated clusters only
- 28. How much does Kafka cost your business?
- 29. The Cost-Efficient Way to Deploy Kafka Confluent Cloud (typical savings) 1 2 3 ⬇60 % Cloud Infrastructure Operational (FTE) Support and other spend Total self-managed Confluent Cloud
- 30. Try Confluent Free De-risk, accelerate, and save with Confluent Request TCO Assessment confluent.io/request-kafka-tco-assessment
- 31. Thank You