Modern Cloud-Native Streaming Platforms: Event Streaming Microservices with Apache Kafka on Kubernetes
Download as PPTX, PDF6 likes3,610 views
The document discusses modern cloud-native streaming platforms, specifically focusing on the utilization of Kafka on Kubernetes for event streaming and microservices. It highlights the transition from traditional application delivery to a more flexible, microservices architecture that supports continuous delivery and operational efficiency. Additionally, it covers the capabilities and benefits of the Confluent Operator for automating Kafka deployment and management within Kubernetes environments.
Modern Cloud-Native Streaming Platforms: Event Streaming Microservices with Apache Kafka on Kubernetes
- 1. Modern Cloud-Native Streaming Platforms: Event Streaming Microservices with Kafka on Kubernetes
- 2. Speakers Kamala Dasika Product Team Cloud Platform and Ecosystem Pivotal @DasikaKN Michael Ng Product Manager Confluent @Micha8LNg
- 3. Changing Model for Application Delivery Development Deployment Sparingly at designated times Ready for prod at any time Architecture Abstraction Layer “Day 2” Ops App Server on Machine App on “disposable” infrastructure App and Data Monolith Microservices / Composite app Linear / Sequential Short cycles, test driven, iterative Many tools, ad hoc automation Manage services, not servers
- 4. Driven by Modern Cloud Native Platform Cont inuous Del ivery An idea in the morning can ship by evening Microservices Release #1 Microservices Release #2
- 6. Hardware IaaS CaaS PaaS FaaS HPE, Dell, IBM, Lenovo Cloud providers: AWS, GCP, Microsoft Azure, VMware Kubernetes Cloud Foundry, Heroku AWS Lambda, Azure Functions, OpenWhisk, kubeless, PFS
- 7. Hardware IaaS CaaS PaaS FaaS Strategic goal: Push as many workloads as technically feasible to the top of the platform hierarchy Higher flexibility and less enforcement of standards Lower development complexity and higher operational efficiency
- 8. Hardware IaaS CaaS PaaS FaaS • Data warehouses • Storage with regional implications • Hypervisors • Relational DBs • Packaged software • Virtual appliances • Application platforms (cloud abstraction) • Containerized Monoliths • Stateful workloads • Databases • Messaging • 12 -15 factor apps • stateful and stateless options • APIs • Streaming apps • DB copies • Change synchronization • Real time stream analytics
- 9. Where to run what?
- 10. Goal: Operational consistency AND multiple abstractions FREEDOM: Run Your App on Any Cloud AUTOMATION: Lowers Ops Cost and Increases Velocity RESILIENCY: Auto-Scale, Blue-green Deploys, Health Mgmt. GOVERNANCE: Common Developer-Self-Serviced Frameworks VISIBILITY: Full Stack Logging, Monitoring & Auditability SPEED & SCALE: Cut Cycle Time from Months to Days/Hours Built for Cloud Native New Initiatives Existing Workloads Modernize to Cloud-Native?
- 11. Outcomes trump technology bingo Needs High Dev Efficiency Needs Low Dev Efficiency App Rarely changes App Frequently changes • Traditional apps • Leave as-is • Re-architect • Replatform • CI/CD • Evaluate • Evaluate
- 12. Evaluating the Enterprise App Portfolio Business Technical Economic Score • Business criticality • Risk tolerance • Change frequency • Lifecycle stage • Domain expert availability • Licensing costs (HW / SW) • Time-to-market implications • Revenue impact-fullness • Codebase • Suitable framework / runtime • Relatively lightweight footprint • No “hard wired” file system dependencies • Usage / workload For each app
- 13. Application Transformation Strategies Re-Host Containerize aka “lift and shift” to CaaS and gain infrastructure + platform ops benefits Re-Platform Minimal readiness effort to run on PaaS (4 -7 factor apps + data service) that can run on PaaS Re-Factor Convert to cloud- native/microservice (12-15 factor app + data service) runs well on PaaS/FaaS Re-Build Decompose and rewrite from the ground up Application Portfolio Funneling Process Learn More: Tools and Recipes to Replatform Monolithic Apps to Modern Cloud Environments
- 14. Evolution of the enterprise application portfolio Learn More: Microservices, Events, and Breaking the Data Monolith with Kafka Microservices or Dist. Monolith
- 15. Now what about Kafka?
- 16. “The world’s most valuable resource is no longer oil, but data” - The Economist, May 6 2017
- 17. Apache Kafka is a Distributed Event Streaming Platform Process streams of events In real time, as they occur 110101 010111 001101 100010 Publish and subscribe to streams of events Similar to a message queue or enterprise messaging system 110101 010111 001101 100010 Store streams of events In a fault tolerant way 110101 010111 001101 100010
- 18. Events A Sale An Invoice A Trade A Customer Experience
- 19. What is a company? A business is a series of events and reacting to those events.
- 20. Events enable Microservice Architectures - Let’s refactor! Submit Order Orders Service Shipping Service Customer Service Webserver E-Commerce Microservices (with RPC) - Orders Service calls Shipping Service to tell it to ship item. - Shipping service looks up address to ship to (from Customer Service) - No Kafka 😢shipOrder() getCustomer()
- 21. Refactoring Orders and Shipping Message Broker (Kafka) Submit Order Order Created getCustomer() RPC Orders Service Shipping Service Customer Service Webserver - Orders Service no longer knows about the Shipping service (or any other service). - Events are fire and forget.
- 22. Refactoring Customers Customer Updated Submit Order Order Created Webserver KAFKA Orders Service Shipping Service Customer Service - Call to Customer service is gone. - Data is replicated, as events, into the shipping service, where it is queried locally.
- 23. Events are the key to scalable microservices Orders Service
- 25. Why run Kafka on Kubernetes? 33% of Confluent Customers plan to run Production Kafka on Kubernetes
- 26. Microservices - driving container and container orchestration adoption Breaking down the monolith - enables dev teams to work and scale independently µ Containers allow services to be independently deployable services ● Each service has its own purpose & implementation language ● Each service has its own dependencies Orchestration - running containers at scale ● Microservices = container deployments at scale ● Service discovery ● Scale services independently ● High Availability, Self Healing of services ● Hardware Abstraction
- 27. The world seems to think Kafka on Kubernetes is Difficult
- 28. There are some challenges Translating an existing architecture to Kubernetes: K8s pods, nodes, stateful sets, custom resources External access: What do we do with advertised listeners on Kubernetes? Managing configuration of Kafka, Zookeeper and all Confluent Components Persistent Storage options on prem and clouds Security Configuration and Upgrades
- 29. Confluent Operator enables you to: Automate provisioning of Kafka pods in minutes Monitor SLAs through Confluent Control Center or Prometheus Scale Kafka elastically & Automate rolling updates Built on our first hand knowledge of running Confluent at scale Confluent Operator is our Kubernetes Operator and our first step towards becoming cloud native CONFIDENTIAL
- 30. Helm - The Confluent Operator Package Manager ● Confluent Operator leverages Helm Charts to deploy, upgrade and uninstall Confluent Platform custom resources and pods ● Configuration front end for users to specify how a Confluent Platform Cluster is deployed: ○ # of replicas for Kafka, Zookeeper ○ Security and Authentication configuration ○ Persistent Storage configuration ● Cluster configuration edits are also performed using Helm Operator Helm Charts - yaml
- 31. Confluent Operator - Automated Provisioning
- 32. Confluent Operator - Scale Horizontally Automate Scaling: Spin up new brokers, connect workers easily Distribute partitions to new brokers: Determine balancing plan Execute balancing plan Monitor Resources
- 33. Confluent Operator - Rolling Upgrade of all components Automated Rolling Upgrades of all components - Kafka Brokers, Zookeeper, Connect, Control Center Kafka Broker Upgrades: 1. Stop the broker, upgrade Kafka 2. Wait for Partition Leader reassignment 3. Start the upgraded broker 4. Wait for zero under-replicated partitions 5. Upgrade the next broker
- 34. Confluent Operator - Automated Security Configuration SASL PLAIN and Mutual TLS Authentication Automate configuration of truststores and keystores with secret objects Automate configuration of Kafka and all Confluent Platform Components
- 35. Confluent Operator GA Plans We are in private Preview Release now 24 customers testing the Operator in Preview: ● Global customers ● Banks, Fin Tech, Retailers, Consumer Tech We are in the final stages of Preview and about to launch soon.
- 38. Demo – Provision a Complex Cluster in K8s with Confluent Operator ● Provision: ○ 3 Kafka Brokers, 3 Zookeeper Nodes ○ Confluent Control Center ○ Provisioned on GKE (7 node cluster) ● Mutual TLS Authentication, External Access via Service Endpoints, DNS ● Pub/Sub example (remote access to Kafka Cluster)