Operating Kafka on AutoPilot mode @ DBS Bank (Arpit Dubey, DBS Bank) Kafka Summit 2020

Kafka is among the few cool technologies that play a pivotal role in the digital journey of DBS to become one of the best bank in the world. Kafka is the foundation for our vision of application modernisation by: - making our systems event-driven - moving batch workloads to real-time - serving various financial industry use-cases that warrants proactive actions i.e Fraud detection, Anomaly and unusual behaviour detections. To realize our Digital vision and to keep up with the pace of this transformation, we decided to run Kafka ""as-a-self-managed-service"" at large scale. This requires minimising reliance on human operators/ engineers and have intelligent apps make all important operation related decisions. In this session, I would like to deep-dive into these intelligent apps that not only helped us maintain 100% availability but also provides self-service experience and a truly democratised Kafka ecosystem for our community. Some of these apps include : 1. Self-service on-boarding and stream discovery built by esp. leveraging Kafka admin API 2. AI Ops - using machine learning for predicting resource utilisation and early anomaly detection 3. Automated Cluster scaling, deployment and maintenance 4. Self-healing services 5. Intelligent Chaos engineering - to make apps ready for production 6. Black box Kafka monitoring 7. Load simulator for performance testing - to bring production like experience in development environments 8. Automatic tiering and streams archival - to meet several regulatory compliance and to make brokers lightweight