Bandwidth: Use Cases for Elastic Cloud on Kubernetes
0 likes1,888 views
Bandwidth has been an avid user of the Elastic Stack for aggregating their logs from its many data centers. Learn how Bandwidth uses Elastic Cloud on Kubernetes to help satisfy various use cases.
Ad
More Related Content
What's hot (18)
Similar to Bandwidth: Use Cases for Elastic Cloud on Kubernetes (20)
![Hybrid Cloud, Kubeflow and Tensorflow Extended [TFX]](https://cdn.slidesharecdn.com/ss_thumbnails/tfx-kfp-191031073013-thumbnail.jpg?width=560&fit=bounds)
Ad
More from Elasticsearch (20)
![Modernising One Legal Se@rch with Elastic Enterprise Search [Customer Story]](https://cdn.slidesharecdn.com/ss_thumbnails/modernisingonelegalserchwithelastic-jsedit14thjune2021-210625013449-thumbnail.jpg?width=560&fit=bounds)
Ad
Recently uploaded (20)
Bandwidth: Use Cases for Elastic Cloud on Kubernetes
- 1. 1 Jeff Moore, Systems Engineer, Bandwidth Sam Mingolelli, Systems Engineer, Bandwidth 4/23/20 Use Cases for Elastic Cloud on Kubernetes (And how we got there)
- 2. 2 616 vs. 58
- 3. 3 Presenters… ● Tech stack: Kubernetes, Elastic Stack, AWS ● Focus: Openshift SRE, Elastic SRE ● Fun facts: Certified Kubernetes Administrator with plans to pursue Elastic Certified Engineer exam ● Tech stack: Kubernetes, Elastic Stack, AWS ● Focus: Openshift SRE, metrics & logging, hybrid cloud ● Fun facts: One of 5 moderators on the Stack Exchange site for Unix & Linux, father of 3 kids & 2 cats Jeff Moore is… Sam Mingolelli is…
- 5. 5 O U R S E R V I C E S
- 6. 6 Fun Facts About Bandwidth Incorporated 1999 Headquarters Raleigh, NC Number of Employees 700+ NASDAQ BAND
- 7. 7 Who's the Systems team at Bandwidth ● Systems team is separated into 2 sub teams ○ Infrastructure ○ Platform ● Platform Team manages (6 ppl): ○ Openshift ○ Elastic Stack ○ AWS services ● Infrastructure Team manages (5 ppl): ○ Storage (NetApp) ○ Virtualization (RHV) ○ Most everything else….
- 9. 9 Quick Snapshot of Today
- 11. 11 High Level Logging Needs ● Maintain 10+ Data Centers ● Guarantee integrity of log & metric data ● Perform maintenance of logging infrastructure without loss ● Retain logs & metrics for 30-60 days
- 12. 12 2016
- 13. 13 2016 - BELK + Kafka ● Kafka acts as a “queue” between the shippers & Elasticsearch ● Logstash performs the ETL of moving data from Kafka to Elasticsearch ● Kafka also a “queue” between Data Centers ● Kafka provides data rehydration in the event of data loss (Elasticsearch)
- 14. 14 2016 - BELK + Kafka - (Architecture) D daemon DP/DS K8s deployment VM Virtual Machine ML Machine Learning
- 15. 15 2017
- 16. 16 2017 - OpenShift - take 1 ● OpenShift/k8s is our strategic “platform” play ● Start building out Elastic Stack infrastructure within OpenShift/k8s ● OpenShift/k8s provides: ○ self-service for end users ○ Hybrid cloud across our Data Centers and AWS ○ Standardizes how we scale capacity ● Move from complicated Ansible roles/playbooks to k8s deployments (IaaS)
- 17. 17 2017 - OpenShift - take 1 - (Architecture) D daemon DP/DS K8s deployment VM Virtual Machine ML Machine Learning
- 18. 18 2018
- 19. 19 2018 - OpenShift - take 2 ● Convert monolith Logstash VMs into individualized Logstash per shipper type ● Now just a deployment ● Scaling is easy, bump replicas from X to Y ● Simplified Logstash configurations ● Removed complexity of multiple threads per Logstash, now 1 per CPU core
- 20. 20 2018 - OpenShift - take 2 - (Architecture) D daemon DP/DS K8s deployment VM Virtual Machine ML Machine Learning
- 21. 21 2018 - OpenShift - take 2 - (Example)
- 22. 22 2018 - OpenShift - take 3 ● Take 2 worked so well, move rest of beats into deployments/daemonsets ● Strategically better aligns for OpenShift 4.x and CoreOS ● Everything is a deployment, no daemons local to nodes ● Simplifies architecture
- 23. 23 2018 - OpenShift - take 3 - (Architecture) D daemon DP/DS K8s deployment VM Virtual Machine ML Machine Learning
- 24. 24 2018 - OpenShift - take 3 - (Example) Logstash Deployments Filebeat & Metricbeat Daemonsets
- 25. 25 2018 - Scaling the Consumption of Elastic Stack ● Self service to more efficiently scale our group's ability to support ES (5 ppl) ● Allow our group to not be in critical path of all ES consumption ● Act more as consultant to ES usage
- 26. 26 2018 - Elastic Cloud ● New use case with an unspecified amount of data appeared ● Use case required ML ● Use case was different enough from standard log & metrics ● Spin up Elastic Cloud (AWS) as PoC/PoT ● Self-Service
- 27. 27 2018 - Elastic Cloud - (Architecture) D daemon DP/DS K8s deployment VM Virtual Machine ML Machine Learning
- 29. 29 2019
- 30. 30 What is ECK? (Architecture) DP K8s deployment KB Kibana CRD STS K8s statefulset NS K8s namespace ES Elasticsearch CRD
- 31. 31 2019 - ECK in our Infrastructure - (Architecture) D daemon DP/DS K8s deployment VM Virtual Machine ML Machine Learning
- 32. 32 2019 - Why ECK? ● Additional use cases continued to materialize for Elasticsearch ● Use cases were unique enough to justify their own clusters ● Continue to Strengthen OpenShift “platform” strategy by increasing features ● It fits well into the *aaS goal
- 33. 33 ECK Use Cases ● Replace InfluxDB with ES for APM style metrics ● Network FlowLogs ● Disposable ES for testing & upgrade PoC/PoT ● Self Service ES - (ownership)
- 34. 34 Replace InfluxDB with ES for APM style metrics ● Move away from InfluxDB supported stability at scale ● 12.5k events/sec ingested via logstash ● Mostly Java metrics
- 35. 35 Network FlowLogs ● Network Team needed visibility into their hardware ● Used ElastiFlow plugin ● ECK allows custom container images and init containers
- 36. 36 Disposable ES for testing & upgrade PoC/PoT ● Safe testing environment vs. testing on production clusters ● Accessible with production clusters with cross-cluster search and cross-cluster replication
- 37. 37 Self Service ES - (ownership) ● Allows for all other future use cases ● Strengthens our offerings as a team
- 38. 38 616 vs. 58
- 39. 39 Traditional Deployments vs. ECK - Deployment Ansible ECK Lines of Code 616 58 File count 9 2 Certificate management Manual Automagic Scaling Manual Automagic
- 40. 40 ECK Deployment Example - pt. 1
- 41. 41 ECK Deployment Example - pt. 2
- 42. 42 ECK Challenges... ● Becoming Kubernetes-native is hard ● Becoming Kubernetes-aware is much harder ● Operational benefits outweigh challenges by a very large margin
- 43. 43 2020+ - Future Plans ● Continue “refactoring” components into OpenShift ○ Kibana ○ ES Masters ○ ES Data Nodes? ECK? ● Build Operators within OpenShift to IaaS ES APIs (or maybe Elastic can…?) ○ Watchers ○ RBAC ● ES for… ○ SIEM ○ APM ○ ILM ● Elastic as a data source - Add what you want on top (Kibana, Grafana, etc.)
- 44. 44 How to Reach Us Jeff https://github.com/slmingol https://www.linkedin.com/in/sammingolelli/ Sam https://www.linkedin.com/in/jeff-moore-k8s https://github.com/geoffmore
- 45. 45 We'll answer questions in chat
- 46. 46 Thanks!