DCSF19 CMD and Conquer: Containerizing the Monolith
0 likes248 views
The document outlines the transition from monolithic software to microservices at Splunk, highlighting the challenges and benefits involved. It emphasizes the use of containerization and orchestration tools like Docker and Ansible to enhance scalability, fault tolerance, and deployment efficiency. The future direction includes improving accessibility and dynamic orchestration while bridging the gap during the migration process.
DCSF19 CMD and Conquer: Containerizing the Monolith
- 1. CMD and Conquer PACKAGING THE MONOLITH
- 2. Software Engineer, Splunk Tony Lee Senior Software Engineer, Splunk Nelson Wang
- 7. Time Intensive Disruptive Expensive $ monolith | microservice
- 9. Our Journey
- 10. Taking the First Step
- 12. APPLICATION REQUIREMENTS SCHEDULER REQUIREMENTS ACCESSIBILITY ADAPTABILITY AUTONOMY FAULT TOLERANCY SCALABILITYSTATELESS
- 13. BASE PLATFORM LAYER SPLUNK LAYER FORWARDER LAYER
- 14. BASE PLATFORM LAYER As Splunk evolves and develops, the image should grow with it
- 15. • Add file/directory monitoring • Open a TCP/UDP socket & listen for input • Enable receiving of syslog data • Setup HTTP event collection token • Forward data to another service • Setup administrator password • Create user accounts for access control • Install an enterprise license • Connect to an existing license master • Setup indexes • Touch configuration files • Install apps • etc. Usability
- 16. • Asynchronous dynamic provisioning performed by Ansible • Eliminate reliance on any third-party service registrar/catalog • Parallelized setup removed ordered dependencies and results in faster time-to-value Micro-Orchestration Layer entrypoint.sh Dockerfile
- 17. Why Ansible? Domain-specific, developer-friendly language that enables the dynamic configurability
- 18. APPLICATION REQUIREMENTS SCHEDULER REQUIREMENTS ACCESSIBILITY ADAPTABILITY AUTONOMY FAULT TOLERANCY SCALABILITYSTATELESS
- 19. • Intermittent service interruption is expected within configuration layer • Health-checks and probes to assist in lifecycle management Fault Tolerance
- 20. • Orchestration engine administers the pods/containers • Application-context is shared amongst cluster members via environment variables Distribution of State SPLUNK DEPLOYMENT Container Indexer 2 Container Indexer 1 Container Cluster Master Container Search Head
- 21. SPLUNK DEPLOYMENT Container Indexer 2 Container Indexer 1 Container Cluster Master Container Search Head
- 22. SPLUNK DEPLOYMENT Container Indexer 2 Container Indexer 1 Container Cluster Master Container Search Head
- 23. SPLUNK DEPLOYMENT Container Indexer 2 Container Indexer 1 Container Cluster Master Container Search Head
- 24. SPLUNK DEPLOYMENT Container Indexer 2 Container Indexer 1 Container Cluster Master Container Search Head
- 25. SPLUNK DEPLOYMENT Container Indexer 2 Container Indexer 1 Container Cluster Master Container Search Head
- 26. Dynamic scaling of specific roles How do I add a new indexer? Scalability SPLUNK DEPLOYMENT Container Cluster Master Container Search Head 1 Container Indexer 2 Container Indexer 1 Container Indexer 3
- 28. Our Results
- 29. External Usage • Customers have been waiting to run Splunk on Docker! • Adoption amongst many members of the open-source community • Examples for both swarm + k8s manifests in our GitHub repo
- 31. Internal Usage Support • Cheap, reproducible environments • Support any size topology, any version Open-source images are fed back into our internal deployment provisioning service (orca) used across the company Sales • Rapid, reliable demo instances on-demand • Support for local scenarios for the field Dev/QA • Disposability around testing changes • Accelerates iteration cycles in CI/CD
- 32. How? orca leverages Docker Enterprise and Universal Control Plane (UCP) Node Worker Node Worker Node Worker Node Worker swarm mode cluster Node Worker Node Worker Dev/QA/Sales Team Using Swarm/K8s Support Team using Swarm/K8s Ops Team DOCKER ENTERPRISE EDITION
- 33. ORCA
- 35. Image CI/CD • Support for multiple platforms • Integrated security scanning • Functional tests for various configurations and topologies • Version control, custom dependencies for individual teams
- 36. Pipeline GitHub IBM s390x Agent $ build splunk- debian Linux Agent $ build splunk- debian Linux Agent $ push splunk-debian DockerHub Public/Open-source Internal
- 37. The Future • Deployment frameworks (helm, operators) • Find ways to make Splunk more accessible and reduce time-to- value • Always consider the future of the monolith!
- 38. Summary • Find ways to bridge the gap while migrating monolithic software to microservice architecture • Plan for the future • Dynamic self-orchestration made our software offering more nimble.
- 39. Get Hands On! Repositories • https://github.com/splunk/docker-splunk • https://github.com/splunk/splunk-ansible Documentation • https://splunk.github.io/docker-splunk • https://splunk.github.io/splunk-ansible #docker @ http://splunk-usergroups.slack.com/