MesosCon_Asia_2016_DevOps_at_GSShop
0 likes90 views
This document discusses the transition from traditional operations and development practices to a DevOps model over time. It begins with the challenges of the past state including long lead times, lack of collaboration between teams, and reliance on big bang releases. The beginning of the change involved adopting principles like infrastructure as code and shared tooling/platforms to improve productivity. Fully adopting DevOps required establishing new roles, processes, and addressing cultural issues. The road ahead focuses on continuous improvement areas like architecture constraints, multi-tenant clusters, and advanced application deployment techniques. The overall summary discusses the evolution from initial changes to mature DevOps practices.
Ad
More Related Content
What's hot (18)
Viewers also liked (20)
Ad
Similar to MesosCon_Asia_2016_DevOps_at_GSShop (20)
Ad
More from Vivek Juneja (9)
MesosCon_Asia_2016_DevOps_at_GSShop
- 2. IT Innovation Center *Dev*Ops*, Cloud Infrastructure, Microservices, Containers 2015 - Current Founding Member, Container Platform Team vivekjuneja
- 4. Happy families are all alike every unhappy family is unhappy in its own way. 幸福的家庭大抵相同，而不幸的家庭却各 有其不幸。
- 5. Productive teams are all alike every unproductive team is unhappy in its own way.
- 6. Productivity = Happy Teams
- 7. AGENDA
- 8. NOT A LONG AGO BEGINNING OF THE CHANGE ADOPTING THE CHANGE THE ROAD AHEAD 1995 - 2015 2015 - 2016 2016 - 2017 2017 - 2019
- 9. NOT A LONG AGO BEGINNING OF THE CHANGE ADOPTING THE CHANGE THE ROAD AHEAD
- 10. Source Repo NEXUS BUILDER & DEPLOYER DEV TEST STAGE PROD Build & Deploy Maintenance Development
- 11. Build & Deploy 7days 10 changes 10 days 3 changes Deploy Frequency Lead Time for Change Per developer per week Changes per Deploy
- 12. Introducing…
- 14. OPERATIONS* DEVELOPER* Stable system Minimal Changes Control on changes New Features Fast Changes Quick rollout to Prod
- 15. Service Management Monolithic App Simple well-understood Management Primitives Minimal Moving Parts
- 16. Multi-Apps / Microservices New and Complicated Management Primitives Too many Moving Parts Service Management
- 18. Yawn Driven Deployment (打哈欠) Deploy Code at 3 AM to Production
- 19. NOT A LONG AGO BEGINNING OF THE CHANGE THE ROAD AHEAD ADOPTING THE CHANGE
- 20. Know thy Issues
- 21. We manage our machines like pets. Pets get old, die and it is sad
- 22. Each team invents their own tools and processes
- 23. It takes a long time for developers to get feedback
- 24. Big Bang releases that are treated like religious events
- 26. Inspiration
- 28. Inverse Conway Maneuver Who is Melvin Conway
- 29. Conway’s Law organizations which design systems are constrained to produce designs which are copies of the communication structures of these organizations
- 30. organizations which design systems are constrained to produce designs which are copies of the communication structures of these organizations Conway’s Law
- 31. Inverse Conway Maneuver Design systems that impose constructive constraints on the teams to change the way they communicate and manage
- 32. Design systems that impose constructive constraints on the teams to change the way they communicate and manage Inverse Conway Maneuver
- 33. O-ring theory of economics Tasks of Production must be executed proficiently together in order for any of them to be of any value Michael Kremer
- 34. Adrian Colyer
- 38. Tenets
- 39. Disposable Apps Developer Productivity Shared Multi-tenant Infrastructure and Tooling Automate Service management primitives Measure and Log everything Tenets
- 40. Reverse Conway Maneuver O-ring theory of devops Apply ! Master Plan
- 41. Service Delivery Platform The building blocks for building reliable software at scale
- 42. End to End Workflow CODE Build Process Docker Image Deploy Preparation Deployment Manifest ZDD Load Balancer Reload DNS Update DEPLOY PHASE (for all environments)
- 43. APM Logging Dashboard Notification End to End Workflow
- 44. One Deployment Manifest to rule them all
- 47. DEV TEST STAGE PROD
- 48. Blue Green Deployment NEWOLD OLD OLD OLD OLD OLD HAProxy ZDD Control Marathon Mesos Control disabled
- 49. Blue Green Deployment Zero Downtime Deployment Ideally
- 50. Blue Green Deployment Deploy when awake !
- 51. Notifications
- 52. Custom Dashboard Metadata based Rollback Comprehensive Health Check uses Service Discovery Supports Multiple Data Centers / Platform Regions Common API for Developers for integration with CI Server
- 55. Devops Metrics
- 56. APM Monitoring and Alerts Hardware Monitoring (VM, Physical Machine) Service Monitoring (container, non-container) Container Platform Stack Monitoring Service Latency Service Tracing Audit Trail
- 57. APM Hardware Monitoring (VM, Physical Machine) Service Monitoring (container, non-container) Container Platform Stack Monitoring Service Latency Service Tracing Audit Trail Monitoring and Alerts
- 60. Fault Identification Notification APM Log Dashboard …….TRACEID………... …….MESOS_TASK_ID………. Marathon …….MARATHON_APP……… Kill / Kill and Scale
- 62. Fair Share Usage Gravity Platform DEV Environment #1 TEST Environment #1 DEV Environment #2 TEST Environment #1 DEV Environment #2 Disposable Transient Dev/Test Environments Hours
- 63. Platform Provisioning Mesos Agent Docker Monit Log Forwarder cAdvisor Worker Node Mesos Master Marathon Monit Log Forwarder Master Node Prometheus cAdvisor HAproxy Marathon-LB Standardization
- 64. Platform Provisioning FLEET PKG REPOSITORY WORKER WORKER WORKER MASTER MASTER
- 65. Platform Provisioning FLEET WORKER WORKER WORKER MASTER …... WORKER WORKER WORKER MASTER …... WORKER WORKER WORKER MASTER …...
- 66. Platform Provisioning FLEET WORKER WORKER WORKER MASTER …... WORKER WORKER WORKER MASTER …... APPLICATIONS SYSTEM MGMT WORKER WORKER WORKER MASTER …...
- 67. & many more
- 68. NOT A LONG AGO BEGINNING OF THE CHANGE THE ROAD AHEAD ADOPTING THE CHANGE
- 69. Change is hard !
- 70. Evaluation Experience in Production Confidence in Adoption Tipping Point Maturity with Devops Timescale
- 71. Our Adoption Playbook Confidence in Technology Compare and Contrast Create new Roles
- 72. Present Present Future Future Unified Deployment Centralized Logging Consolidated Monitoring Common Notifications Evaluation Experience in Production Confidence in Adoption
- 73. Compare and Contrast L4 (load balancer) Dedicated VM App Server Dedicated VM App Server Dedicated VM App Server Marathon LB Marathon LB Mesos Agent Mesos Agent App Container App Container App Container App Container 100%.................50%...................0% 0%.................50%...................100% Launch Stability Confidence Launch Stability Confidence
- 74. New Roles
- 75. OPERATIONS DEVELOPER Create Self Service Automate Primitives Shared Goal with Dev Use Self Service Ops friendly code Shared Goal with Ops
- 76. NEW SHARED GOALS Reduce Time from Code checkin to Production Release Ensure Releases can be performed during normal business hours Reduce unplanned work and increase productivity
- 77. Reality Check Allocate VM to a Service Less upfront capacity allocation meetings, and more work done ! 1 Let Software decide
- 78. Availability and Tolerance = manual mgmt. Let Software decide Less manual intervention, and more time to spend on improving quality 2 Reality Check
- 79. Time to Production influenced by lot of manual monotonous work Minimal Manual work and increased Self-service Ops work made more accessible to Devs via Self Service 3 Reality Check
- 80. Limited reusability and lack of standards across teams Standardize through Containers and Deployment primitives Reusability across teams, and more time to focus on innovation 4 Reality Check
- 81. Less upfront capacity allocation meetings, and more work done ! Less manual intervention, and more time to spend on improving quality Ops work made more accessible to Devs. Ops spend more time improving quality Reusability across teams, and more time to focus on innovation 1 2 3 4 Reality Check
- 82. But DevOps is also about Architecture
- 83. Architecture Constraints Ops friendly development Metrics friendly development Build systems which are failure aware Everything is distributed
- 84. NOT A LONG AGO BEGINNING OF THE CHANGE THE ROAD AHEAD ADOPTING THE CHANGE
- 85. Multi-Tenant CLuster Mix workloads for increased efficiency Share Common platform primitives Performance and Isolation guarantees Avoid Noisy neighbours Mesos Agent Selection Isolated Load balancer and Discovery Isolated Container Registry Resource Reservation
- 86. Global Workload Allocation Not all workloads are same ! C Cost P Performance I Isolation
- 87. Mesos Cluster MULTI-TENANT Physical Machines ( Mesos Agents ) SINGLE-TENANT Physical Machines ( Mesos Agents ) MULTI-TENANT VMs ( Mesos Agents ) SINGLE-TENANT VMs ( Mesos Agents ) CP IP CP CI Global Workload Allocation
- 88. Mesos Cluster Custom FrameworkFenzo Mesos Master Physical Machine ( Mesos Agent ) Physical Machine ( Mesos Agent ) VM ( Mesos Agent ) VM ( Mesos Agent ) Global Workload Allocation
- 89. Custom Framework ● Recommendation System for Resource allocation ● Integrate with Billing system to provide cost-efficient allocation scheme Global Workload Allocation
- 90. Custom Framework ● Recommendation System for Resource allocation ● Integrate with Billing system to provide cost-efficient allocation scheme Global Workload Allocation ● Support more advanced bin-packing with Fenzo #2430 (mesosphere/marathon)
- 91. Application aware scheduling Tenant Allocation Framework Mesos Cluster Mesos Agent Mesos Agent Mesos Agent Mesos Agent Tenant Tenant Tenant Mesos Master
- 92. Application aware scheduling Cost Reduction
- 93. Canary Release and Architecture A/B Mesos Agent Mesos Agent Mesos Agent Mesos Agent Mesos Master Marathon Auto Scaling based on automated workflows Automated Workflows for Testing new versions Self Healing and automated rollback to previous versions
- 94. THE EARLY YEARS BEGINNING OF THE CHANGE THE ROAD AHEAD ADOPTING THE CHANGE
- 95. 1DAY ONE
- 97. Change is possible. Microsoft joins Linux Foundation - November 16, 2016
- 98. We Mesos community ! Thanks 고맙습니다 谢谢 Questions ?
- 99. We Mesos community ! Thanks Want to help us build this further ? We are hiring ! 고맙습니다 谢谢