DevOps Approach (Point of View by Ravi Tadwalkar)
Download as PPTX, PDF44 likes14,198 views
The document discusses a 5-step approach to implementing a DevOps journey: 1) Intake and Planning, 2) Discovery Phase, 3) Roadmap Phase, 4) Pilot Phase, 5) Wider Rollout. It describes each step in detail, covering activities such as defining goals and scope, conducting assessments, creating recommendations and roadmaps, training, and socializing outcomes. The goal is to help organizations improve their DevOps capability maturity over time through this phased approach.
DevOps Approach (Point of View by Ravi Tadwalkar)
- 1. DevOps Approach- People, Process, Tools, Governance and Metrics (From Ravi Tadwalkar’s Point of View) Source: “2015 Annual State of DevOps A GLOBAL PERSPECTIVE ON THE EVOLUTION OF DEVOPS & A BENCHMARK FOR SUCCESS”, Delphix and Gleanster
- 2. DevOps Approach People Process Tools Governance Metrics We do lot of Lean-Agile Coaching and DevOps assessment work with clients around US, and based on that, we have come to the conclusion that implementing DevOps is more than implementing deployment pipeline for a technology stack (such as IBM, Microsoft, SAP, Oracle or Atlassian) based end-to-end tool chain. In fact, tool chain is only one of the 4 critical success factors, the other 3 being people (changing roles), process (waterjile/agilefall (hybrid scrum), pure scrum, scrum at scale, scrumban or Kanban?) and lean governance model (specially around upstream Portfolio Kanban system coupled with downstream Release Engineering Kanban system). What kind of metrics matter for assessing success of DevOps implementation? We will describe these concepts, in this brief slide-deck.
- 3. Step 1: Intake & Planning (upto 1- 2 weeks) •Define Goals -e.g. 1. Visibility at all levels 2. Frequent releases for TTM with quality •Define Scope: early adopters- programs/products •Identify key stakeholders: from Business, Dev/Engg, SQA/Test, Release Engg and/or Operations • Identify landscape: Value chain, initiatives, priorities, expectations, impediments, and known constraints Step 2: Discovery phase (upto 3-4 weeks) •Apply "learning to see” approach to create value by eliminating muda (non-value-add aka NVA waste). Narrow down and validate current state value stream map (VSM)– identify current process workflows, analyze timings and identify NVA waster (impediments) causing delays in flow. Visualize future state VSM- “single piece flow” in ideal DevOps world. •Workshops- to identify gaps in key DevOps tenets related to people, process, tools, governance & metrics •Assessment of DevOps Capability maturity •Formal Report- Submit assessment worksheet highlighting gaps Step 3: Roadmap phase (upto 4-6 weeks) • Recommendations for Agile& Lean adoption with DevOps- •People- Agile and DevOps adoption related new roles •Process -Practices and ceremonies for chosen process •Tool chain for Deployment Pipeline for selected engineering stack •Governance & Metrics • 30-60-90 plan specific to identified gaps in order to improve DevOps Capability Maturity level #Step 4: Pilot Phase • Identify people for Agile & DevOps related Roles from selected teams • Training & Coaching - Process Option 1- “At Cadence” Delivery Option 2- “Continuous” Delivery and - People - changing roles and responsibilities - Tools - tool chain synchronized with changing roles and processes • Build-measure-learn -learn to either pivot or persevere • Validated learning -socialize outcomes and communication plan of continuous improvement #Step 5: Wider Rollout •Move the needle •Create Community of Practice (CoP) -core team & charter •Add champions, evangelist to CoP •Cadence of events •Execute Continuous improvement plan of Agile & DevOps at scale, apply process evaluator tool with a centralized or autonomous governance model: Option 1- revolutionary (scaled) framework e.g. SAFe, DAD, LeSS,… Option 2- evolutionary (phased) framework e.g. Kanban and/or Scrumban DevOps Approach: 5 steps for DevOps Journey #Time line is context dependent after step 3 3
- 4. Tenets Parameters Context Lean-Agile practices Business Requirements & Change Management Agility to accommodate changing requirements Sprint Model Avoid Big Requirements Up Front Self organized teams Needs Leader, not commander Continuous integration (CI) Build automation Stabilize Release Management Code quality metrics Quality Built-In rather than afterthought SW Regression Coverage Reduced Regression ~ Predictability New Test & Automation Automate for Faster Time to Value CI/CD pipeline Management Deployment pipeline ~ Faster Time to Market, with Quality Continuous delivery (CD) Code Promotion Model No more “But it works on my system” Application Deployment Enabler for Business Agility Predictive Monitoring & Analytics Amplify Feedback Loops amongst Business, Dev, QA & Operations CD pipeline management Enabler for Environment Provisioning Agile Testing Test Automation (Unit, Functional, Regression, Performance, Security) Avoid manual testing in order to increase Time to Market, with Quality TDD & Refactoring Productivity requires Reduced Technical Debt Tenets Parameters Context Environment Management Environment provisioning Predictability requires reduced environmental inconsistencies Service virtualization for Dynamic scaling Scalability for Upstream and Downstream “customers” Test data management Reproducible & consistent behavior Prod – non prod environment closeness No surprises for environment refreshes Lean Architecture for DevOps Modular release strategy Frequent Releases SW deployment modularity Reliable Release Calendar Design for resiliency and serviceability Deal with Non-Functional Requirements Early Culture & Org Alignment Org alignment Business Embracing Culture of Collaboration Dev & Ops Collaboration Model Dev, QA & Ops Embracing Culture of Collaboration Team profile Increase Flow with Cross-Functional Teams Operations Resilience & Disaster Recovery Robustness Performance Scalability Testing Enabler for High Availability Preliminary Assessment Requires Holistic Template 4
- 5. DevOps maturity model (Preliminary Assessment Report) Key DevOps Tenets Parameters Context "As Is" state: Consolidate w/ consensus (crawl/walk/run/sprint) Crawl Walk Run Sprint Culture & Alignment Org Alignment Business Embracing Culture of Collaboration C Dev & Ops Collaboration Model Dev, QA & Ops Embracing Culture of Collaboration C Team Profile Increase Flow with Cross-Functional Teams C+ Lean-Agile Practices Biz Requirements & Change Management Agility to accommodate changing requirements W Scrum/Kanban/Scrumban Model Avoid Big Requirements Up Front C+ Self organized teams Needs Leader, not commander W CI (Continuous Integration) Build Automation Stabilize Release Management R Code Quality Metrics Quality Built-In rather than afterthought W SW Regression Coverage Reduced Regression ~ Predictability C- New Test & Automation Automate for Faster Time to Value W CI/CD pipeline Management Deployment pipeline ~ Faster Time to Market, with Quality C CD (Continuous Delivery) Code Promotion Model No more “But it works on my system” C Application Deployment Enabler for Business Agility C Predictive Monitoring & Analytics Amplify Feedback Loops amongst Business, Dev, QA & Operations W CD Pipeline Management Enabler for Environment Provisioning C- Agile Testing Test Automation Avoid manual testing in order to increase TTM, with Quality C TDD & Refactoring Productivity requires Reduced Technical Debt C Environment Management Environment Provisioning Predictability requires reduced environmental inconsistencies C+ Service Visualization for Dynamic Scaling Scalability for Upstream and Downstream “customers” W Test Data Management Reproducible & consistent behavior C+ Prod & non-prod Environments Closeness No surprises for environment refreshes W Lean Architecture for DevOps Modular Release Strategy Frequent Releases C SW Deployment Modularity Reliable Release Calendar C Design for Resiliency & Serviceability Deal with Non-Functional Requirements Early C Operations Resilience & Disaster Recovery Robustness W Perf Scalability Testing Enabler for High Availability W As Is To Be 5
- 6. “As-Is” Value Stream Map – Illustrating Kaizen Lightening Bursts Customer Plan (Idea / Discovery) Build (Delivery) Run (Operations) Business Case PlanningPrioritization - Ideation - Biz & Tech Assessment - Go/No-go/Defer - Capacity Planning - Project Planning - Prioritization - Project Selection - RFE Approval Design (Arch) Build (Dev) Test (QA) - Technical Design - WBS - Integrate - Test Planning - Ops Planning - Get Approval - Deploy to QA - Configure Tests - Manual Testing - User Acceptance Production Go-Live Business-As-Usual Change Mgmt (ITIL) Environment Provisioning- Ops Planning - Config&Deploy - Wait for DBA - Get Approval Prod Support & Monitoring Ops Mgmt (DB, Network) 3 to 5 m 4 to 6 m 2 to 3 m Sustenance mode Business Portfolio: Governance & Oversight Environments & Tools Time Ladder (approx.) Lead Time ? Value Add Time ? Non-VA Time ? Electronic Information Flow Offshoring Teams 0.5 m 1 m 1 m 1 m 1-3 m 0.5 m 2 m 0.5 m 0.5 m 0.5 m 1-2 m 1-2 m 1-2 m 1 m WIP (Inventory)PUSH- scheduled before neededExternal source LEGEND Eliminate Waste Time Analysis Non-Value Add vs. Total 45% Value Add vs. Total 55% Value Add vs. Non-VA ? Sustenance 6
- 7. AS Is State of “Agile and DevOps” Permit to Build Permit to Operate No visibility of portfolio, program and teams Need PO for team Architect overloaded as Scrum Master; PM role unclear Dev and Ops work in silos Agile process practices only for execution Waterfall still preferred for multi-site planning, Dev, & operations Atlassian tool chain not fully integrated with Amazon Cloud infrastructure Big Design Up Front (BDUF) approach “Agile Testing” is done very late in sprint Teams “inspect & adapt” via demo feedbacks, but deployment not at sustainable pace In onsite-offshore model, PO, Dev & Testers mostly work in silos, when scrumming during sprint. Need MVD (min viable documentation) template with Cucumber, Jira and Confluence Need centralized OMS for “pre-prod” i.e. production-like environment provisioning & data refresh, wherever feasible Build Automation exists with Jenkins; Test Automation: start w/ Cucumber; Release Automation is in its infancy. Customer demo and feedback happens frequently, but no interim release cadence hampers that communication. Environment provisioning and cost is not a challenge with Amazon Cloud; but database setup still a challenge. Pre-prod not always feasible due to: Distributed digital assets Student privacy concerns Long Release Cycle: features deployed twice/year (seasonality) Big-bang release demo for users (True even for pilots, no “fail fast”) OMS can be challenge to move from 6 weeks internal release to monthly release cadence Manual deployment- error prone Seasonality a big factor in maintaining status-quo on environment mismatch 7
- 8. DevOps maturity model (Goal Setting for 30-60-90 Plan) Key DevOps Tenets Parameters Context Normalized state transition "As Is" -> “To Be”: Consolidate w/ consensus (crawl/walk/run/sprint) Crawl Walk Run Sprint Culture & Alignment Org Alignment Business Embracing Culture of Collaboration C -> W Dev & Ops Collaboration Model Dev, QA & Ops Embracing Culture of Collaboration C -> W Team Profile Increase Flow with Cross-Functional Teams C+ -> W Lean-Agile Practices Biz Requirements & Change Management Agility to accommodate changing requirements W Scrum/Kanban/Scrumban Model Avoid Big Requirements Up Front C+ -> W Self organized teams Needs Leader, not commander W CI (Continuous Integration) Build Automation Stabilize Release Management R (better!) Code Quality Metrics Quality Built-In rather than afterthought W SW Regression Coverage Reduced Regression ~ Predictability C- -> W New Test & Automation Automate for Faster Time to Value W CI/CD pipeline Management Deployment pipeline ~ Faster Time to Market, with Quality C -> W CD (Continuous Delivery) Code Promotion Model No more “But it works on my system” C -> W Application Deployment Enabler for Business Agility C -> W Predictive Monitoring & Analytics Amplify Feedback Loops amongst Business, Dev, QA & Operations W CD Pipeline Management Enabler for Environment Provisioning C- -> W Agile Testing Test Automation Avoid manual testing in order to increase TTM, with Quality C -> W TDD & Refactoring Productivity requires Reduced Technical Debt C -> W Environment Management Environment Provisioning Predictability requires reduced environmental inconsistencies C+ -> W Service Visualization for Dynamic Scaling Scalability for Upstream and Downstream “customers” W Test Data Management Reproducible & consistent behavior C+ -> W Prod & non-prod Environments Closeness No surprises for environment refreshes W Lean Architecture for DevOps Modular Release Strategy Frequent Releases C -> W SW Deployment Modularity Reliable Release Calendar C -> W Design for Resiliency & Serviceability Deal with Non-Functional Requirements Early C -> W Operations Resilience & Disaster Recovery Robustness W Perf Scalability Testing Enabler for High Availability W As Is To Be 8
- 9. “To-Be” Value Stream Map – Seek “Build” Process Improvements Customer Plan(A “Release”) Build(Deployables) Run (Operations) TDAD Multi- Sprint Plan Backlog Prep <Story Map> - Technical Design - Multi-team - At cadence - Ops work - 1 story map - per hour - per epic Design (Artifacts) Sprint Plan + Sprint Retro Iterate (Dev & QA) - MVD (wiki) - Test Plans - UX artifact - 2 mtgs, 4hr each - Ops work included - Build w/ Unit Tests - Execute test cases - Expedite bug-fixes - Dev & Test NFRs Demo Business-As-Usual Change Mgmt (ITIL) Environment Provisioning- Env on demand - PO Acceptance - Production bugs - Proactive DBAs Prod Support & Monitoring Ops Mgmt (DB, Network) 3m 3m (4 sprints) 1m Sustenance mode Business Portfolio: Governance & Oversight Environments & Tools Time Ladder (approx.) Lead Time ~4m Value Add Time ~2m Non-VA Time ~2m Time Analysis Non-Value Add vs. Total 50% Value Add vs. Total 50% Value Add vs. Non-VA 1 Electronic Information Flow Offshoring Teams 2m 1w 1w .5 w NVA:1d 1m 0.5 w 1w.5 w 2w NVA:1d 1w WIP (Inventory)PUSH- scheduled before neededExternal source LEGEND Eliminate Waste 1w 3m (4 sprints) Sustenance 9
- 10. To Be State of “Agile and DevOps” Permit to Build Permit to Operate Visualize work with Kanban: plan-build-run Active Product Management: match funding & seasonality? Scaled Agile: Conduct multi- sprint planning with user story map & feature planning board Common Release and sprint cadence for teams Defined Roles & Responsibilities for SM, PO, Program Mgr & Product Mgr Evangelize with communities using open space events Get into Scrumming Kata (e.g. retrospectives ) and Kanban Kata for continuous improvement PO be accountable for release & sprint goals TDD & BDD: Use frameworks for development, rather than after-the-fact during testing Improve cycle time in each stage of tool chain for better overall lead time: e.g. pre-commit level code review rather than post-commit Version everything: code, metadata, databases, build artifacts… Test automation implemented & relevant for execution *during* sprint Integration, Regression, Load, performance, security and acceptance testing is not a hardening event Build-Measure-Learn: leverage analytics where possible Validated Learning: Conduct experiments to evangelize “fail fast” mindset: pivot or persevere Visualize deployment of releases with Release Engineering Kanban Release Automation: Release and Deployment management is automated Production Monitoring and incident management automation Management 3.0 tools for leaders: Knowledge management for team building and developing generalists out of specialists 10
- 11. 11 People: DevOps extends Product Owner & Scrum/Flow Master Roles DevOps extends Product Owner & Scrum Master roles with additional responsibilities. DevOps also adds higher level “Agile Product Manger (aka Business Owner)” and “Agile Program Manager (aka Release Manager)” roles. Role Responsibilities Role Responsibilities Agile Product Manager aka Business Owner •Org Alignment during Portfolio-level Planning: •Take ownership of delivering business capabilities based on business priorities •Participate in pre-planning activities e.g. envisioning workshops for business goal, vision & roadmap •Share business objectives with key stakeholders e.g. PMO, Business Management, and System Architects •Communicate business capabilities for next release planning aka multi-team or multi-sprint planning •Org Alignment during Multi-team or Multi-sprint Planning: •Take ownership of delivering business capabilities based on total (multi-)team capacity •Participate in driving release planning during sprint 0 •Participate in management review for scope adjustment, problem-solving & improvisations and compromises •Participate in leadership level obstacle escalation & removal in order to act on obstacles & dysfunctions •Org Alignment during Release Management & Deployment: •Participate in providing feedback during demos Agile Program Manager aka Release Manager •Org Alignment during Portfolio-level Planning: •Collaborate with leadership (from Biz, Dev, QA & Ops) to support release cadence •Org Alignment during Multi-team or Multi-sprint Planning & Execution: •Take ownership and responsibility of Multi-team or Multi-sprint release planning •Facilitate release planning aka multi-team or multi-sprint planning event •Define & maintain release plans which include content, dates & success criteria •Manage release dependencies across applications, products and/or components •Define & measure metrics to “inspect & adapt” for continuous improvement •Facilitate change management •Org Alignment during Release Management & Deployment: •Take ownership and responsibility of release management •Lead cross-functional teams (Biz, Dev, QA & Ops) to support release variability •Coordinate with business owners, product owners, scrum masters & Ops team •Define and manage release readiness •Ensure deployment automation / continuous delivery setup •Execute a low-fidelity obstacle escalation & removal process •Define release types, associated workflows & approval requirements for artifacts •Provide oversight for deployments into production for Service Delivery •Guide the completion of root cause analysis to ensure issues do not recur/regress Product Owner •Responsible for aligning Operations & Development on business goal, vision & roadmap •Consider operations needs as input to prioritization •Create product backlog during sprint 0; maintain it thereafter Scrum Master aka Flow Master •Facilitate collaboration (Dev<->Ops) •Ensure impediment free execution by Dev and Operations team •Ensure adoption of agile practices & ceremonies, DevOps processes & tools •Facilitate cross-training of team members and rotating the roles of team members
- 12. 12 People: DevOps Extends Agile Team Member Role Aspect of Role DevOps Specific Core Responsibilities Dev • Collaborate with Operations to decide on Sprint goal and commit to deliverables • Collaborate with Operations to identify environment configuration & provisioning requirements • Resolving recurring Ops issues • Understanding and developing as per Ops requirements • Understand Ops tools & processes • Feature development and testing + • Continuous testing • Test automation • Test data management • Data deploy automation • Performing Ops-like testing for Ops-like metrics ( e.g. security) + • Collaborate with Development to detail out Operations specific needs • Provision & Configure environments as per the needs identified jointly with development team • Understand Dev tools & processes • Maintain continuous delivery setup & help with troubleshooting • Debug & fix environment specific issues • Contribute to development using available bandwidth • Monitor continuous delivery setup and help fix issues slowing down the progress Test Ops
- 13. From “Plan->Build->Run” perspective, planned releases & sprints are executed using iterative & incremental delivery workflow depicted here. Each delivery cycle consists of these steps. Note that the Sprint cycle (steps 2 to 5) is repeated for all sprints in each release: 1. Release Prep • Portfolio Kanban System: • Agile Product Manager brings visibility to “product” feature set for initiative(s) identified, by shaping demand via portfolio backlog. • Agile Product Manager brings visibility to features (including tech debt items), by auditing each feature on Portfolio Kanban system. • Product Owner writes user stories based on User Story Maps for each feature. • Release Planning meeting: • Agile Product Manager drives the Release Planning event that marks the end of “sprint 0”, a time-box used for purpose of planning. • Agile Program Manager facilitates creation of Release Plan, emphasizing that release plan is an estimate/forecast, not commitment. • Activities: Epic & tech debt intake, story mapping, create user stories & related documents, and create Feature Planning Board. • Agile Program Manager correlates feature-level forecast from the Release Plan with sprint-level commitments during entire release. 2. Sprint Startup • Sprint Planning meeting: On day 1 of Sprint, scrum master facilitates Spring Planning meeting to create Sprint Plan WIP-limited by available team capacity. 3. Sprint Execution- Delivery • Sprint Execution: After Sprint Planning meeting, scrum master facilitates Spring execution, by removing impediments faced by team during sprint. • Mid-Sprint Backlog Grooming meeting: In this meeting, held around mid-point during the sprint duration, Product Owner gives a glimpse of stories & defects for upcoming sprint, so that team can do any course correction needed to attain current and future sprint goals. 4. Sprint End- Acceptance • Sprint Demo and Retrospective meetings: On the last day of Sprint, scrum master facilitates Sprint Demo or System Demo meeting, followed by team-only Sprint Retrospective. • High performing teams may conduct current-sprint demo and next-sprint planning during same meeting, followed by retro next day. 5. Deploy to Production • Release Engineering Kanban System: • Agile Program Manager uses feature deployment status on Release Engineering Kanban system to validate done-ness of deployment artifacts related to work-items, and to relay feedback to team(s) on issues during each deployment. • Agile Product Manager provides feedback so that executive leadership can make go/no-go/defer decision based on that feedback. Process: Option 1: “At Cadence” Delivery with Frequent Releases Release Prep (Portfolio Kanban System) Deploy to Production (Release Engineering Kanban System) Sprint Execution- Delivery Sprint End- Acceptance Sprint Startup 13
- 14. From “Plan->Build->Run” perspective, planned releases are executed using iterative and/or incremental delivery workflow depicted here. Each delivery cycle consists of these steps. Note that the Kanban/Scrumban cycle (steps 2 to 5) is repeated for all releases until the ship date: 1. Release Prep • Portfolio Kanban System: • Agile Product Manager brings visibility to “product” feature set for initiative(s) identified, by shaping demand via portfolio backlog. • Agile Product Manager brings visibility to features (including tech debt items), by auditing each feature on Portfolio Kanban system. • Product Owner writes user stories based on User Story Maps for each feature. 2. Release Planning meeting • Agile Product Manager drives the Release Planning event that marks the end of “sprint 0”, a time-box used for purpose of planning. • Agile Program Manager facilitates creation of Release Plan, emphasizing that release plan is an estimate/forecast, not commitment. • Activities: Epic & tech debt intake, story mapping, create user stories & related documents, and create Feature Planning Board. • Agile Program Manager correlates feature-level forecast from the Release Plan with sprint-level commitments during entire release. 3. Kanban/Scrumban in the middle • Execution: “Focus on the work, not the worker!” After Planning meeting, “Flow Master” facilitates execution for single-piece flow, removing impediments faced by team during release. • System Capability Review (aka Input Queue Replenishment) meeting: In this meeting, held at cadence based on lead time stats, Product Manager and Product Owner – both - give a glimpse of work remaining, so that team can do any course correction needed to attain current and future release goals. 4. Continuous Improvement (4 Kaizen events) • Feedback Loops: At regular cadence, based on increments of continuous delivery- at least once a month- management facilitates Kaizen meeting, using Kanban Depth framework. Over 4 such Kaizen events, team & management observe areas for continuous improvement in structured way. • High performing teams may conduct such Kaizen events right after incremental or end-to-end system demo & Scrum-style retrospectives. 5. Deploy to Production • Release Engineering Kanban System: • Agile Program Manager uses feature deployment status on Release Engineering Kanban system to validate done-ness of deployment artifacts related to work-items, and to relay feedback to team(s) on issues during each deployment. • Agile Product Manager provides feedback so that executive leadership can make go/no-go/defer decision based on that feedback. Process: Option 2: “Continuous” Delivery with Frequent Releases Release Prep (Portfolio Kanban System) Deploy to Production (Release Engineering Kanban System) Kanban/Scrumban in the middle Continuous Improvement Release Planning 14
- 15. Agile and DevOps RACI matrix* * This is only an example. Modify based on your org structure and needs! 4 main roles: •Product Owner (PO) •Scrum Master (SM) •Scrum team member •Ops team member Other pertinent roles: •Proxy-PO •User Experience Lead •Architect •Agile Program Manger •Functional Manager •Agile Product Manager Process Step -> ----------------------------------- ExistingNew Role Portfolio Kanban System Release Planning Sprint Start- Kick-off Sprint Execution - Delivery Sprint End- Acceptance Release Engineering Kanban System Agile Product Manager R R I C C I Product Owner C R R R A I Systems Architect A C C C C C Team Lead C C C R R C Functional Manager C C I C I I Agile Program Manager I A R C I I Scrum Master I R A C I I Scrum Team Member I C C A C C Ops Team Member I C C R C A Ops Manager I I I C C R 15 15
- 16. 16 Process: Scrumban Process Framework for DevOps • Scrumban is our recommended process framework for DevOps. • Dev and Ops are part of a single team following Scrumban and effectively handling Dev and Ops user stories. • Dev and Ops are on a single team. • Alternatively Dev and Ops will be on one team, supported by (horizontal) Central Ops team. • Team has single product owner who owns both Dev and Ops requirements in single product backlog and has authority to prioritize between Dev and Ops work items • Based on history of unplanned work items, effort division is considered. E.g. 80% planned work (including Dev and Ops) and 20% unplanned work is expected. In this example, let us assume- • 70% Dev efforts would be required • 30% Ops efforts would be required • Out of 30% Ops efforts, 10% may be planned and 20% would be unplanned work. • Team will plan for 70% of planned Dev work as part of scrum (it could be less, depending on buffer). • Team will plan for 10% planned Ops work as part of scrum. • Planned work (both Dev and Ops work items) will be stack ranked i.e. prioritized for sequencing. • Remaining 20% bandwidth is reserved for Kanban– WIP limits to funnel the flow of unplanned work e.g. WIP limiting the expedited severity 1 issues
- 17. 17 Process: How to use Scrumban Process Framework for DevOps • From Day 1 of sprint until unplanned high priority incident is raised, team works on prioritized user stories. • As soon as an unplanned severity 1 incident is raised, priority is set as 0 for this incident. • A designated or team member having expertise in that area takes up this work item, while deprioritizing the current planned work item. • Once the unplanned incident is completed and delivered for release, this team member resumes work on the planned work item. • If the volume of unplanned work items becomes higher than 20%, PO ensures that some stories are deprioritized and impact on planned work is acceptable by the key stakeholders. • Similarly, if there are less unplanned incidents, the available bandwidth is used to complete any planned Dev and Ops user stories. • It’s a continuous process to observe and control flow of unplanned work (with WIP limits) and accordingly adjust the planned vs unplanned distribution for upcoming sprints to take appropriate number of work items in scope. • Due to single team and single product backlog, it is easier to ensure both Dev and Ops work on agreed priority and have common goal of delivering business value during each sprint. • Team can initially start with separate Dev and Ops resources; and slowly learn complementary skills. • Single PO helps in reducing Dev and Ops conflicts. • Following scrum ceremonies ensures that best practices like retrospectives are followed in DevOps. • Following Kanban ensures that due attention is given to high priority items and are released with no constraints of aligning with remaining Dev items. Execution Exceptions Benefits
- 18. 18 Process: Key Considerations for Adopting Process Frameworks with DevOps Dev teams need to start thinking beyond feature development. Ops teams need to start thinking beyond infra support. What are availability, capacity, security, service levels and service continuity requirements? What is the plan to meet those? What infra capacity is required to achieve operational goals and sustain it? What is the impact on ongoing operations? What constraints does it put on design and development? What are the requirements to manage operational issues & support? (scripts, alerts, logs, trails…) How are knowledge sharing requirements? How to enable it through documentations, cross-silo and trainings? How to do incremental validation of build quality for production release? How to ensure production release readiness? Is there plan for proactive detection and escalation of possible failures? How to implement plan to enable continuous monitoring of applications with metrics, alerts, etc.? How to avoid future issues by quick resolution of service interruptions and root cause analysis? How to provide feedback into development for improvement in specific operational aspects? How to manage changes in requirements for impact on operational goals? How to ensure that appropriate runtime infra setup is available? How to validate operational goals against evolving design? How to standardize release & deployment plan? How and what deployment & release artifacts should be made available?
- 19. • Adopt a correct team structure like single DevOps team or single team with Central Ops • Adopt a right process which can cater to both Dev and Ops needs, e.g. Scrumban • Have an integrated product backlog for better prioritization among Dev and Ops work items • Have a single product owner (PO) to manage integrated or otherwise both Dev and Ops backlog. PO should have authority to resolve dependencies and conflicts • Ensuring PO has clear visibility of both Dev and Ops work, and is accountable for business value delivered • Understand the need of DevOps teams to work on high severity items which demands them to stop work on existing work items. Such context switching may impact throughput. DevOps teams should be measured on reduced change lead time, not velocity/productivity • Focus on tools and technologies which enable easier release management considering needs of DevOps teams to deploy multiple times while still continuing development of planned work 19 Process- Key Success Criteria
- 20. End to End Atlassian Tool Chain for DevOps with SAFe Constructs Development Tools Atlassian BitBucket (GIT) Developer writes code and check-in Associate JIRA Issue Types (Tasks / Bugs) Build gets triggered Microsoft Visual Studio (For .NET) Eclipse (For JAVA) Atlassian BAMBOO for CI Build Definitions for: Database .NET Java Atlassian BAMBOO for Continuous Deployment (CD) Approve / Reject Build DIT1 Functional Testing • Continuous Integration • Scheduled Build Approve / Reject Build DIT2 Approve / Reject Build SIT Regression Testing Selenium Approve / Reject Build UAT UAT Testing Approve / Reject Build PROD Functional Testing DEVELOPMENT Branch QA Branch MASTER Branch FEATURE Branch Service & Data Mock Epics e.g. Ship to Home Pre-pay (JIRA Issue Type) User Stories (JIRA Issue Type) Monitoring Ngaios Initiatives e.g. Sports on Spectrum (JIRA Issue Type) Release Stream aka Agile Release Train (ART) Themes (Strategic Goals) Test Planning Test Execution Traceability & Reporting Defect Creation in JIRA Track Coverage & Quality Metrics UX Team DB Team ART Engineer Product Managers Note: Integrate Atlassian Bamboo with “Chef” for non-windows deployments DevelopmentTasks Visualize Portfolio Timeline by Releases EasyBI plugin (Reporting & Charting) On top of existing JQL queries Visualize Investment by Themes PMO-level Status Reporting Testing Tasks 20 Kanplan: backlog for Kanban and Scrumban teams 20
- 21. During planning, create org alignment to improve flow (1st way of DevOps), using Portfolio Kanban system. 21 Lean Governance Model- Applying 3 ways of DevOps Enable managers in shaping various communities of practice: create culture of continual experimentation & learning (3rd way of DevOps). • Begin with succinctly defined portfolio hierarchy or swim lanes that consist of strategic (CAPEX) and tactical (OPEX) investments: • Themes • Initiatives • Features • Create single-piece flow of portfolio items with Portfolio Kanban board. Apply lean metrics for continuous improvement. • Apply first way of DevOps: emphasize on performance of entire system by analyzing value streams, WIP, lead times & due date performance. During execution, enable continuous feedback loops (2nd way of DevOps) with obstacle boards to act on impediments & dysfunctions. • This requires low-fidelity obstacle escalation process. This can be a 2-level (team->management) process, or a 3-level (team->program>portfolio) process. • Example of obstacle board and related workflow:
- 22. Upstream Kanban: Downstream Kanban: 22 • Begin with the end in mind! • Use Portfolio Kanban board across Org Design (Team of Teams) • Use a simple Kanban board with “ToDo -> Doing -> Done”, “Plan->Build->Run” or similar workflow • Visualize portfolio feature level bottlenecks to "Stop Starting, Start Finishing"! • Supplement Portfolio Kanban system (“Upstream Kanban”) with Release Engineering Kanban system (“Downstream Kanban”) Run F E I Idea Deploy to Prod G D GY PB MN 2 ∞ AB Biz Design Plan Build (Dev & QA) Architecture, resource plan Core team OMS … 3 3 DE Explicit policy: Feature is “done” only when last team working on it has completed “Deploy to Prod” Lead time ends at 1st infinite queue! There could be additional queues with no WIP limit. Lead time “clock” starts at 1st queue i.e. WIP limited input queue! 3 3 Deployed (Note touch time) NPD (New Prod Dev) (60%) Defect& Tech Debt (30%) Innovation (10%) 5 2 1 Expedite 1 3 Fixed Date Standard Intangible 2 2 3 WIP Limits based on classes of service (Release Engineering) WIP Limits based on work types (feature teams) Lean Portfolio Management and Release Engineering
- 23. 0 20 40 60 80 100 120 140 160 4851 4 6 8 101214182022242628303234363840424446485052 1 3 5 7 9 111315171921232527293133353739 2012 2013 2014 Throughput (# of resolved tickets / week) Average Lead Time / week Average Flow Efficiency % / week Metrics: Lean Metrics for DevOps Teams * Lead Time Distribution indicates Predictability * Average Lead Time impacts SLA Adherence * High Productivity means better “Flow Efficiency” * Monte-Carlo forecasting for planning fixed bid projects How do you coach a kanban team on adjusting their "Total WIP“? By observing trends in throughput, lead time & flow efficiencies! Apply Little’s law: Total WIP (for each workday) = Avg. delivery rate (per workday) * Avg. lead time (in workdays) 23
- 24. Key Capabilities- Begin with End in Mind Desired Capabilities (Work Streams) Description Reduced Change Lead Time DevOps Ecosystem Visibility - people, process & tools Quality Built-In Build & Test Automation Zero Manual Intervention (single click deploys) Release Automation Zero config/env level dependencies Agile Testing: Code commit should include TDD, BDD & regression tests. Enable a fully integrated deployment pipeline, integrating code & tests with quality metrics. Infrastructure as a Code: Ability to virtualize services and third party systems to enable testing without any dependency on config/env. Multiple environments can spin up and down to support multiple deployments. CI & CD Implementation: Code commit in CI-CD pipeline should automatically trigger test scripts & notify results for each build and deployment. Mechanism to automated deploy the successful build in System Test (aka pre-prod) environment. End to end requirements traceability Socialize DevOps approach (with training & coaching) : Approach that visualizes DevOps Ecosystem with tool chain synchronized from portfolio to deployment. It integrates People (changing roles) into Process (“no” 1-size-fits-all). It deals with kaizen bursts from “To Be” value stream map. Our experience says • At least 10~15% reduction in configuration management defects and ensures traceability • 30~40 reduction in effort & time for deployment and re-deployment thereby allowing more time to work on more user stories • Elapsed time from code check-in to testing to reduce by 35% ~ 45% • Improve early detection of defects at least by 30% leading to Reduction in cost of rework by 20% DevOps Goal #1: Visibility at all levels DevOps Goal #2: Frequent Releases 24
- 25. Prioritization of Work Streams: Value Management Ease of Implementation HighMedLow LowMedHigh ValuetoBusiness ID Work Streams WS01 DevOps Ecosystem Visibility- people, process and tools WS02 Quality Built-In WS03 Build & Test Automation WS04 Release Automation ID Quick Wins QW01 Lean/Agile/DevOps Training (WS01) QW02 Lean Portfolio & Operations (WS01) QW03 Migrate from manual tests to BDD (WS02) QW04 MVD & (proactive) Code Reviews (WS02) QW05 Builds integrated with xUnit & BDD (WS03) QW06 Experiment with “Infra as Code” (WS04) Desired Capabilities (Work Streams) WS02 QW01 WS01 WS03 QW05 WS04 QW04 Top 6 Quick Wins QW03 QW06 QW02 25
- 26. DevOps implementation roadmap Quality Built-In DevOps Ecosystem Visibility Release Automation Build Automation Test Automation Training: Release Planning (Mock Training) Visibility 3~4 Months 4~5 Months Tools Standardization & Rationalization Build CI Pipeline (Proof of Concept) Build CD Pipeline (Proof of Concept) Integrate Builds with CI Integrate Dev Testing with CI Automate Deployments to Dev & QA Environments Deploy To System-Test/Pre-Prod Decide on tools & plugins for Agile Testing: Unit test frameworks, Integration testing w/ UX & service layers, Behavioral & Regression testing Define Test Automation Strategy Integrate Existing Test Cases with pre-defined “Acceptance Criteria” within CI/CD Pipeline Define “Infra as Code” Strategy Allocate infra & implement tools Prepare configuration files (e.g. Puppet manifests) and Pilot for a Project Integrate With CI/CD Pipeline QW01 QW02 QW04 Monitor, Review & Improve Test Automation Framework: POC & Rollout Extend to Program WS02 Integrated tool chain for Functional and NFR (Non- Functional Requirements) 5~6 Months Training: Kanban basics (GetKanban Game) Lean/Agile/DevOps Training for teams, leadership, product, project, program, & functional management Training: Advanced Kanban (2 workshops- team process selection & lean metrics) Training: CI, Agile Testing- Concepts, Tools & Plugins Training: CD, Automating Infra w/ Code, Atlassian tool chain, Amazon cloud infrastructure Are We There Yet? About to Walk Ready? You Are Here! Crawling now Inspect & Adapt #1 Inspect & Adapt #2 Inspect & Adapt #3 Tactical Strategic Review current landscape around People, Process & Tools using formal assessments, leadership team feedback & informal input via open spaces for course correction Start Release Planning (Scrum In the middle) Create Kanban teams (Kanban In the middle) Scrumban for DevOps (Scrumban In middle) Start Operations (Downstream) Kanban Manage Portfolio w/ Kanban Manage Operations w/ Kanban Start Portfolio (Upstream) Kanban Review current Governance & Metrics Enable Changing Roles & Responsibilities (multi-site) Training Enable Lean Governance Model w/ Lean Metrics 1~2 Months QW03 Quality WS03 QW05 Build & Test Automation WS04 QW06 Release Automation 2~3 Months 26
- 27. Creating DevOps Culture with Agile, Lean, Lean Startup and TQM Top down approach: 30-60-90 strategic plan for continuous improvement with baseline assessment of DevOps capability maturity Bottom up approach: Crowd sourcing tactical plan for continuous improvement with assessment based open space events Inside out approach: PDCA Kanban board for feedback driven visibility with Lean Startup method (build-measure-learn loop) and TQM (PDCA loop) Implementing DevOps requires mindset & culture that is open to all 3 approaches!
Editor's Notes
- #15: Flow Master: (these are facilitators and certainly not of this kind: https://www.google.com/search?q=Flow+Masters&tbm=isch&tbo=u&source=univ&sa=X&ved=0ahUKEwi8mImmwbnNAhUSfVIKHbNmD7EQsAQIRw)
- #16: RACI: Responsible, Accountable, Consultant, Informed
- #24: There are 3 primary metrics in kanban: average lead time, due date performance, and flow efficiency. There is growing trend of using Monte-Carlo simulation based to forecast delivery dates for planning fixed bid projects.
- #26: Speak on Cloud
- #28: Implementing DevOps with agile, lean, lean startup and TQM 1. Top down approach: 30-60-90 planning 2. Bottom up approach: crowd sourcing with agile cafe event 3. Inside out approach: continuous improvement Kanban board with build-measure-learn and PDCA loops