How to address operational aspects effectively with Agile practices - Matthew Skelton - Agile In The City 2015
4 likes2,460 views
Treating operational aspects of software as 'non-functional requirements' and 'an Ops problem' rather than a core part of the software product leads to poor live service and unexplained errors in Production. Traceability, deployability, recoverability, diagnosability, monitorability, and high quality logging are key features of a software system, along with user-visible features surfaced via the UI, or a capability of an API endpoint. However, many Product Owners understandably feel uneasy about taking on the (necessary) responsibility for prioritising operational features alongside user-visible and API features. This session brings Scrum Masters and Product Owners up to speed on operational features and covers proven practices for improving operability in an Agile context, empowering Product Owners to make effective prioritisation choices about all kinds of product features, whether user-visible or operational.
![“How does [the use of] this tool
help people to collaborate*?”
* Work together, at the same
keyboard/screen](https://image.slidesharecdn.com/addressing-operational-aspects-within-agile-agileinthecity-skeltonthatcher-151120165339-lva1-app6891/85/How-to-address-operational-aspects-effectively-with-Agile-practices-Matthew-Skelton-Agile-In-The-City-2015-141-320.jpg)
Ad
More Related Content
What's hot (20)
Similar to How to address operational aspects effectively with Agile practices - Matthew Skelton - Agile In The City 2015 (20)
Ad
More from Skelton Thatcher Consulting Ltd (20)
Ad
Recently uploaded (20)
How to address operational aspects effectively with Agile practices - Matthew Skelton - Agile In The City 2015
- 1. How to address operational aspects effectively with Agile practices Agile in the City – 20th November 2015 #agileinthecity Matthew Skelton Skelton Thatcher Consulting @matthewpskelton
- 2. “Operational Features” how to develop and test prioritisation techniques collaboration approaches
- 3. availability is the best feature
- 4. transforming technology and teams Cloud, Agile, DevOps high impact expertise
- 7. “the properties of a system which make it work well in Production”
- 8. Not PIMP MY RIDE MORE Greasy Mechanic
- 9. Not PIMP MY RIDE MORE Greasy Mechanic
- 10. Terminology
- 11. what happened to NFRs? (non-functional requirements)
- 13. language impact
- 15. nonsense !
- 18. How did we get to this?
- 19. admission: IT folk have been guilty of making operational features quite scary & mysterious
- 21. long lists of requirements crazy test plans
- 22. poor explanation of needs failure to engage stakeholders gold-plating
- 25. better approach pragmatic and effective rapid, safe, valuable
- 26. “the properties of a system which make it work well in Production”
- 31. non-linear increase in complexity and problems
- 33. we can no longer deal manually with the scale/volume of potential problems
- 35. agility and response to incidents
- 37. remote car hacking: security as an operational feature
- 38. “We have ‘cloud’ now” (HA + DR + Backup + Metrics + Diagnostics + …)
- 39. think: "when it fails, how will we recover?“ it will fail
- 40. How do we develop and test Operational Features?
- 41. defined features testable and measurable
- 42. ahead lie the ‘ilities’...
- 44. 1. What 2. How to test
- 49. Configurability
- 51. re-read config (SIGHUP) text files in version control inject settings – no ‘black boxes’
- 53. toggle features via config “Postcode lookup unavailable” better UX
- 54. Deployability
- 56. immutable artefacts concurrent releases (SxS) symlinks
- 58. Maintainability
- 60. holding page as MVP!
- 61. live system component diagrams
- 62. modularity ability to upgrade version numbering (SemVer?)
- 66. Testability
- 68. every component has a /health endpoint
- 70. Recoverability
- 71. asynchronous service start expect services to be erroring logs are not wiped (rotated: okay) avoid flooding logs
- 73. no nasty zombies after failures MTTR more important than MTBF* * for most kinds of F
- 74. Performance
- 76. run key 'hotspot' areas early use a deployment pipeline ‘critical path’
- 78. early pipeline tests act as a barometer for later performance problems
- 80. derive transit time metrics
- 81. Monitorability
- 83. stream of metrics transaction tracing
- 84. Resilience
- 86. assume missing or failing Chaos Monkey don’t crash on HTTP 503
- 88. Scalability
- 90. concurrent workers queues and bottlenecks throttling is your friend
- 94. securability by practice SSL certs & HEARTBLEED
- 96. Availability
- 98. “available but unusable" synthetic transactions
- 99. special HTTP header: trigger additional metrics/reporting
- 100. How the organisation affects Operational Features
- 101. Budgets
- 102. bonuses: story points delivered tickets closed
- 103. Capex vs Opex tax breaks
- 104. avoiding the Capex/Opex evil
- 105. Developers seen as more valuable than Ops people 3x hiring bonus for Devs (!)
- 106. improved awareness in product teams
- 108. share ownership and decision making
- 112. Product Owner on call for incidents
- 113. tricky! high degree of maturity honesty about the product
- 114. Product Owner and Tech Lead are both on the hook for outages
- 116. 15-30% ‘tax’ on product budget for operational aspects
- 117. AVOID ‘user features’ always taking precedence over ‘operational features’
- 118. How to evaluate Operational Features vs User Features
- 119. treat Ops team folk as another user persona
- 121. alternatives to User Stories?
- 122. NOT: "as a logging subsystem, I want..."
- 123. Metrics
- 124. Live: downtime, A/B for operational aspects (speed) Pre-live: time spent re-deploying
- 126. metric-ify your delivery and test infrastructure 99.99% uptime, but 20 redeployments every time
- 128. Heuristics for operational features 30% of total product budget 30% of dev team time
- 131. Run Book •Detailed description of how the system operates •Maintenance •Repair •Error recovery
- 132. Run Book / Ops Manual • 1 Table of Contents • 2 System Overview • 2.1 Service Overview • 2.2 Contributing Applications, Daemons, and Windows Services • 2.3 Hours of Operation • 2.4 Execution Design • 2.5 Infrastructure and Network Design • 2.6 Resilience, Fault Tolerance and High-Availability • 2.7 Throttling and Partial Shutdown • 2.8 Required Resources • 2.9 Expected Traffic and Load • 2.9.1 Hot or Peak Periods • 2.9.2 Warm Periods • 2.9.3 Cool or Quiet Periods • 2.10 Environmental Differences • 2.11 Tools • 3 Security and Access Control • 4 System Configuration • 4.1 Configuration Management • 5 System Backup and Restore • 5.1 Backup Requirements • 5.1.1 Special Files • 5.2 Backup Procedures • 5.3 Restore Procedures • 6 Monitoring and Alerting • 6.1 Error Messages • 6.2 Events • 6.3 Health Checks • 6.4 Other Messages • 7 Operational Tasks • 7.1 Deployment • 7.2 Batch Processing • 7.3 Power Procedures • 7.4 Routine Checks • 7.4.1 System Rebuilds • 7.5 Troubleshooting • 8 Maintenance Tasks • 8.1 Maintenance Procedures • 8.1.1 Patching • 8.1.1.1 Normal Cycle • 8.1.1.2 Zero-Day Vulnerabilities • 8.1.2 GMT/BST time changes • 8.1.3 Cleardown Activities • 8.1.3.1 Log Rotation • 8.2 Testing • 8.2.1 Technical Testing • 8.2.2 Post-Deployment • 9 Failure and Recovery Procedures • 9.1 Failover • 9.2 Recovery • 9.3 Troubleshooting Failover and Recovery • 10 Contact Details
- 133. Run Book / Ops Manual 2.1 Service Overview 2.2 Contributing Applications, Daemons, and Windows Services 2.3 Hours of Operation 2.4 Execution Design 2.5 Infrastructure and Network Design 2.6 Resilience, Fault Tolerance and High- Availability 2.7 Throttling and Partial Shutdown 2.8 Required Resources 2.9 Expected Traffic and Load
- 134. Run Book collaboration Dev team is responsible for the first draft “But I know nothing about Production!” Encourages collaboration with Ops team
- 135. Will Gray
- 136. not documentation build trust and understanding automate more over time http://runbookcollab.info/
- 137. choose tools that encourage collaboration
- 141. “How does [the use of] this tool help people to collaborate*?” * Work together, at the same keyboard/screen
- 143. ‘How to choose tools for DevOps and Continuous Delivery’ http://bit.ly/ChooseDevOpsTools
- 144. test early and often for operational readiness
- 145. operational readiness network testing security testing performance testing auxiliary infrastructure testing: monitoring log aggregation …
- 146. small set of rapid ‘weathervane’ tests for early warning
- 147. Network testing iTrinegy network emulators •Scripted setup and automated test runs •http://www.itrinegy.com/ Saboteur: •Network fault injection tool •https://github.com/tomakehurst/saboteur
- 148. Security testing Gauntlt: http://gauntlt.org/ SSL certs HTTP SQL injection … # nmap-simple.attack Feature: simple nmap attack to check for open ports Background: Given "nmap" is installed And the following profile: | name | value | | hostname | example.com | Scenario: Check standard web ports When I launch an "nmap" attack with: """ nmap -F <hostname> """ Then the output should match /80.tcps+open/ Then the output should not match: """ 25/tcps+open """
- 149. When I launch an "nmap" attack with: """ nmap -F <hostname> """ Then the output should match /80.tcps+open/
- 150. Deployment pipeline Make operational testing visible
- 152. MVP: ‘service unavailable’ page
- 153. test early for operational features using a deployment pipeline
- 154. single product backlog: (user) features + (operational) features
- 155. availability is the best feature