Data driven devops as presented at Codemash 2018
1 like438 views
The document discusses the concept of data-driven DevOps and its impact on software development, emphasizing issues like overconfidence, the second-system effect, and the importance of collaboration among metrics across departments. It highlights how development choices influence operational costs and the necessity for continuous improvement in quality assurance practices. The presentation also addresses the significance of proper metrics for enhancing trust and predictability in development and operations processes.
Data driven devops as presented at Codemash 2018
- 1. #DataDrivenDevOps How to cook a precise software in a modern world @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
- 2. About me @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
- 3. shownotes @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops www.jfrog.com/shownotes Slides Video (tomorrow!) All the links! Ratings, comments Raffle!
- 4. Cobbler's children have no shoes @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
- 5. Poll time! @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops How many of you are software engineers? How many of you are optimists? How many of you are self-confident in their work?
- 6. Dunning-Kruger Effect a.k.a. ”optimism” People suffer from illusory superiority, mistakenly assessing their cognitive ability as greater than it is. Wikipedia @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops “
- 7. Second-system effect a.k.a. “self-confidence” The tendency of small, elegant, and successful systems, to be succeeded by over- engineered, bloated systems, due to inflated expectations and overconfidence. The Mythical Man-Month @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops “
- 8. The consequences @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
- 9. disillusion @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops Software I like Software I know really well
- 10. overconfidence @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops Every software is 80% ready 80% of the time Every developer, ever “
- 11. Hard to give up @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
- 12. @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
- 13. @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
- 14. How departments measure @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops Sales Marketing Support HR Finance Anecdotes from daily standup Engineering
- 15. How departments measure @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops Sales Marketing Support HR Finance Anecdotes from daily standup Engineering
- 16. @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
- 17. @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
- 18. @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
- 19. Engineering metrics in pre-devops era @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops Dev • Velocity • Architecture • Compliance QA • Incidents • Defects • MTTR • Code coverage • Test suite stability Ops • SLA • Cost env setup • Average customer cost • Tools stability
- 20. DEVOPS! @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops ?
- 21. Velocity to agile is like ? To devops… @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops ?
- 22. Velocity to agile is like NPS To devops… @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops NPS
- 23. Also, it’s a lot like profits @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops Metric Easily understandable Unity actionable Profit ✓ ✓ ✗ Velocity ✓ ✓ ✗ NPS ✓ ✓ ✗
- 24. Also, it’s a lot like profits @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops Metric Easily understandable Unity actionable Profit ✓ ✓ ✗ Velocity ✓ ✓ ✗ NPS ✓ ✓ ✗
- 25. Also, it’s a lot like profits @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops Metric Easily understandable Unity actionable Profit ✓ ✓ ✗ Velocity ✓ ✓ ✗ NPS ✓ ✓ ✗
- 26. Poll time! @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops Who knows what velocity is? Who knows what burndown chart is? Who has a burndown chart? Who looks at the burndown chart? Who trusts the burndown chart? Who knows what to do if it doesn't look right?
- 27. What’s the problem? @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops Velocity
- 28. True of all three! @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops Metric Easily understandable Unity actionable Profit ✓ ✓ ✗ Velocity ✓ ✓ ✗ NPS ✓ ✓ ✗
- 29. Too coarse! @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops NPS
- 30. Too coarse! @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops DevOps
- 31. This is devops* @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops Ops QADev Common goals, tools, culture Deep specialization *Unless you’re netfliX **You’re not
- 32. This is devops @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops Ops QADev
- 33. Same for metrics! @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops Ops QADev NPSSpecialized metrics
- 34. Same for metrics! @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops Ops QADev
- 35. How do metrics collaborate? @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops dev ops qa dev • Time to build • Release pipelines stability • Artifact replication topology • False reopen count • QA coverage ops • Avg cost of customer • Smoke tests quality qa • Test suite stability • Incoming defect velocity • Test execution time • Time to create new QA lab influenced affected
- 36. How do metrics collaborate? @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops dev ops qa dev • Time to build • Release pipelines stability • Artifact replication topology • False reopen count • QA coverage ops • Avg cost of customer • Smoke tests quality qa • Test suite stability • Incoming defect velocity • Test execution time • Time to create new QA lab influenced affected
- 37. How do metrics collaborate? @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops dev ops qa dev • Time to build • Release pipelines stability • Artifact replication topology • False reopen count • QA coverage ops • Avg cost of customer • Smoke tests quality qa • Test suite stability • Incoming defect velocity • Test execution time • Time to create new QA lab influenced affected
- 38. How do metrics collaborate? @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops dev ops qa dev • Time to build • Release pipelines stability • Artifact replication topology • False reopen count • QA coverage ops • Avg cost of customer • Smoke tests quality qa • Test suite stability • Incoming defect velocity • Test execution time • Time to create new QA lab influenced affected
- 39. How do metrics collaborate? @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops dev ops qa dev • Time to build • Release pipelines stability • Artifact replication topology • False reopen count • QA coverage ops • Avg cost of customer • Smoke tests quality qa • Test suite stability • Incoming defect velocity • Test execution time • Time to create new QA lab influenced affected
- 40. How do metrics collaborate? @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops dev ops qa dev • Time to build • Release pipelines stability • Artifact replication topology • False reopen count • QA coverage ops • Avg cost of customer • Smoke tests quality qa • Test suite stability • Incoming defect velocity • Test execution time • Time to create new QA lab influenced affected
- 41. How do metrics collaborate? @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops dev ops qa dev • Time to build • Release pipelines stability • Artifact replication topology • False reopen count • QA coverage ops • Avg cost of customer • Smoke tests quality qa • Test suite stability • Incoming defect velocity • Test execution time • Time to create new QA lab influenced affected
- 42. Cooking with metrics @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
- 43. Metrics categorization @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
- 44. Why? @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
- 45. ROI – is this activity is the right thing to do? @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
- 46. Continuous improvement @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
- 47. trust @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
- 48. sampling @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
- 49. Complete sample @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
- 50. @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
- 51. Representative sample @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
- 52. How effective are we? Samples are good enough for that As long as they are representative Need to be collected over time @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
- 53. Predicting the future @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
- 54. Let’s get practical @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
- 55. Development affects operational costs @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops Central container images repository Developers keep pushing images Storage prices skyrocket Not all PoPs need all images
- 56. Continuous improvement @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
- 57. Development affects QA @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops Anecdotes about test suite stability shared at daily startup Special issue type “test suite stability”
- 58. Return on investment @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
- 59. Development influences Operations trust @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops Operations want to predict costs and be trustworthy Will application updates generate more load?
- 60. Trust and predictability @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
- 61. Dos and Don’ts
- 62. Measuring the right thing @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
- 63. LoC LOL, anyone? @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
- 64. Avoid data overload @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
- 65. Use reliable data @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
- 66. Metrics generate incentives, be aware! @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
- 67. Tweak as you go @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
- 68. Elevate! @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops Raw Data Product Specific Dashboards Integrated Dashboards
- 69. @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
- 70. @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
- 71. @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
- 72. Step into data-driven life @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops Removes blame game Builds accountability and trust Creates common base for discussion
- 73. Q&A and links @jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops @jbaruch #codemash #datadrivendevops www.jfrog.com/shownotes