Making the Shift Left - Bringing Ops to Dev before bringing applications to production (DigitalXchange 2023)
Download as PPTX, PDF0 likes39 views
The document discusses bringing operations considerations into the development process earlier, referred to as "shifting left." It advocates designing applications with operations in mind from the beginning. This includes understanding operational objectives, constraints, and service level agreements. Application telemetry and monitoring are also important to incorporate from the start. The document provides examples of how to implement operational practices like deployments, health checks, and incident response processes in a shifted left model where development and operations work more closely together.
Ad
More Related Content
Similar to Making the Shift Left - Bringing Ops to Dev before bringing applications to production (DigitalXchange 2023) (20)
More from Lucas Jellema (20)
Ad
Recently uploaded (20)
![Get & Download Wondershare Filmora Crack Latest [2025]](https://cdn.slidesharecdn.com/ss_thumbnails/revolutionizingresidentialwi-fi-250422112639-60fb726f-250429170801-59e1b240-thumbnail.jpg?width=560&fit=bounds)
Ad
Making the Shift Left - Bringing Ops to Dev before bringing applications to production (DigitalXchange 2023)
- 1. Classificatie: vertrouwelijk Making the Shift Left – Bringing Ops to Dev before bringing applications to production
- 2. Classificatie: vertrouwelijk Making the Shift Left Bringing Ops to Dev before bringing applications to production DigitalXchange – 3rd June 2023 Lucas Jellema, CTO & Architect Conclusion Application Platform
- 3. Classificatie: vertrouwelijk Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 Architect on many integration initiatives with dozens of large and small organizations in The Netherlands Oracle ACE Director, Java Rockstar, author of two books on integration frequent speaker on conferences & active blogger Lucas Jellema Cloud Solution Architect & CTO [email protected] | technology.amis.nl | @lucasjellema | lucas-jellema 3
- 4. Classificatie: vertrouwelijk • 25 years Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 4
- 5. Classificatie: vertrouwelijk • Light Bulb moment Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 5
- 6. Classificatie: vertrouwelijk • Production! Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 6
- 7. Classificatie: vertrouwelijk Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 7 business process Application Application Application Platform Platform Platform Platform Infrastructure Infrastructure Infrastructure Infrastructure Infrastructure
- 8. Classificatie: vertrouwelijk Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 8 business process Application Application Application Platform Platform Platform Platform Infrastructure Infrastructure Infrastructure Infrastructure Infrastructure
- 9. Classificatie: vertrouwelijk Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 9
- 10. Classificatie: vertrouwelijk Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 10 • you make it run after you have built it • you keep it running
- 11. Classificatie: vertrouwelijk Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 11 • you make it run after you have built it • you keep it running
- 12. Classificatie: vertrouwelijk Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 12 • you make it run after you have built it • you keep it running
- 13. Classificatie: vertrouwelijk DevOps Cycle Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 13 https://medium.com/t%C3%BCrk-telekom-bulut-teknolojileri/devops-lifecycle-continuous-integration-and-development-e7851a9c059d
- 14. Classificatie: vertrouwelijk Shift Left – Operations & Development without hand-over Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 14
- 15. Classificatie: vertrouwelijk Shift Left – Operations and Development without hand-over Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 15
- 16. Classificatie: vertrouwelijk Shift Left – Operations and Development without hand-over Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 16 Operations Application Platform & Infra Development
- 17. Classificatie: vertrouwelijk What is Ops? • Make Up, Make Perform (as needed) • Handle Exceptions • Ensure Safe & Secure • Watch | Control | Reduce Costs • Optimize for Sustainable/Green run • Grease the wheels & Clean the floors • clean, prune, small technical maintenance, odd little jobs • (prepare for) Disaster Recovery • Report on Day to Day operations Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 17
- 18. Classificatie: vertrouwelijk Objectives & Constraints for Operations • Service Level Agreement (contract) • Business Requirements • Principles, Guidelines, Rules, Constraints • architecture, • security, • regulatory • Professional responsibility, ethical Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 18
- 19. Classificatie: vertrouwelijk What to Measure and what to Aim for? Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 19
- 20. Classificatie: vertrouwelijk What level of care? • Application + Platform + Infra need to allow for the required level of care • observe status (& trend ) • compare with goals • work within constraints • decide act upon action • have protocols to follow and instruments to take action • This requires early preparation Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 20
- 21. Classificatie: vertrouwelijk Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 21
- 22. Classificatie: vertrouwelijk Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 22 Basware invoices record, approve/deny daily approved invoices batch MS Business Central - Finance integration € € € €
- 23. Classificatie: vertrouwelijk Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 23
- 24. Classificatie: vertrouwelijk Common thread Requirements Development Build, Test, Q&A Release & Deploy Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 25
- 25. Classificatie: vertrouwelijk Common thread Requirements Development Build, Test, Q&A Release & Deploy Operations Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 26
- 26. Classificatie: vertrouwelijk Shift Left Requirements Development Build, Test, Q&A Release & Deploy Operations Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 27 Ops by Design
- 27. Classificatie: vertrouwelijk Required for successful operations • Definition of what is success • What are non-functional constraints and conditions? • Security, availability, performance, costs, CO2 • KPIs and targets? Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 28 Whenever the monitoring indicated that a service was down – the entire portal was stopped and restarted One particularly unstable service determined whether today was the user’s birthday. It triggered multiple restarts per day …
- 28. Classificatie: vertrouwelijk Application “Fingerprint” • Describe for each Application: • business value and owner/stakeholders • priority / criticality • business process • trigger (when/why) • result (& verification) • source and destination systems • data structures & filtering | mapping • non happy flows • quality controls • security • operational targets Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 29 Foundation for Solution Design and Test plan, basis for Operational mechanisms & processes
- 29. Classificatie: vertrouwelijk Dev => Ops Dev Intake / Ops Acceptance Deploy / Rollout / Activate Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 30 Observe Interpret Act Learn & Improve Routine Management Certificate, credential, user, endpoints; Backup, Archive, Purge, Patching, Cost Allocation Support Desk Fire drills / Chao(p)s Testing Report & Improve Recovery, Fail Over, Hot Fix, Rollback of Rollout, Scale Out, … Improve instrumentation, scalability, recovery, non happy flow handling, configurability Ops
- 30. Classificatie: vertrouwelijk Intake: Check on Operability of Application • Fingerprint • Definition of Success • QA-ed • Operability of Application Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 31 Dev Intake / Ops Acceptance Photo by Priscilla Du Preez on Unsplash
- 31. Classificatie: vertrouwelijk Ops Acceptance Test – confirm that operations can be done • Simulate realistic production runtime scenarios • Areas of interest: • detection | analysis | instruction | facilities | action | result Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 32
- 32. Classificatie: vertrouwelijk Deployment | Roll Out | Activation • Automated • Configurable environment specific dependencies at Application, Platform & Infra • Verifiable through post-rollout Smoke Tests • Activate for subset of workload: Blue/Green, Canary • Scenario to Rollback Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 33 Dev Intake / Ops Acceptance Deploy / Rollout / Activate
- 33. Classificatie: vertrouwelijk Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 34 API
- 34. Classificatie: vertrouwelijk Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 35 • Saturday Midnight • The loss of service was experienced on Monday morning Catalog Service API Products Service Products Service ?
- 35. Classificatie: vertrouwelijk Beyond Smoke Test: Health Checks & In-Production Testing • Periodically check availability of owned and called endpoints • In Production Testing: dummy business objects “business side-effect free” but otherwise very real Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 36 API ping No side effect – test all the way accessibility API create API validate Process business objects that are known to be dummies – not sent to external systems
- 36. Classificatie: vertrouwelijk Instrument & Emit - Observe – Interpret - Act Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 37 application application Platform & Infra telemetry Collect process & interpret Alert Report Visualize Analyze Act AIOps
- 37. Classificatie: vertrouwelijk Telemetry – M(E)LT • The need to know • what happens • what is the status • Dimensions • real time and after-the-fact • fine grained and coarse grained • business & functional, security, cost, CO2 and technical • Types of Telemetry: MELT • Events, Profiles & Exceptions Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 38
- 38. Classificatie: vertrouwelijk The Invisible Route Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 39
- 39. Classificatie: vertrouwelijk Tracing provides CCTV-like insight: when, where, what, why, who, how long • Track individual session | flow • Powerful aggregation across traces • group by component, status, origin, trace context attributes Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 41
- 40. Classificatie: vertrouwelijk • Log event (typically when something happens): logs can be associated with a job | request | transaction: • Logs provide drill down details for trace-span • Consolidated logs – from across application landscape – are powerful analysis tool • timestamp is crucial to sort and correlate platform & infra logs Logs – fine grained reports Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 42
- 41. Classificatie: vertrouwelijk Metrics – periodic mini status reports Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 43 platform & infra metrics • Timestamped measurements • Sometimes pushed, often polled/scraped
- 42. Classificatie: vertrouwelijk OpenTelemetry • Cross industry standard for collection of Telemetry • Virtually all tools for monitoring, analyzing, visualizing telemetry support OpenTelemetry as source Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 44
- 43. Classificatie: vertrouwelijk Events – telemetry context for analyzing and predicting • Black Friday • Horizontal Scale Out (double capacity for Service X) • Rollout/activation of new version of Service Y • Start of batch job Z • Snowstorm in the Köln area • Upgrade of Library P • New certificate for Component Q • Purged 2 TB of data from Time Series Database R • Outage in Azure Data Center (Frankfurt region) • Dortmund .. Bayern … Borussia .. nein, München meister • New engineer joined the DevOps team • Recovery test on database C • Ticket logged regarding poor performance • Expiry of Certificate K Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 45
- 44. Classificatie: vertrouwelijk Profiling • Dynamic analysis at runtime • Periodic system snapshots (sampling) • Metrics • CPU & Memory usage per process & child process | thread • Application Call tree • Duration & frequency of function calls • Detailed bottleneck analysis Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 46
- 45. Classificatie: vertrouwelijk Collect, Correlate, Interpret, Alert • Create the overall picture • Detect • Find what’s mssng – unexpected unevents • Automate Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 47
- 46. Classificatie: vertrouwelijk Ops Obs is a veritable BI challenge Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 48 platform infra Integration WebApp Function Job API Service Traces, Log entries, Metrics, Context Events Instrumentation & Configuration determines telemetry Telemetry Lake
- 47. Classificatie: vertrouwelijk The Ideal Dashboard … Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 49
- 48. Classificatie: vertrouwelijk The Ideal Dashboard … Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 50 Dashboard Past & Future Now
- 49. Classificatie: vertrouwelijk The Ideal Dashboard … Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 51 Dashboard - Now Past & Future Now Alerts Notifications Actions Heartbeat – Proof of Health
- 50. Classificatie: vertrouwelijk The Ideal Dashboard … Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 52 Dashboard – Past & Future Past & Future Now Trend & Pattern Charts Predictions Simulations / What-If explorations
- 51. Classificatie: vertrouwelijk Act upon Alert • Need to know what to do when and how to do it then • how important is it? what is priority? Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 53 ?
- 52. Classificatie: vertrouwelijk Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 54
- 53. Classificatie: vertrouwelijk Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 55
- 54. Classificatie: vertrouwelijk 1202 Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 56
- 55. Classificatie: vertrouwelijk Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 57
- 56. Classificatie: vertrouwelijk Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 58
- 57. Classificatie: vertrouwelijk Means & Mandate to act – Application & Platform/Infra • Inspect • Reconfigure • Scale • Restart • Resubmit • Apply Patch • (Re)schedule job • Perform recovery • Provision & Update Infra • Acquire (temporary) elevated privileges Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 59 Observe Interpret Act Learn & Improve Routine Management Fire drills / Chao(p)s Testing Report & Improve
- 58. Classificatie: vertrouwelijk Operations Firedrills • Testing Resilience of • systems & automated procedures • processes • people Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 60
- 59. Classificatie: vertrouwelijk Improvements over time • Observe and extrapolate • Explore What-If scenarios • Automate • Calibrate • Refine schedules • Fine-tune Tools and Procedures • Extend configurability • Automate (AI), dynamic, predictive configuration • Refine instrumentation • Improve telemetry processing • Decommission Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 61
- 60. Classificatie: vertrouwelijk Summary • Operations for successful business functionality • Define success • Design & Build Observability & Operability Test Operations • Shift Left – “Ops by Design” • Not DevOops but DevOps Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 62 Application Platform
- 61. Classificatie: vertrouwelijk Thank you for your attention I hope this was useful Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 [email protected] | technology.amis.nl | @lucasjellema | lucas-jellema 63
Editor's Notes
- #14: https://medium.com/t%C3%BCrk-telekom-bulut-teknolojileri/devops-lifecycle-continuous-integration-and-development-e7851a9c059d
- #22: angry suppliers their bills have not been paid. Why Not? They have been approved (in the invoice management app) - but not forwarded to the financial system where payment is done
- #24: the web portal was never the fastest application in the world but usually performed at a stable if barely acceptable level several times during the month its performance collapsed and it became unusable. 20-30 seconds wait times and time outs. it was unclear why that happened – at those specific moments
- #33: Common scenarios: Loss of network Crash of server (container) (mid-job) Peak traffic load Incorrect message payload Inaccessible endpoint Invalid credentials Power failure (forced restart)
- #40: https://www.google.com/url?sa=i&url=https%3A%2F%2Fptop.only.wip.la%3A443%2Fhttps%2Fwww.atlasobscura.com%2Farticles%2Fantwerp-invisible-route-map-surveillance-cameras&psig=AOvVaw1abfitkkT8VjPbxt27ywgm&ust=1669532135034000&source=images&cd=vfe&ved=0CBEQjhxqFwoTCLD4iICiy_sCFQAAAAAdAAAAABAE
- #42: https://opentelemetry.io/docs/concepts/signals/traces/#span-links
- #55: https://www.slideserve.com/ayala/se-3910-real-time-systems
- #60: expired certificate Product Owner cannot be reached Wiki is down Storage inaccessible Critical vulnerability in library DDOS attack detected Ransomware alert Data Corruption (human error) failing DNS server Azure availability zone is down team tech lead falls ill P1 bug in recent production release Edge connectivity lost Power outage
- #61: expired certificate Product Owner cannot be reached Wiki is down Storage inaccessible Critical vulnerability in library DDOS attack detected Ransomware alert Data Corruption (human error) failing DNS server Azure availability zone is down team tech lead falls ill P1 bug in recent production release Edge connectivity lost Power outage