![Continuous Monitoring using RER
32
HttpResponse RER Dashboards
RER
Dashboard
Inputs
Response Codes
URI
Elapsed Time
Error Rate measuring isn’t new:
•Companies like Google uses Error Rates as a key part of their Error Budgets for managing risk and their SRE
program
•Apps exceeding their ’budget’ aren’t allowed to release features
• In the simplest form, Raw Error Rate is:
[successful transaction count / total transaction count]
• Successful transaction do not include transactions that exceed the response SLA
Raw Error Rate Example: (Example: outage @ 2 AM vs 2 PM with a 1 hour outage)
ADTM: 98.9% availability for the day where as
Raw Error Rate: An application that serves 10,000 requests in 24 hours issues 30 errors over an hour at
2am. 99.7% availability for the day.
Agile Gurugram](https://image.slidesharecdn.com/agilegurugram2020valueofdevopsjourneyfromautomationtonoops-final2-200206052505/85/Agile-Gurugram-Conference-2020-Value-of-DevOps-Journey-from-Automation-to-NoOps-are-we-keeping-up-the-pace-Harsha-Vardhan-32-320.jpg)
![Site-Reliability-Engineering-v2[6241].pdf](https://cdn.slidesharecdn.com/ss_thumbnails/site-reliability-engineering-v26241-221023035909-82e9559b-thumbnail.jpg?width=560&fit=bounds)
More Related Content
What's hot (20)
Similar to Agile Gurugram Conference 2020 | Value of DevOps - Journey from Automation to NoOps, are we keeping up the pace? | Harsha Vardhan (20)
More from AgileNetwork (20)
Recently uploaded (20)
Agile Gurugram Conference 2020 | Value of DevOps - Journey from Automation to NoOps, are we keeping up the pace? | Harsha Vardhan
- 1. Value of DevOps - Journey from Automation to NoOps are we keeping up the pace? Harsha Vardhan Sr Director Software Engineering Optum Global Solutions Agile Gurugram The contents in this presentation is prepared by Harsha Vardhan in his personal capacity and to the best of his knowledge. No liability whatsoever for the accuracy and completeness of the above information is howsoever assumed. The opinions/ contents expressed/ provided in this article are the author's own and do not reflect the views of the Optum Global Solutions (India) Private Limited (“company”). Neither company nor the author shall not be held liable for any improper or incorrect use of the information described and/or contained herein and assumes no responsibility for anyone's use of the information.”
- 2. 2 “If the compiler knows that semicolon is missing at line 16 in Hellworld.java, then why it can’t simply put a ‘;’ there instead of telling us …”, new normal of software engineering ! Agile Gurugram
- 3. Everything as code ! 3 Agile Gurugram
- 4. Key take away 4 Agile Gurugram Reflection on Software engineering , DevOps maturity Enable software engineering team towards NoOps Deep dive on NoOps enablers : Cloud, PaaS, SRE, FSE Software Engineering Culture
- 5. Agenda 5 Everything as code DevOps is enabling NoOps NoOps deep dive - Paas, SRE, FSE Software Engineering Culture / Q&A 1 2 3 4 Agile Gurugram
- 6. Intelligent DevOps 6 Continuous journey (Continuous Integration, Continues Testing, Continuous Deployment, Continuous Operations etc.) of DevOps to become Intelligent i.e. Intelligent Integration, Intelligent Testing, Intelligent Deployment and Intelligent Operations Adaptive Automation + Predictive outcomes + CX Automation + Speed + Collaboration Agile Gurugram
- 7. Continued growth of DevOps 7 More than 80% of enterprise companies and 70% of small businesses are in the process of adopting DevOps practices DevOps Systems that Do Continuous Insight DevOps Systems that Think DevOps System that learns Next level of maturity needed as DevOps team raises the problem and send it to Dev to solve ! High performing IT organization practicing DevOps have 2.5x Faster lead times 3x Lower change failure rate 24 x Faster mean time to recover (MTTR) 10% Less rework Agile Gurugram
- 8. What is NoOps? 8 DevOps NoOps Development and operations work together early and often throughout the software development life cycle to ensure quality is built in at every phase Development and operations never need to interact with one another to get their job done Operations collaborates with development to select, build, monitor and maintain self-service solutions Development relies on self-service infrastructure , continuous integration and deployment solution Makes NoOps possible through principles and practices that are independent of any specific software solution Is only possible through the use of specific software solution NoOps is successful DevOps “I don’t want DevOps. I want NoOps.” Mike Gualtieri of Forrester Research Agile Gurugram
- 9. FAANG companies are early adopters of NoOps, its our turn now ! 9 Key drivers for NoOps Platform As Service (PaaS) Site Reliability Engineering (SRE) Full Stack Engineers (FSE) Agile Gurugram
- 10. Application/Services evolution journey … 10 Based on proprietary appliances or platforms like Oracle, WebSphere. Monolithic design, changes require a lot of planning. Relies on vendors or specialists to update/maintain "Lift and shift" from proprietary to open source. Embracing automation, CI/CD. Refactoring to modern frameworks and possibly breaking monoliths Primary workloads containerized or server-less, able to scale portions of the service horizontally. Fully embraced automation and uses IAC (Infrastructure As Code) to manage environments. Application designed for resiliency and availability, multi-region, circuit breakers Legacy Modernized Cloud Enabled Platform Enabled Streaming, API’s, Materialized Views, Standardized Data on top of a resilient and scalable platform Legacy Modernize Cloud Enable Platform Enable Agile Gurugram
- 11. Its all about Value arbitrage ! 11 Labor savings Labor Infrastructure savings Infrastructure Automate Optimize Modernize Agile Gurugram
- 12. Data platform approach 12 Access Patterns: APIs, Events, Analytics Agile Gurugram
- 13. Get ready for PaaS 13 • Create Networks, NACL’s, FW’s, Ingress, Egress. • Deploy VM’s, packages, etc.. • Storage systems: block storage, provisioned IOPs. PaaS • Whitelisted set of PaaS services available over HTTPS. • NO VM’s, NO Networks, No TVM. • FaaS, NoSQL, API Gateway, Kafka aaS, etc. IaaS Legacy patterns Future Development Agile Gurugram
- 14. Benefits of PaaS (Serverless) 14 PaaS requirements: • 100% self-service • No noisy neighbors • Tier 0 services • Hyper Elasticity • Per second billing • 30% cheaper compute/storage • 50% security tools/ops savings • 20% reduced capital for infra builds Hard savings • Faster Time to Market. • Greater Resiliency. • Hyper Elasticity, pay only for what you use. Soft savings • 80% of attack vectors removed (no networks or vm’s) • Threat modeling focus on app logic/libraries • Decreases cognitive load for app teams Security benefits Agile Gurugram
- 15. Unlocking power of SRE 15 DevOps team raises the problem and send it to Dev to solve, whereas the SRE team take the ownership of solving it too ! SLO/SLI | Eliminating toil | Release engineering | Error budget Agile Gurugram
- 16. SRE by Numbers , an illustration 16 We delivered significant business value We celebrated our successes • 9.8% Increase in transactional NPS from Q1 thru Q4 2019 • 52 % Growth in Virtual Visits from 2018 to 2019 • Reduced TCO ( total cost of ownership) by xx million • TTB (Time to build) improved from 5.4 wks to 2.4 wks • TTV ( Time to value) improved from 8 wks to 4.2 wks • Mean deployment frequency at 1.7 wk, this difference to TTB attributed to SRE • Release compliance automation improved from 0% to 18% • TTD ( Time to detect) reduced to 0 min • TTR ( Time to resolve) Reduced from 70 min to 12 min • Improved availability from 99.96% to 99.98%, RER < 5%, ↑5k Training Hours Staff in Engineering Roles 83% We invested in ourselves Agile Gurugram
- 17. SRE success culture at LinkedIn (a case study) 17 Fighting fire in the early years in early yrs , they were not even called as SRE. Their role was more of mixture of firefighting, release mgt, and conventional it ops. Linked In went through hyper growth, when things finally leveled up a little, they reorganized as SRE team and tasked themselves with a clear goal in mind and that was to keep the website up and running all the time. this overhaul to an extent was successful. The site was moved into a more stable position and the role of operations team to solve the issue via software instead of people and process. Dealing with the culture debt initially they focused on technical skills above all in their mgt and hiring process. After a while the experience became so painful for everyone involved that a need for change was observed. This was similar to how a tech debt builds up in a long lived code base. Philosophy The head of Engg and Operations at LinkedIn was David Hanke, who began promoting the SRE to have mindset of attacking the problem and not the person involved. It fostered a culture of equality and inclusion . People Formalizing interview process Process Now a days LinkedIn SRE team consists of 100+ SRE located in different geo location. If a defensive behavior is observed, they will tell the team to attack the issue and not the person. They don’t make believe to be perfect and understand that there is still a lot of work to be done. Agile Gurugram
- 18. The “Stack” in “Full Stack Engineer 18 Front End / Client Side Language Middleware / Services DBMS / Database Server Side Language & Framework Platform HTML, JavaScript, JQuery, CSS, Angular JS, ReactJS, AJAX, NodeJS, Ruby etc. AdaptableAcrossTechnology DevOps C# .Net, ASP .Net,J2ee, JSP, Servlets, Ruby on Rails etc. Transformation Tools / Mainframe Mule ESB, Web Services, , DataPower , Layer7 etc. Structural DB - Oracle, MySQL etc & No SQL – MongoDB, Hbase, Redis. WTX, DataStage, Talend , Mainframe etc. CI/CD Tools (Jenkins, XLdeploy etc), Continous Testing (Selenium, Jmeter etc), Continuous Monitoring Cloud (AWS, OpenShift, Azure), Unix, Linux etc. Full Stack Engineer(FSE) (Adaptable Across Technology and Proficient in Engineering Competency) Xamerin, iOS, Andriod etc.Mobile Application Technology Stack Software Engineering Competency Stack Algorithm & Data Structure Problem Solving & Coding Skills Data Structure Basics, Time Memory Complexity, Big O Notation concept, Sorting Algorithm (Heap, Merge, Insert, Bubble etc), Recursion, Linked list - double, circular etc) Tower of Hanoi, Find Celebrity, Reference to problems /puzzles: https://www.hackerrank.com/domains/java https://codingbat.com/java https://www.geeksforgeeks.org/puzzles/ S/w Engineering Practices Agile (Scrum, Kanban), ITIL / ITSM Design Concept & Design Thinking High & low level design Design Patterns (IOC, MVC, Singleton, Factory etc) Design Thinking - Solve wicked problems by applying the possibilities of technology, and the requirements for business success Agile Gurugram
- 19. Challenges 19 Winners do take PitStop to stay ahead in the race ! Quick wins Vs Long term solution RPA / Process Re-engg API performance ownership vary from ms to sec Business affinity with legacy system and funding challenge for modernization Infrastructure as code automation with Xx manual touchpoints Seasonal nature of work /release frequency Why to change something that’s working fine mindset Agile Gurugram
- 20. Its all about software engineering culture… 20 Optimize for iteration speed Push relentlessly toward automation Build the right software abstractions Develop a focus on high code quality with code reviews Maintain a respectful work environment Build shared ownership of code Invest in automated testing Build a culture of learning and continuous improvement Hire the best talent Engineering Excellence Engineering Excellence Agile Gurugram
- 21. Thank you
- 22. DevOps Systems that Do 22 Code Management Code Merge Packaging Build (CI) Server Unit Testing Code Coverage App Config Publishing Deploy Auto Test/ ATDD Dashboard/ Reporting PublishingDashboard/ Reporting Smoke Testing System/ Integration SecurityScan Dashboard/Reporting Network Config Continuous Testing Continuous Delivery Continuous Integration Non-Functional/ Performance Dev Tech Lead Tech Lead Architect Dev/Architect Dev/Architect Architect Architect Dev/ArchitectSDETSDETSDETSDET Architect Architect Architect Architect Architect ArchitectDev/SDET Medium High One Time ContinuousLow Efforts / Complexity Regression 1 1 1 1 3 1 2113 3 3 3 3 3 3 3 3 3 3 Agile Gurugram
- 23. Continuous Insight 23 In the spirit of the “Continuous” philosophy in DevOps, we envision a new solution approach around “Continuous Insight” where analytical insights are generated and acted upon continuously (and autonomously) as processes execute Incident Data Log Data Monitoring data NPS data Agile Gurugram
- 24. DevOps Systems that Think 24 Agile Gurugram
- 25. DevOps System that learns 25 CX-driven DevOps (or CX-Ops) that requires data analytics and cognitive approaches (such as natural language processing or NLP) to decode meaningful insight • Transactional NPS feeding into log monitoring • Monitoring triggers drives Bots / Agents availability for chat / call • Peak Season readiness Agile Gurugram
- 26. Public cloud comparison overview 26 Built their cloud last. Learned from mistakes of MSFT and AWS. Leapfrogged technically. • Most future ready model. • Least mature service offerings. • Least industry support. • Strengths: AI/ML, Serverless, No SQL, Mobile • Weaknesses: Traditional IT, Hosts, Networking, Industry support • Excellent security controls and visibility. • Hungry to displace Amazon and willing to partner more deeply to do so. • Good service breath, but not equivalent to AWS. • Strengths: BI, MSFT platforms, Security. • Weaknesses: Stability, Breath of Offerings, CDN • Provides mature services and capabilities. • Breath of service offerings is staggering. • Best industry support. • Largest third party marketplace. • Strengths: Stability, Scale, Service Maturity, Industry Support • Weaknesses: AI/ML, Security Visibility, MSFT platforms Google Microsoft Amazon Google Cloud Microsoft Azure Amazon AWS Agile Gurugram
- 27. RCA - Release Compliance Automation workflow 27 Agile Gurugram
- 28. 28 How might we leverage machine learning to identify changes with the highest potential risk for negative outcomes? Agile Gurugram
- 29. Solution Architecture 2929 ITSM Datamart Data Ingestion Completed Changes (Scheduled Batch) Model Building, Training, Re- Training, and Validation Additional insights to supplement model Trained Model ET Data Lake New change request Enhanced Risk assessment Machine Learning Model API Service Now API Planned Changes (real-time) Web Service Dashboard updated daily result input Source code Data Processing / Transformation NLP Agile Gurugram
- 30. Model Training & Selection – The Recipe 30 • Performance metrics, several models, pick the best • Gradient-Boosted Trees implementation XGBoost • Ensemble combines many “weak” learners to create strong learner TPR FPR Algorithm AUC Precision Recall F1 Logistic Regression 0.825 0.162 0.135 0.147 Decision Tree 0.831 0.192 0.176 0.184 Random Forest 0.854 0.213 0.191 0.201 Gradient Boosted Trees 0.906 0.292 0.269 0.281 Neural Network - LSTM 0.848 0.089 0.176 0.142 Test-set performance metrics Higher risk Agile Gurugram
- 31. Availability 31 Traditional approach of uptime vs Raw Error Rate An availability metric that is an accurate indicator of true customer experience Agile Gurugram
- 32. Continuous Monitoring using RER 32 HttpResponse RER Dashboards RER Dashboard Inputs Response Codes URI Elapsed Time Error Rate measuring isn’t new: •Companies like Google uses Error Rates as a key part of their Error Budgets for managing risk and their SRE program •Apps exceeding their ’budget’ aren’t allowed to release features • In the simplest form, Raw Error Rate is: [successful transaction count / total transaction count] • Successful transaction do not include transactions that exceed the response SLA Raw Error Rate Example: (Example: outage @ 2 AM vs 2 PM with a 1 hour outage) ADTM: 98.9% availability for the day where as Raw Error Rate: An application that serves 10,000 requests in 24 hours issues 30 errors over an hour at 2am. 99.7% availability for the day. Agile Gurugram
- 33. Customer Experience Monitoring 33 Why enhance monitoring? Agile Gurugram