Using Machine Learning to Optimize DevOps Practices
Download as PPTX, PDF2 likes423 views
The document discusses using machine learning to enhance DevOps practices by optimizing monitoring and feedback through adaptive and fixed algorithms. It highlights the importance of data generated during development, continuous integration, and after deployment, as well as the role of machine learning in predicting future issues and diagnosing failures. The conclusion emphasizes that DevOps, being a data-rich environment, is well-suited for machine learning applications, where fixed systems are useful for immediate diagnosis and adaptive systems for trend analysis.
Using Machine Learning to Optimize DevOps Practices
- 1. Using Machine Learning to Optimize DevOps Practices Building Learning into Monitoring and Feedback Peter Varhol
- 2. About me • International speaker and writer • Degrees in Math, CS, Psychology • Technology communicator • Former university professor, tech journalist • Cat owner and distance runner • [email protected]
- 3. Agenda • What is machine learning? • How is machine learning applied to DevOps? • Challenges in training these systems • What constitutes an issue? • Summary and conclusions
- 4. What is Machine Learning? • Layered algorithms that change parameters based on feedback from know data • Can be linear or nonlinear • Algorithms can be fixed in production or adaptive • Fixed – algorithms do not adjust once deployed • Adaptive – algorithms continually adjust to new data • Usually part of a larger system
- 5. Adaptive Systems • Airline pricing • Ticket prices change three times a day based on demand • It can cost less to go farther • It can cost less later • Ecommerce systems • Recommendations try to discern what else you might want • Can I incentivize you to fill up the plane?
- 6. Why Use Adaptive? • The “right” result will vary over time • Trying to optimize a particular result • Revenue • The problem domain is not static Confidential, Dynatrace LLC
- 7. How Are Fixed Systems Used? • Transportation • Self-driving cars • Aircraft/Drones • Ecommerce • Recommendation engines • Medical • Diagnosis systems
- 8. Why Use Fixed Machine Learning Systems • The problem domain is static • The expectations remain constant • The right answer is known under most conditions • The original algorithms remain valid over a long period of time
- 9. DevOps Practices Generate Data • During development • Agile metrics, JIRA issues, test case metrics • During continuous integration • System test metrics • During continuous deployment • Quality metrics for deployments • After deployment and into production • Application availability and performance • Usage log files
- 10. Focus on Monitoring • Ongoing data on availability and performance • RUM • Synthetic tests • Application monitoring • Monitoring tackles the back end of DevOps • Identifying unhealthy trends • Diagnoses failures and poor performance • Recommends action • Fixed or adaptive depends on your goals
- 11. Where Do Predictive Analytics Come In? • Big data makes possible predictions of future events • Are we going to fail? • How will we perform with traffic surges? • As well as past events • What went wrong and how do we fix it • We can rely on past data • Adaptive systems may not perform as well • Clear goals needed
- 12. What Technologies Are Involved? • Neural networks • Genetic algorithms • Rules engines
- 13. Neural Networks • Set of layered algorithms whose variables can be adjusted via a learning process • The learning process involves training with known inputs and outputs • The algorithms adjust coefficients to converge on the correct answer (or not) • You freeze the algorithms and coefficients, and deploy • Or you optimize on a particular set of characteristics
- 14. A Sample Neural Network
- 15. Genetic Algorithms • Use the principle of natural selection • Create a range of possible solutions • Try out each of them • Choose and combine two of the better alternatives • Rinse and repeat as necessary
- 16. Bringing in DevOps • DevOps has data that can be used to train neural networks • Health of the application • Trends in application traffic and responsiveness • Application failure
- 17. Machine Learning Helps DevOps • Decisions are complex • Why is the CPU maxed? • What is causing disk thrashing? • Why did the network slow? • Why did the application fail? • Data is massive • Potentially thousands of data points a day
- 18. How Good Are Decisions? • Expert versus machine • Given the same data • In many domains they tie • With additional data, the human can be better • But machine learning will get better • But only as good as the data
- 19. We Want to Do Two Things • Identify trends that may indicate future problems • Increasing response times • More page errors • Diagnose faults once they have happened • Why did the application fail? • How can we fix it as quickly as possible?
- 20. Fixed Algorithms Work for Some Problems • Immediate performance and failure identification • Diagnosis of failures and performance issues • These are readily identifiable from known data
- 21. Adaptive Systems Supplement These Tools • Predictions of future events • Performance • Availability • The target is moving • So we need current data to adjust the algorithms
- 22. The Machine Helps the DevOps Expert • The machine learning app provides: • Early warning on possible performance issues and failures • Immediate notification of failure or impending failure • Trend analysis of data to predict unhealthy outcomes • The machine learning is an assistant • It can’t fix anything • It can’t necessarily identify the root cause
- 23. What is the Goal? • We have many ways of monitoring • Many of them are represented at this conference • Each measures something a little different • Latency, response time, availability, network, DNS . . . • Too much data can be no better than no data at all • Machine learning can correlate across measurements • Focus to eliminate false positives
- 24. Intelligent Systems Are Sometimes Wrong • The problem domain is ambiguous • There is no single “right” answer • “Close enough” is good • We don’t know quite why the software responds as it does • We can’t easily trace code paths
- 25. Testing Machine Learning Systems • Have objective acceptance criteria • Test with new data • Don’t count on all results being accurate • Understand the architecture of the network as a part of the testing process • Communicate the level of confidence you have in the results to management and users
- 26. A Cautionary Tale • All events are not created equal • AI systems treat events equally • A failure of a system during busy season is the same as any other • DevOps pros know otherwise • And can exert additional effort in response • And actually fix the problem • We can’t automate what we don’t understand • You need the human in the loop Confidential, Dynatrace LLC
- 27. Conclusions • DevOps is a natural environment for machine learning systems • Any activity that generates data and requires a decision is fair game • Monitoring is low-hanging fruit • Fixed systems for failure and diagnosis, adaptive for trend analysis Confidential, Dynatrace LLC
- 28. References • https://qz.com/989137/when-a-robot-ai-doctor-misdiagnoses-you- whos-to-blame/ • https://pvarhol.wordpress.com/2017/07/22/what-brought-about- our-ai-revolution/ • https://pvarhol.wordpress.com/2017/06/21/analytics-dont-apply-in- the-clutch/ Confidential, Dynatrace LLC
Editor's Notes
- #8: These types of software are becoming increasingly common, in areas such as ecommerce, public transportation, automotive, finance, and computer networks. They have the potential to make decisions given sufficiently well-defined inputs and goals. In some instances, they are characterized as artificial intelligence, in that they seemingly make decisions that were once the purview of a human user or operator.
- #14: Most machine learning systems are based on neural networks. A neural network is a set of layered algorithms whose variables can be adjusted via a learning process. The learning process involves using known data inputs to create outputs that are then compared with known results. When the algorithms reflect the known results with the desired degree of accuracy, the algebraic coefficients are frozen and production code is generated. Today, this comprises much of what we understand as artificial intelligence.
- #21: But there is a type of software where having a defined output is no longer the case. Actually, two types. One is machine learning systems. The second is predictive analytics, or adaptive systems.
- #27: Have objective acceptance criteria. Know the amount of error you and your users are willing to accept. Test with new data. Once you’ve trained the network and frozen the architecture and coefficients, use fresh inputs and outputs to verify its accuracy. Don’t count on all results being accurate. That’s just the nature of the beast. And you may have to recommend throwing out the entire network architecture and starting over. Understand the architecture of the network as a part of the testing process. Few if any will be able to actually follow a set of inputs through the network of algorithms, but understanding how the network is constructed will help testers determine if another architecture might produce better results. Communicate the level of confidence you have in the results to management and users. Machine learning systems offer you the unique opportunity to describe confidence in statistical terms, so use them. One important thing to note is that the training data itself could well contain inaccuracies. In this case, because of measurement error, the recorded wind speed and direction could be off or ambiguous. In other cases, the cooling of the filament likely has some error in its measurement.